Considering caffeine is a proven trigger for anxiety and panic attacks, eliminating could be one of the most important steps in finding relief. By eliminating such a stimulant from your habit, it can aid in lowering your adrenaline and cortisol levels, thus making it easier for you to deal with the main cause of that anxiety and panic.
However, you should not cut caffeine cold turkey. Doing so is proven to have certain withdrawal symptoms. Headaches are common. If you want to be sure with how you can best eliminate caffeine, then talk to your doctor.