Stress and anxiety can harm your sperm counts. There’s lot of scientific data to support that statement!
The thing you need to know is that any change you do today in your lifestyle will take ~3months to be reflected (positively or negatively) on your counts. I know it’s easier said that done, but try to find ways to release stress. Do you excercise? that always helps. You can also try acupuncture, meditation, or yoga. The mere act of doing something to improve your anxiety levels will help you feel better (talking from personal experience here!).
You might want to start taking some multi-vitamins that might help improve your numbers. There’s nothing to lose and in my husband’s case we did see an improvement after he started taking Fertility Blend (~$23 for a one month supply in Amazon).
I hope this helps a little bit. Best of luck!!