Pretty bad, I think. It can screw up your kidneys and cause other problems, including nervousness and sleeplessness. I quit caffeine more than ten years ago, and now I don't miss it at all. The first month I slept a lot. A whole lot.
After giving up drinking it, I started getting a caffeine rush from chocolate, strong enough to make my legs twitchy when I tried to sleep. It is a really powerful drug that we have accustomed ourselves to.
Eventually, sooner rather than later, you won't need the caffeine in the morning to feel energetic. The rest you got from a good night's sleep will be all you need.