I think the minute you start thinking about this, you're going to fulfill this fear, si I wouldn't even go with a partial limitation.
If you take cream and sugar, replace it with low fat milk and artificial sweeteners. I always thought caffeine is one of the safer addictions one can have.