What comes around goes around.
If your boyfriend hasn't given you any reason to worry, you shouldn't be worrying. If for some reason he has, than I think you need to talk to him. Speaking with him should allay some of your insecurities. Good luck.
