What the hey, go for it. Just let him know up front your feelings and insecurities.
If you like him and he likes you, why not. Just because you are dating him doesn't mean you will be with him 24/7.
My wife and I have been married for 35 years and we still have our private space where no one can enter. Good luck.
Ken Reed Sr