Sounds to me like there are pieces missing in this puzzle. Perhaps there is something that is bothering him or that he's embarrassed about. You need to open this issue in a very sensitive way and give him a chance to tell you how he feels.
Keep in mind that you might also need professional help to solve this problem. I wouldn't be reluctant to think of going to counseling together.
But first things first talk to him and also share with him how this situation makes you feel.
Good luck.