Hi,
I have to be blunt here. First of all, you do not have any business entertaining a married man and hanging out with him. If he is really interested in you, he would divorce his wife first.
He may be using you for sex . Please show some decency and self-respect. What makes you think that he won't do the same to you in the future?
He does not like to discuss it, so just break off any relationship with him. Youll just end up being hurt and you are also going to hurt his wife.
Hope that you will take this advice for your own sake.