It seems to me that when you have him, you decide that you don't want him for some reason that we are not made aware of. I am not trying to sound mean, but I think that it is probably time for both of you look for another mate. It doesn't sound like the relationship will ever develop so it is time to move on. You both deserve more.
