I think he hasn't grown up to be a very good man. You must remember that the children of parents who were abusive, often become abusive themselves. I think you're doing the right thing, taking time to get to know him better. But really, I think your attraction to him is based on the child you knew rather than the man he now is. And that's naturally a VERY different person than you knew. Just as you are a different person than he knew.
