Sweetie, if he were truly the man of your dreams he would not have walked out on you and the children.
Furthermore, he would not be laying all of the blame upon you for his doing so.
Someone who is the man of your dreams would be man enough to discuss things with you so that you could try to reach a solution, TOGETHER.
That being said, I agree with OronD that his wanting you and the children to come for the weekend is potentially a good sign.
I say you should go ahead and go. And while you are there, do your best not to argue or pick at him - but instead, try to channel the feelings and behaviors that made you fall in love with each other in the first place.
And then, at some point during the weekend, I think you should tell him that you love him and you would like to be able to work things out. Tell him that you understand his explanation of what was bothering him, but that any relationship takes LUST in order to be successful.
LUST is an acronym. It stands for Love, Understanding, Sex and Trust. With a little LUST plus some honest communication (with the help of a counselor, if necessary) the two of you should be able to get back on track - if that is what you both want. (If he doesn't want to make the effort, I'm afraid there's nothing you can do.)
Good luck!