Can't look at me the same way

So, he wants to hold you to a different standard than that to which he held himself?

I think not.

Okay, my first instinct is to tell you to advise him not to let the door hit him in the rump on the way out. On the other hand, you have children and a history together, and I gather you'd like to make things work.

So...

If he is willing to give it a try, couples counseling is your best bet. Try it without a specific goal in mind, say for a 6-month period. At that point, evaluate where you are and if you feel it's worth continuing. With any luck, you'll be able to get back on track and build your relationship into something worth having.

If he's not amenable to counseling, then I submit it wouldn't matter what you had done. He's just using that particular incident as a reason to break up and move on.

What you did while you were broken up with him is really none of his concern. You weren't together, he had no claim on you. He's just being a dickhead. I have no sympathy for him at all. If he can't get over it, find someone that deserves you. He doesn't.

He will NEVER get over his Jealous issues without professional help.  If he doesn't think he needs help, then move on.  Jealousy is not something you can live with.

Your dilemma is a perfect example of why YOU NEVER DIVULGE THE DETAILS OF YOUR SEXUAL HISTORY!  Men will always hold it against you!  Some men more than others, but believe me, they will use it against you at some point in your relationship.  He is suffering from a lack of self confidence and self-esteem!  When people don't feel good about themselves, (which starts in childhood), these insecurities come out in adult relationships as inappropriate rage, jealousy, suspicions, verbal abuse, etc.  Without therapy, he will never be able to change.

Even if he loves you a lot, he still cannot make himself into somebody he's not! He can't change his BRAIN!  It is programmed from childhood and "is what it is!"  So if he feels like he can never love you completely, forgive you for your past indiscretion, and forget your past, then he is choosing to withhold that from you makin it impossible for you two to move on as a couple.  Without therapy, I see no hope for him changing or even realizing that it's HIS PROBLEM, not yours.  Unfortunately, he's SELFISHLY MAKING his problem a "FAMILY AFFAIR". 

Of course, I wish for you success in making this marriage work, but HE WILL HAVE TO PUT IN A LOT OF WORK ON HIMSELF IN ORDER TO ACCOMPLISH THIS AS A COUPLE.  But I fear he is not willing to do so.  If not, then let him move on.... he will have the same problems with his next woman.  His problems will move on with him!  Let this be a lesson:  Take your time getting into another relationship, and when you do, DO NOT UNDER ANY CIRCUMSTANCES DISCUSS YOUR PAST SEXUAL EXPERIENCES!  It's no one's business!  Keep it private!  GOOD LUCK!