Neverwhere, as I don't have enough information to formulate a response, it is difficult to say what is going on behind the scenes. 

Infatuation is not to be confused with love.  One can be in love with a stable and compatible person and still be infatuated with others, but (s)he will always come back to the sensible lover.

Jealousy can be a monster, though.  That is why Moses said, "Thou shalt not commit adultery." 

That occurs in three forms:
1.  One night stands.  It is unfortunate but it happens.  Some secrets are best confessed.
2.  Extended extramarital affairs.  That is a red flag that there is a serious compatability issue and the marriage was doomed from the outset.  Depending on circumstances, divorce can be the humane alternative.
3.  Unacceptable sexual misconduct.  As hard as it might be to believe, some of us do not enjoy kinky sex.  It is one thing to look at it on the internet and masturbate, but it is entirely another to bring an unwilling spouse into bondage/discipline/sadism/masochism.  That goes equally for BOTH partners of a marriage, unless they agree that such is the nature of their "relationship."

No one appreciates being a cuckold spouse.