He may indeed be stressed, but it sounds as though he may also be using that stress as an excuse for resuming his friendship with his ex. Try to have a talk with him at a time when the two of you are not arguing or otherwise on edge.
If he still presents as angry and defensive, you might consider whether this relationship is really heading in the direction in which you wish it to go.
I don't believe in abandoning your partner just because times get hard, but it sounds to me as though your partner may already be looking for a crack in the door. Only you can determine how much of that is due to his mental frame of mind right now, and how much is due to secrets he might be keeping from you.
Good luck.
P.S. Trust is a big factor in the success of any relationship, and it sounds to me as though you are having some trust issues as far as this man is concerned. If you tend to be the jealous type, that may all stem from you and you should acknowledge it and do your best to get over it. But if he has given you good reason not to trust him, then I think I'd show him the door.
