When is it time to really call it quits ...

Only you can answer that. You haven't stated what issues you are having that would give cause for your question. Perhaps your Minister, or some other Counselor can provide more in sight for your question.

Find a good marriage counselor.  Both of you should be ready and willing to speak honestly about how you each feel about your situation.  Don't side-step the real issues at hand.  You both know how you feel and now is the time to really put it on the table with an objective third party who is professionally trained to guide you in the right direction.  It takes a lot of soul searching to come to a final decision and it will take time for both os you.  Best of luck to you.

#1 is destructive abuse (abuse occurs for all genders/gay society). If you like to wake up every day with a huge bruise or broken ribs than stay in the marriage and be their punching bag. You're not Ali's daughter or Mike Tyson. You respect each others bodies.

#2 addiction to affairs (sex). Sex/romance stays in the home not shared by others. If you like all that open tunnels and/or sticks than why get marreid. Stay single and leave the committed life. Marriage is forever.

#3 When the I Love You stops and separate bedrooms exist. Don't be fooled if the separate rooms/beds start occurring by lame excuses that you snore, talk in your sleep, fart too much when sleeping or you kick too much while asleep. This stage has nothing to do with another mate. Its just they get bored and tired of looking at the same old face or nagging.

Ironically, all the above happened prior to my decision of getting a divorce.

Counseling works for some but not others. I would say when you had enough of his bs or bad behavior. Jesus wants you to have peace than seek it but its never going to be easy. Some feel they don't get the needed attention from their spouse so they look elsewhere meaning they start to cheat. Some marriages jump to the wrong conclusions and divorce. I feel the only time if its necessary would be abuse. If your spouse is beating the daylight out of you than yes, seek freedom and safety. Another reason to divorce would be if you came up with STD/HPV/Herpes, etc. That is a redflag of cheating. So many reasons have to play in part to this divorce. I divorced due to my ex's drug problems, abuse, cheating, lies and stealing. I have a drama grief like every other person. We all deal with it a different way. Some seek God. Some seek the bottle. Some seek their graves. I struggle as I speak. I still love my ex. Why? I ask that and many others ask why do you want to go back to saddness. I say in his defense that he could if he wanted to get off drugs and use my love in replace of it. Of course, I've been divorced 3 years and still sad about it. He left me. Yet he had said he never wanted to be divorce. Now, I don't know if he was saying sorry or that he wanted me back in not so much words. I feel my situation is no different than any other person. Some have said it was a control issue. He wanted to manipulate me ok make me think he wanted me but all he was doing was hurting my feelings cuz I am the one that did the divorce. Now, he's moved on and I haven't. Yes, its my fault like many other ladies fault who seem not to let go. It could be the issue of low self esteem or the fact that they lost and their spouse won. Its more of a competition. I feel that way. I may be the one who done the divorce but he wins cuz he now has a girlfriend. At least thats what his mom said. I am losing cuz I have no one in my life but God isn't finished with me. I may never find another love. I believe in God. He did this to save my life. So, the answer to your question would be when God says its time to move on. He rather you be in a safe and happy environment. I know easier said than done but I grieve and its my fault but God is looking down at me and waiting for the right moment to turn the pages of my life. So, once again its when that last page of your chapter is wilted and you've had enough of your spouses behavior. This is the time to divorce when you begin to see circles or bags under your eyes or your skin turns yellowish or you look like death had came over you ok you look so old and stressed out. Thats the time to divorce. Get back your youth and peace.