The answer above sounds helpful as to specific advice. I have been on Social Security Disability since 1993. I haven't yet run into any pay back questions. However, despite not being a practicing lawyer, my guess from my law school days is that they do not have the right to make you repay any of the benefits you have received. The answer above said to read the contract. Many contracts are actually illegal as written. If a contract is "unfair, or unconscionable", it sounds fraudulent to demand any repayment from you. You could have your lawyer simply challenge the contract as illegal and stop any legal ruling that you owe money.
Many years ago in 1992 I was on unemployment through the State of Calif. and I had health problems and couldn't work. I then was advised to apply for disability. As soon as my disability was approved, the State unemployment office sent me a big bill for every little unemployment payment they had ever given me. I also had a strong legal challenge, but with health problems didn't have the strength to challenge it. So I had to repay them about $3,000-$4,000. However, if I had fought it, I'm sure I would have won. They were wrong in their reason for charging me.
Good luck. I think it's clearly a case of fraud against you and you should not pay back anything. With a good lawyer, I know that will be the outcome.