First, it takes two people to make a baby. So he may have had ulterior motives in mind, but he didn't "get" you pregnant. You were equal partners in this conception.
Next, the courts generally do not cancel visitation without a very good reason (the non-custodial parent is a convicted felon, child abuser, illegal drug user, etcetera). And even then, it is possible that the court would order supervised visitation.
Finally, if you are certain you no longer want to be with your baby's father, you need to begin by having a consultation with an experienced Family Law attorney.
If you do not have an attorney, contact your state's Bar Association for a referral to an attorney in your area (there is usually a nominal fee).
When you meet with the attorney, be prepared to ask him or her all about your legal rights and responsibilities, as well as those of the father-to-be.
The attorney will be able to advise you as to how best to proceed.