No, since you can't control which sites link to your site, your ranking will not be harmed them. Nonetheless, you can control on links from your site. These may harm your ranking if linked to a penalized site. Read lot's more about the ranking in
here.