If they're not washed, now this is being very negative, and assuming the person has some sort of virus, you could get a sort of fungal infection - same fungus as athletes foot - easy to treat, I also hear that you could get the herpes simplex virus from towels so I imagine that's a similar situation. But if it's washed, no problem I don't think, but just be sensible, if you've already done it and got no symptoms I wouldn't worry, if you are thinking about doing it, just make sure they've been washed, or......take your own!
hope that's helpful