It seems like the ball is really in your court. He has made himself available, and everybody else noted that he's interested, only you are still not sure. It's ok not to be sure, but I think you are interested, and are afraid to let him know how you feel because you're not entirely sure what he's going to say.
I think you should take the risk, and allow him to take you out sometime.
