First of all, if you only see him every once in a while, I can't imagine that working with him would really be a bad thing.  I think that you should put the working together issue aside, because it doesn't really seem like an issue, to me.  I would just flirt back... I'd make it clear that you're interested, and even ask what he's up to after work.  I would just be in touch with him through IM and get to know him better.  If you guys seem compatible, I'm sure he'll ask you if you want to hang out together.  You can ask him also if he wants to get coffee or do something.  Good luck.