How can I remove stains left by evaporated water ...

I use simple vinegar to remove such stains from faucets and the like, and i heard that lemon juice is also effective. It is natural, biodegradable and environmentally friendly.

The only problem is you have to check first that it doesn't eat away at the brick too, so try a hidden corner before applying all over the place. 

That substance is called efflorescence and is caused by excessive, continual moisture. You can get rid of the dust but it will come back if you don't deal with the moisture problem. Handle it as explained on this HomeTips page: Concrete, Stucco & Brick Efflorescence