It would seem that turning a baby upside down might potentially compromise a baby's brain, neck and breathing apparatus.  Years ago, it was pretty standard that babies and small children were tossed around in all sorts of ways, but as medical knowledge increases, we're learning that many behaviors once considered normal carry unreasonable risks.  

For instance, we now know about Shaking Baby Syndrome, and it would seem logical that one could possibly inflict SBS-type injuries on a child by swinging it around or turning it upside down without sufficient support.  (I don't know the specifics, but it would be a good topic to look up.) 

The fact that you asked the question signals that you probably have common sense concerns about doing it.  Common Sense is a great thing.  Common sense will allow you distinguish between wildly swinging a baby around vs. fully supporting and cradling a child to "swoosh" him/her around a bit.  Follow your best instincts. 

NOTE:  Please read up on baby CPR and choking.  LIfe-saving tactics include some maneuvers which call for placing a baby in unusual positions.  Knowing those tactics is important, and you will probably learn alot about the what is safe or not.