You wait, and wait, and wait, and wait . . . Sorry. I couldn't resist.
It really does take patience and planning but I have found that if you use the widest angle lens possible it allows you to get as close as possible to the building resulting in fewer people to contend with in the viewing screen.
Of course if you get too close you may have to lean the camera upward to get the whole building which results in keystoning, but this is pretty easy to get rid of with software these days.