I had 3 children....wanted more.
I believe you should go by the health and the wealth of the family. If you cannot afford children either emotionally or financially, you should not have them.
If you are both healthy enough physically and emotionally to handle them and if you can afford to feed and clothe and educate them, then have a whole tribe. Any number you can handle.
It's the happy, well adjusted and financially stable couples who need to be populating this world...not the other way around.