Yes it affects your ranking
because googlebot, when it visits your website, it visits it for X period of time
and if it didnt manage to see the entire site within that time, parts of it are not scanned, and information that maybe on those parts wont appear in google
Users are also annoyed when it takes a site ages to load