I manage a few Google AdWords campaigns for businesses, and we charge on the basis of a fixed-rate per visitor that we generate. The final arbiter of how many visitors we've generated is the company's own traffic analytics report, which, curiously, always shows slightly fewer clickthroughs than Google AdWords actually charged us for. So why are 1000 people clicking on the company's Google AdWords ads, and only 995 being registered on the site itself as having arrived from those ads?
I wonder if it's because the person clicking through never sees the page, because it's never delivered by the company server? There's no doubt about it, our impatience when using the web is increasing in line with our ever-increasing access speeds, and if you're presented with the company logo at the top of a page, and then a white space where something's loading, you don't give it long before clicking "back".
Most servers are pretty fast nowadays. The only real server-side problems I come across are where large corporations run their own ones but don't invest enough in them ("I keep telling the German head office how slow the website is, but they won't listen"). Frustratingly, you'd get better service if you put the whole site on a £3.99/month server at one of the big consumer web hosts. The real problem in pages not loading quickly enough for your customers is more likely to be bad website design. Complex scripts, calls to external servers for vanity features, unoptimised images …they're all a potential source of slow page loading, and visitors just won't put up with that.
It would be a huge shame if you put in loads of effort in getting better positions in Google to generate more traffic, then failed to deliver after the clickthrough. One aspect you can look at straight away is discussed in Make Your Images Load Faster on the Marketing Optimization Blog. The article gives some useful free tests you can try today.