You may be thinking that it's a usability issue, not something to do with your site's position in Google, but you'd be mistaken.
It's been more than a year since I first started discussing the importance of having a website which responds quickly, and many people think that speed has become even more significant in that time. You may be thinking that it's a usability issue, not something to do with your site's position in Google, but you'd be mistaken. Google has been penalising slow sites for some time. It's now important enough that you can get a report on your site's speed in Google Webmaster Tools, and in Google Analytics if you've got the feature turned on.
How does Google measure the response time on your site? You might think it's done by its little robot crawlers which index your site constantly. Wrong again. It's actually done through "real world" measurement. You know that Google Toolbar which millions of people have got installed? Every time they visit your pages, the load time is measured, and the results sent back to Google. This means that it's measuring load time from the real browsers of real users.
There are loads of ways to speed up your site. It's possible that your server itself is poor, although that's less likely nowadays. Note that the load time measured by Google will not just be the time taken to see all of the visible elements of the page, but also any add-ons you might have running behind the scenes. So it might be time to ask yourself if you really need that counter saying how many people have visited the page, or how many people are apparently on the site at the moment. All externally-served resources are going to slow down your page. Take a look at the source code behind your page. If it goes on for screen after screen, even when the page has only a few words and pictures, then it might be time to get a new, more efficient content management system.