There are at least three major factors which determine how quickly your website loads.
Web host servers (especially good ones) can host many many websites on the one server. However this directly impacts the performance of each website on the server. It is very unlikely that your website is the only one on the web host, as it costs $thousands/year to run a dedicated server.
Our investigative A.R.&R. for SEO can show you some of the other websites hosted on your server. And we will never overload our own servers - after all, many of our hosting clients are running websites we've developed for them and that we perform SEO for. Since hosting speed is an important SEO factor, you can be assured we will always be conservative with our server load.
The bandwidth, network speed, carrier, and redundancy of the internet connection for your web host directly impacts the speed of your website. The details are many and technical, but suffice to say: Sometimes you only get what you pay for.
Bandwidth in Australia is comparatively expensive, so extremely cheap hosting plans may make use of less-than premium bandwidth. Whatever the case, we can run a website speed test from a neutral third party and compare your current web host with ours. The results are often surprising.
Server hardware continually needs replacing. This is true not just for the processor itself, but all components. Is the hard disk a RAID array? How old are the routers/switches/etc/etc.
Whatever the case, a simple website speed test comparison will give you the facts you need.