A dedicated server will likely cost you more than $100 per month; it's definitely not cheap web hosting. The benefit? Your website lives on a server all by its lonesome, so it takes advantage of the server's full resources. You'll probably need to handle firewalls, updates, and maintenance yourself, however, unless you opt for a managed server, which costs even more money.
All the aforementioned features are valuable parts of the web hosting experience, but none matches the importance of site uptime. If your site is down, new clients or customers will be unable to find you or access your products or services. As if that weren't bad enough, regular customers may lose confidence in your service if they can't reach your site. Or they may simply need access to a service or widget today, which means they might go to a competitor to get the job done—and they might never come back. Anyone who's ever tried to build customer loyalty understands what a catastrophe this is.
You can also purchase Secure Sockets Layer certificates (starting at $69.99 per year), the software that's represented by a green padlock in your web browser when you visit, say, a financial institution's website. If you plan to sell products via your website, consider an SSL certificate an essential purchase, as it safeguards the data exchange between a customer and your servers.