Optimize a Website for Speed, Security, and Easy Management

optimized server setup Over the past 10 or so years I've been directly or indirectly involved in configuring/administrating/hacking thousands of websites, and I realized today that I've actually learned quite a bit about how to really make them work hard for me, instead of the other way around. It came as a mild shock to think of where I was back then vs. now because the improvements and optimizations are hundreds of smaller improvements, but taken together, the optimization hacks I've found through trial and error and much reading are as Donald would say, YOOUUGE compared to a basic website setup.

I use this awesome skeleton setup for all my high-paying clients sorry poor people! and also of course on this blog, which I use as a bleeding-edge dev server for my crazy testing. So realize that I'm already past this setup and using it to do cooler stuff. In order for you to use these more advanced ideas, you first need to get up to speed on what I'm doing so you know what I'm talking about. This article tries to help you accomplish that... remains to be seen.

An Optimized Website, The Real Deal

This first article is to give you some ideas and get you thinking and reading before the first article in this series comes out. This series details how to setup, configure, secure, optimize, and manage a website the best possible way I can come up with. It pieces together all the AskApache hacks and tricks and uses methods and ideas discussed all over this blog and all over the net and glues them all together to show you how to have the most optimized, fastest, best website setup I can think of.

Knowing the why and how behind the operation of a Web Server allows us to optimize that operation. For this example we will be creating the website www.askapache.com, which will be running WordPress and php. We will also set up s.askapache.net to serve all of our sites uploads, images, css and javascript files, flash files, etc. with advanced caching and security using Apache Server .htaccess files. So lets get started and take a look at this site structure for a moment.

/web/askapache.com
|-- /web/askapache.com/backups/
|-- /web/askapache.com/public_html/
|-- /web/askapache.com/inc/
|-- /web/askapache.com/logs/
|-- /web/askapache.com/static/
|-- /web/askapache.com/tmp/
|-- /web/askapache.com/.htpasswd-basic
`-- /web/askapache.com/.htpasswd-digest

/backups/ - For encrypted backups of WordPress database and site files. And any other backups.
/public_html/ - The document root for www.askapache.com
/inc/ - Folder to keep your php include files for extra security and easy management.
/logs/ - Save your php, apache, and other logs here or create symlinks to them.
/static/ - The document root for s.askapache.net
/tmp/ - Only need this if your host doesn't already have a /tmp folder

Strong Security, Top to Bottom

Simply by implementing correct access permissions, file permissions, password protection and segmenting various folders and services we are already ahead of the game. I've always taken security extremely seriously, so you can benefit from alot of the simple solutions I'm recommending for a really locked down site.

Indeed, security is a major part of every step of this setup process, as security concerns are what drives a lot of the motivations I have for coming up with this setup in the first place. We will be doing very simple but very effective site security like the following items, which is a short list compared to everything we will be doing.

Fixing file permissions automatically
Searching for modified files on the server
Encrypting your backups
Get alerted to breakin attempts
Block tons of bad clients
Disallowing cgi scripts or any other handlers, just serve files.
Configuring PHP
Password Protection for certain areas

Ready for Warfare?

My past work for an Internet Service Provider, followed by 4 years of auditing the security of organizations external/internal networks has given me a fresh perspective on website security, and I think it allows me to see what would really be effective at preventing and killing attacks. In fact just last night I was once again doing some research into some off-the-wall security topics, and I discovered a new defense method that I will be writing about very soon. I believe that this new method, could be quickly adopted and implemented by hosting providers and software developers, which would result in us finally taking the Internet back from all those zombies and robots. This method will be discussed in great detail soon, and will be a core part of this site setups security and optimization.

Built to Bleed Speed

Serve's files as fast and efficiently as possible using advanced caching, HTTP Protocols, php/server configurations.

Many of the articles and research on this blog is about improving the speed and efficiency of your website. In fact that is why I am helping develop open-source software to block spammers from WordPress blogs... not because I'm bothered by the spam, but because they make the net slow! So lets look at some of the ideas we'll be implementing.

Many techniques I've been using and tweaking for several years, and recently many of them were included in the high-performance websites list. Of course we will be taking a look at this list in practical terms, meaning almost all of it, the caching, compression, etc., will be automated in keeping with our "comfort" goal, which is to say we want to make the Web Developer and Server Admin's lives as easy and comfy as possible. After all, we do the work right?

Reduce HTTP requests - Reducing 304 requests with Cache-Control Headers
Use a customized php.ini - Creating and using a custom PHP.ini
Add an Expires header - Caching with mod_expires on Apache
Gzip components
Make CSS and unobtrusive Javascript as external files not inline
Reduce DNS lookups - Use Static IP address, use a subdomain with static IP address for static content.
Minimize Javascript - Refactor the code, compress with dojo
Avoid external redirects - Use internal redirection with mod_rewrite, The correct way to redirect with 301
Turn off ETags - Prevent Caching with htaccess
Make AJAX cacheable and small

AskApache.com, Fastest Site Ever!

Ok it might not be the #1, but surely the top 10.. ;)

I'm very proud of the performance I am able to achieve on this site. Very proud. I started looking for ways to improve the wp-cache and wp-super-cache WordPress plugins, and came up with hacks for both of them.. but they still didn't do what I wanted so I started from scratch and wrote my own caching plugin.

With much more advanced caching options and unquestionably higher performance and lower time usage on the machine. I'm hesitant to release it to the public until I get faded on it.. I just really love it.. it has been running my site for several months now and I keep finding ways to improve it.. Stay tuned.

One feature it has is the ability to allow negotiation of a resource between apache and the client. Think almost transparent mutli-lingual blogs, mutliple formats per document (look at the rdf for this page for an example*). But that plugin is the future and this is the present.. so back to it we go.

Pamper the Webmaster with Extreme Comfort

This section alone would make this setup appealing. I have developed all types of techniques and methods to make my life as easy as possible. I could literally DIE right now and this blog would continue to run and operate for years on its own. The general philosophy that I have used to get to where I can goto the beach with my laptop and do all this crazy stuff is the idea of perfection. That may sound a little put-offish, but it basically means I will focus in on one very specific area for improvement or research and just get sick with it. Most of this blogs articles are enlightening examples of this in action. I will take a relatively unknown or unused piece of code or software and experiment with it until I feel I have it down, then I move on to the next item of never-ending research. Mostly I think this is just plain habit from when I was studying security. I'm much better at this then that :)

Apache ErrorDocuments

The 57 HTTP Status Codes and Apache ErrorDocuments article is a prime example. I was simply searching for an authoritative list of HTTP status codes, an issue not many web people find worth their time, and that search led to some of the most useful stuff I've found about the Web

This "Comfort" article will include multi-language, intelligent, and optimized error documents for handling any type of HTTP error with class and allow us to stop spammers, save bandwidth, redirect correctly, etc.. You will probably be surprised at all the uses an Apache ErrorDocument can have.. It IS one of the foundations of the HTTP-based Net after all.

Emphasis on Easy Upgrades

The whole setup is geared to make hassle-free WordPress/PHP/application upgrades possible by keeping different types of files in separate places, keeping backups, other misc tricks and since all of these files are in /web/askapache.com, your FTP connection can still access every file easily. Sometimes security and optimizing your server can lead to it being more of a pain to do updates, backups, and general maintenance. This article tries to overwhelm the balance with a trifecta of goals.

Move to a new host? Ok!

Comfort to me also means being able to pack up the whole website and database and move to another web host in under an hour. I can move the whole AskApache site to one of several other hosting providers accounts I have in about 30minutes. If this was a clients site or I was getting paid more, I'd also be focused on round-robin DNS technology, balance-load setups, and just go crazy making it fast.

Staying Online, Improving Uptime

Ever since I started sharing information and software to stop all these resource hogging zombies attacking everything I've been attacked several times. Normally I get over 10K exploit attempts or requests per day, which I pretty much block 100%. But a few times they've actually tried to DDOS me off the net in a distributed attack. I have implemented several "poor mans" techniques to put up your best effort at surviving, which I did. Basically you want to configure your server to KILL connections just as fast as possible and prevent your server resources from skyrocketing and surpassing your quotas. A skilled attacker could easily shut you down even without the use of a widespread botnet if they are clever, which could be devastating to your small blog or site if it goes down at a crucial instant.

Organization with Templates and Systems

I used to work with a guy who did alot of the coldfusion programming for us, and I used to cringe every time I was called in to upgrade a site or do a re-design. Files and folders EVERYWHERE! Literally images in every folder, multiple index.html, index1.html, index-old.html, and on and on it went.. It would take me hours just to reverse-engineer the site enough so I could modify files on it without having some unkown consequence happen.

Do You Have a Cluttered Desktop?

Everyone has this problem, what I do all the time is just grab everything on my desktop and put it in a folder named with the date. Then the process repeats itself and invariably a few months later I'm looking at a cluttered screen again.

This absolutely is the worst thing that can happen to a website, worst for security, comfort for webmaster, and speed. So this setup addresses that issue completely heads on. With all the different pages, tools, and resources available on this blog, I can almost promise you that my site has less files than yours. No small feat to be sure, but worth every second I spent researching how to do it now that its on and popping.

What's a Website really?

All hosts are different, but any host worth their salt is running some kind of BSD/Linux operating system, and that is good news because those operating systems all use very similarly excellent file/folder structures with huge organization systems. If your web hosting provider is running on a Windows based operating system or other locked/proprietary software than this article is not for you and I would recommend switching hosts to a BSD/Linux open-source operating system.

Listening for Requests with Web Hosting and DNS

First you set your website up so it can start serving.

You buy your domain name, which just gives you the right to use it.
You pay your webhost for an account on their machine running a Server connected to the Net via a fast connection link.
You pay a DNS provider to redirect requests for your domain name to be sent to your webhosts machine running the server.

Sub-Domain for Serving Assets

This is a very cool method I've been using more and more frequently because it makes updates, upgrades, and changes so much easier to manage. And segmenting various parts of the site is smart security, and even smarter in the way of speeding up a website and keeping your servers running mean and lean.

Full Site Structure Expanded

/web/askapache.com
|-- /web/askapache.com/backups/
|-- /web/askapache.com/public_html/
|   |-- /web/askapache.com/public_html/about/
|   |-- /web/askapache.com/public_html/admin/
|   |-- /web/askapache.com/public_html/cgi-bin/
|   |-- /web/askapache.com/public_html/.htaccess
|   |-- /web/askapache.com/public_html/index.php
|   `-- /web/askapache.com/public_html/robots.txt
|-- /web/askapache.com/inc/
|   |-- /web/askapache.com/inc/config.inc.php
|   `-- /web/askapache.com/inc/settings.inc.php
|-- /web/askapache.com/logs/
|   |-- /web/askapache.com/logs/access.log
|   |-- /web/askapache.com/logs/error.log
|   |-- /web/askapache.com/logs/logins.log
|   |-- /web/askapache.com/logs/modsec_audit.log
|   |-- /web/askapache.com/logs/modsec_debug.log
|   `-- /web/askapache.com/logs/php_error.log
|-- /web/askapache.com/static/
|   |-- /web/askapache.com/static/css/
|   |-- /web/askapache.com/static/flv/
|   |-- /web/askapache.com/static/img/
|   |-- /web/askapache.com/static/js/
|   |-- /web/askapache.com/static/mp3/
|   |-- /web/askapache.com/static/pdf/
|   |-- /web/askapache.com/static/swf/
|   |-- /web/askapache.com/static/.htaccess
|   |-- /web/askapache.com/static/index.html
|   `-- /web/askapache.com/static/robots.txt
|-- /web/askapache.com/tmp/
|-- /web/askapache.com/.htpasswd-basic
`-- /web/askapache.com/.htpasswd-digest

Full Expanded Structure

/web/askapache.com
|-- /web/askapache.com/backups/
|-- /web/askapache.com/public_html/
|   |-- /web/askapache.com/public_html/about/
|   |   `-- /web/askapache.com/public_html/about/index.html
|   |-- /web/askapache.com/public_html/admin/
|   |   |-- /web/askapache.com/public_html/admin/.htaccess
|   |   `-- /web/askapache.com/public_html/admin/index.html
|   |-- /web/askapache.com/public_html/cgi-bin/
|   |   |-- /web/askapache.com/public_html/cgi-bin/bin/
|   |   |   |-- /web/askapache.com/public_html/cgi-bin/bin/.htaccess
|   |   |   |-- /web/askapache.com/public_html/cgi-bin/bin/php.cgi*
|   |   |   |-- /web/askapache.com/public_html/cgi-bin/bin/php.ini
|   |   |   |-- /web/askapache.com/public_html/cgi-bin/bin/php4.cgi*
|   |   |   `-- /web/askapache.com/public_html/cgi-bin/bin/php5.cgi*
|   |   |-- /web/askapache.com/public_html/cgi-bin/private/
|   |   |   |-- /web/askapache.com/public_html/cgi-bin/private/.htaccess
|   |   |   |-- /web/askapache.com/public_html/cgi-bin/private/debug.php
|   |   |   `-- /web/askapache.com/public_html/cgi-bin/private/stats.php
|   |   |-- /web/askapache.com/public_html/cgi-bin/.htaccess
|   |   |-- /web/askapache.com/public_html/cgi-bin/login.php
|   |   |-- /web/askapache.com/public_html/cgi-bin/printenv.cgi*

|   |   `-- /web/askapache.com/public_html/cgi-bin/redir.cgi*
|   |-- /web/askapache.com/public_html/.htaccess
|   |-- /web/askapache.com/public_html/index.php
|   `-- /web/askapache.com/public_html/robots.txt
|-- /web/askapache.com/inc/
|   |-- /web/askapache.com/inc/config.php
|   `-- /web/askapache.com/inc/functions.php
|-- /web/askapache.com/logs/
|   |-- /web/askapache.com/logs/access.log
|   |-- /web/askapache.com/logs/error.log
|   |-- /web/askapache.com/logs/logins.log
|   |-- /web/askapache.com/logs/modsec_audit.log
|   |-- /web/askapache.com/logs/modsec_debug.log
|   `-- /web/askapache.com/logs/php_error.log
|-- /web/askapache.com/static/
|   |-- /web/askapache.com/static/css/
|   |   `-- /web/askapache.com/static/css/apache.css
|   |-- /web/askapache.com/static/flv/
|   |   `-- /web/askapache.com/static/flv/apache.flv
|   |-- /web/askapache.com/static/img/
|   |   |-- /web/askapache.com/static/img/apache.gif
|   |   |-- /web/askapache.com/static/img/apache.jpg
|   |   `-- /web/askapache.com/static/img/apache.png
|   |-- /web/askapache.com/static/js/
|   |   `-- /web/askapache.com/static/js/apache.js
|   |-- /web/askapache.com/static/mp3/
|   |   `-- /web/askapache.com/static/mp3/apache.mp3
|   |-- /web/askapache.com/static/pdf/
|   |   `-- /web/askapache.com/static/pdf/apache.pdf
|   |-- /web/askapache.com/static/swf/
|   |   `-- /web/askapache.com/static/swf/apache.swf
|   |-- /web/askapache.com/static/.htaccess
|   |-- /web/askapache.com/static/index.html
|   `-- /web/askapache.com/static/robots.txt
|-- /web/askapache.com/tmp/
|-- /web/askapache.com/.htpasswd-basic
`-- /web/askapache.com/.htpasswd-digest

Merchant Account Services

If you want to make it easier for your customers to shop at your site, check out merchant account services from Network Solutions. Services like these can help bring credibility and security to your online business.

Apache is Open-Source

The buzz about apache and open-source is very real, apache is becoming more of a discussed topic as people realize the power and importance of Doing it and Doing it and Doing it well. - Movable Type Apache Installs made easy, Checking out Apache Web logs, Securing Wordpress with .htaccess, WordPress Permalinks and .htaccess, New search and replace module for apache!, creating an .htaccess template, .htaccess allow directive

Check back in a week for the first article, or better yet subscribe to my rss feed or use the comment form below to get notified.

Hosting Apache HTTP Server Caching Htaccess HTTP linux operating systems round-robin DNS technology Web Developer Web server 18 Feb, 201018 Feb, 2010

« Bayonetta TrailerPortaPutty Auto-Reconnecting SSH Tunnels on an Encrypted TrueCrypt Portable USB Key w GPG »