At Slicehost, users occasionally enter our chat to complain that their Slice has suddenly come to a crawl. In nearly every occurrence of this behavior the underlying cause of the slowdown is consistent swapping. And in nearly every occurrence of consistent swapping, the culprit is Apache.

What is Swap?

Note: an oversimplified example ahead!

To understand this problem let’s first examine swap, which is a partition of the hard drive used when memory is full.

Let’s say we have 5 processes each using 50MB of RAM, and assume we have only 256MB of RAM in our machine. That means we’re using 250 of the total 256MB of RAM. If we open another process that will use another 50MB of RAM, the operating system will swap one of those processes’ memory space into the swap space to make room for the new process.

Swap is good in this case, because it allows us to start a process without having to kill another one to make room. Occasional swapping is expected, but consistent swapping causes slowdowns.

According to Wikipedia, swap is “typically many orders of magnitude slower than RAM”. If you’re using a processes whose memory space is in swap, that process will be slowed. In other words, the system has to use the disk as memory, therefore performance degrades substantially.

Consistent swap is the real problem

How can one tell if they’re swapping consistently? The free command shows only how much swap space is used. Instead, we rely on vmstat. Run vmstat 1 10 which polls at a 1-second interval for ten iterations.

Your output should be similar to:

$ vmstat 1 10
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0     64  22072  46852  89620    0    0     1     3    4    7  0  0 100  0
 0  0     64  22080  46852  89620    0    0     0     0   37   30  0  0 100  0
 0  0     64  22080  46852  89620    0    0     0     0   17   22  0  0 100  0
 0  0     64  22080  46852  89620    0    0     0     0   19   31  0  0 100  0
 ...snip...

The important columns are si and so, which mean swap operations in and out, respectively. Observe that this example has exactly zero swap operations. A few operations here and there are typically fine. However if you’re at 50 for each iteration, that means you’re performing 50 swap operations per second.

How does this happen?

Our users typically use the default Apache configuration which is tuned for higher RAM overhead. An increase in traffic causes Apache to spawn more processes, each consuming a chunk of RAM. We commonly see users with upwards of 30 Apache processes consuming anywhere from 20-80MB of memory, at which point we determine that the user is swapping consistently.

Fixes

There are many potential fixes for this problem. One is to simply reduce the amount of processes Apache may spawn, as described in this thread.

Another is to use a different web server entirely, the two most popular alternatives being Nginx and Lighttpd. These are much lighter-weight alternatives to Apache boasting better performance and a smaller footprint.

Another method of reducing load is caching. Wordpress, which is commonly installed on our Slices, is inefficient—a fact made more prevalent with its popularity. Installation of WP Super Cache decreases load dramatically in both the application and database.

The method used for my websites (developed in Django) is a hybrid of these three methods. I use Nginx to serve up all static files, and proxy back to Apache only when a file does not exist. I use StaticGenerator as a proactive cache mechanism. This significantly reduces the load on Apache, yet still allows for usage of its excellent modules, like mod_php and mod_wsgi.

Nevertheless, Apache is not the only cause of this issue. Poor code, too many processes, too heavy reliance on a memory-intensive process, low memory overhead—the list goes on.

Whatever the case, it is worth finding and fixing the cause of the issue. Too often a user will simply upgrade their Slice instead. We will gladly accept more money, but this is only a short-term fix; The problem is likely to crop up again soon.

Deploying a web application with even a small amount of logic can overwhelm an improperly-tuned server, and even a small amount of shoddy code can overwhelm a well-tuned server. It is always important to find the best fix, not the quickest.

Yet more importantly, one should be proactive in efficiency. Deploying inefficient code in production—or using an inefficient production environment—is recipe for disaster.

Ram photo used under Creative Commons License