Commavee

This is a new feature that I hope you enjoy - Tech Tip of the Day. With a technology background that dates back to just *before* the introduction of the IBM PC, I have had the pleasure of working with some cool, world-changing systems, software, and gear as both my business computing career and the Business Computing Industry have grown up together.

Today’s Topic: Apache 2.x and Mutexes

Background

There are good reasons why the Apache http server is widely recognized as the most popular web server on the planet: flexible configuration options, cross-platform compatibility, a user-friendly license, and zero dollar cost. Together, these factors compel many to embrace Apache implementations.

I’ll dive into history in another post, but suffice it to say that Apache 1.3x and Apache 2.x are very different in many, substantive ways. In extremely simplistic terms, there may not exist enough compelling reasons for a shop to migrate from a fully functional, well understood & supportable Apache 1.3x implementation to an Apache 2.x implementation.

Freepository’s servers have run Apache 1.3x for a looooong time. We understood it extremely well, and could support it 24 hours per day from anywhere on the globe. Recent business development decisions have resulted in the very technology-laden decision to upgrade to Apache 2.x. We researched this, modeled it on a development box and decided this was a low-risk operation and we proceeded to upgrade our servers.

The upgrade itself went smoothly; no surprise there. But then we hit a dead stop. Part of our hosted service necessitates frequently restarting Apache; after just a few of these restarts, on the next attempt Apache failed to restart.

Mutex Handling

If you upgrade any of your Apache servers to the 2.x release, be alert to a nasty mutex issue that shows up when mod_ssl (and probably mod_python) are used. Briefly, the symptom is that at startup the server will allocate one or more mutexes (mutual exclusion lock files) to manage IPC inside the server. In the 2.0 codeline, the default behavior is broken when SSL is dynamically loaded at run time.

At a restart the mutexes are not released. The default has the mutexes existing in reserved memory space, and after enough restarts of the Apache server, this memory space fills up and Apache will not start. There is an ominous message in the log about ‘Out of disk space’. This message is bogus (it would be accurate only for disk-based mutex files that haven’t been used for a while). The error *does* mean that you must clear those mutexes before restarting Apache.

You can do this manually (which is tedious but effective), or you can simply reboot the physical server. Rebooting clears the memory (of course), but if you don’t fix your Apache installation the issue is sure to recur (it did for me twice before I figured out exactly what was happening and how to fix it).

If this occurs, you can quickly check for the mutexes by running ‘ipcs’ (checks for shared memory segments). The output here should show a reasonable number of mutexes owned by the user under which Apache runs. “Reasonable” could be one or thirty; page after page is unreasonable. If you wish to manually clear these (ie. w/o rebooting the physical server), then run the ‘ipcrm’ command with the ‘-s’ switch plus the semaphore ID for each one. Here’s an example:

If I wanted to get rid of these, I would construct a command line like this:

#ipcrm -s 65536 -s 98305 -s 131074 <Enter>

You can pass as many ‘-s semid’ pairs to the command line as you wish. Once these are cleared out (run ‘ipcs’ again to check), you will be able to start Apache again. Be sure you have disabled the mutex handling, though (see below). The OS really can handle this adequately, so don’t worry. If /when it gets fixed in Apache, you can re-enable it. Here’s the code in your httpd.conf or ssl.conf file:

# Semaphore:
# Configure the path to the mutual exclusion semaphore the
# SSL engine uses internally for inter-process synchronization.
SSLMutex none

You can also add the following workaround to your /etc/init.d/httpd script so that the mutexes are automatically cleared when you stop httpd (courtesy of David Fraser):

stop() {
echo -n $"Stopping $prog: "
killproc $httpd
RETVAL=$?
removeipcs=`ipcs -s -c | grep nobody | cut -d ' ' -f 1`
numipcs=`echo $removeipcs | wc -w`
if [ $numipcs != 0 ]
then
echo
echo -n $”and removing $numipcs ipcs: ”
echo $removeipcs | xargs -n 1 ipcrm -s
sleep 2
fi
echo
[ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}

This assumes that the apache is running as user ‘nobody’ if not, adjust as required.

Tags: freepository, apache, mutex, tech-tip