12 Ago 11 Apache: [error] (28)No space left on device: Cannot create SSLMutex

When you work as a horizontal support tech consultant for a very big company you may have to do with people that has basic linux/opensource knowledge and commercial system stuff (win, unix) skills. You may even have to do (… !?!?) with very prepared people (…).

Someone here takes care of apache installations and apache itself is very robust and stable so we all can sleep quietly, the problem comes out when someone with less OS system skills fires “kill -9” on apache processes to stop it.

[Thu Aug 11 17:47:01 2011] [error] (28)No space left on device: Cannot create SSLMutex
Configuration Failed

And apache does no longer come up and running.

The trouble may be weird because one could read “No space left on device” without reading the real error message: “Cannot create SSLMutex”.
But the problem is really easy to understand if you try to figure out what is going on at syscall OS level when apache starts up. This can be done with “strace” command on linux (“truss” or “tusc” on other expensive unix environments…).

The problem is caused by IPC SysV semaphores still standing up on the system from the previous apache kill.

The definitive solution here is to stop apache with “apachectl” command or calling “kill -15” (and not kill -9 !!!!) on the apache father process. This way you are instructing apache to stop gracefully, the father kills his childs and cleanup semaphores and the like, the clean way.

To solve your problem you have to cleanup hanging semaphores. You could reboot linux, but this is avoided on any serious environment, so which is the magic command?

If apache runs with “apache” user you can call this command to clean up semaphores created by the “apache” user:

ipcs -s | grep apache | perl -e ‘while (<STDIN>) {@a=split(/\s+/); print `ipcrm sem $a[1]`}’

You could do “ipcs -s | grep apache” to see the semaphores first, and then call ipcrm on each to clean it up.
Try to start apache now and the problem would solve.

The other way is to change the apache serialization mechanism from semaphores to pthread mutexes or fcntl. To do so you have to:

1) set “AcceptMutex fcntl” on httpd.conf
2) set “SSLMutex pthread” on httpd-ssl.conf

Hope this help someone… 🙂

Ciao, Dino.