Pick our brains...

Apache Httpd is Running But Unresponsive, Website is Down, The Scoreboard is Full!

Here’s an interesting one:

Your server suddenly goes unresponsive, and your alerting service awakens the response team at 2am. Upon logging into the server, httpd appears to indeed be running, load is nil, mysqld is fine, but your website is still down. What’s up with that?!

In the error log, you see lots of repeated entries such as:

[Thu Jun 11 05:26:03.752629 2015] [mpm_event:error] [pid 1979:tid 139920401913792] AH00485: scoreboard is full, not at MaxRequestWorkers
[Thu Jun 11 05:26:04.753748 2015] [mpm_event:error] [pid 1979:tid 139920401913792] AH00485: scoreboard is full, not at MaxRequestWorkers
[Thu Jun 11 05:26:05.754868 2015] [mpm_event:error] [pid 1979:tid 139920401913792] AH00485: scoreboard is full, not at MaxRequestWorkers
[Thu Jun 11 05:26:06.755984 2015] [mpm_event:error] [pid 1979:tid 139920401913792] AH00485: scoreboard is full, not at MaxRequestWorkers

The above error message indicates that the apache scoreboard was full. The apache scoreboard keeps track of the number of allowed connections and the current state of each ongoing connections.

When the scoreboard is full, new connections are added to a queue. Connections in the queue are not served until the scoreboard has an open slot. And so it may appear that apache is down, when in fact it is at its configured capacity.

The fix? Restart the apache httpd process, of course!

service httpd {start | stop | restart}

Which is an alias of:

/bin/systemctl {start | stop | restart} httpd.service

Note that the "{" and "}" are not meant literally. They encase some of the available commands, from which you should choose one.

Or, on some servers:

/etc/init.d/httpd stop
/etc/init.d/httpd start

And boom, your website is living and breathing again.