Apache kept timing out while the server sat idle

TL;DR4 steps

01Load average near zero and Apache timing out means the box is fine and Apache ran out of something else. On prefork, that something is worker slots: check MaxRequestWorkers in mpm_prefork.conf.
02Look at the live scoreboard with curl -s http://127.0.0.1/server-status?auto. On a WordPress host that returns the homepage until you exempt /server-status from the catch-all rewrite, and the RewriteCond has to sit directly above RewriteRule . /index.php.
03Count who is holding the slots: awk '{print $7}' access.log | sort | uniq -c | sort -rn. In my case /xmlrpc.php was the top URL, from a single /24.
04Check ParentServerConfigGeneration in that same scoreboard. 169 after 18 days means something is restarting Apache nine times a day. Mine was certbot failing to renew two certificates that had expired years ago.

Raise the worker cap to what the RAM allows, block what fills it, and stop the restart loop. Done.

A client's WordPress site had been going unreachable a few times a day for a couple of weeks. Not down, exactly. The page would spin for a minute, sometimes come back, sometimes not. The person watching the box had already done the obvious checks and sent me the summary: CPU ok, memory ok, Apache timing out.

That summary is more useful than it looks. When the machine is idle and the web server still can't answer, the machine isn't the problem. Something inside Apache has a ceiling, and the ceiling is a number in a config file.

I didn't have SSH access to this one, so the whole investigation ran as a relay: I'd send a batch of commands, the client would paste the output back. It works fine, as long as you ask for everything you'll need in one go.

Eight

The first batch confirmed the idle part. Uptime 18 days, load average 0.10, 1.2 GB available out of 2 GB, swap untouched. Apache 2.4.29 on Ubuntu, prefork MPM with mpm-itk on top. Then the MPM config:

<IfModule mpm_prefork_module>
    StartServers         2
    MinSpareServers      2
    MaxSpareServers      3
    MaxRequestWorkers    8
    MaxConnectionsPerChild 1000
</IfModule>

Eight. The entire site, plus two other virtual hosts on the same box, could serve eight connections at a time. Not eight requests per second. Eight connections, each one held by a worker process for as long as the client stays attached.

And the global config made that worse than it sounds:

Timeout 300
KeepAlive On
KeepAliveTimeout 5

Prefork hands every connection its own process, and that process does nothing else until the connection ends. A client that opens a connection and goes quiet holds a slot for up to five minutes. Eight slow clients, and the ninth visitor sits in the kernel's listen backlog until their browser gives up. The box is idle because the workers aren't computing anything. They're waiting.

Someone had tuned this down at some point, probably on a smaller instance, and nobody had looked at it since. The default is 150, which would have been wrong in the other direction on 2 GB with MySQL on the same machine.

Who was sitting in the slots

With a cap that low, the question isn't whether it fills up. It's what fills it. The access log answered in one line:

awk '{print $7}' /var/log/apache2/access.log | sort | uniq -c | sort -rn | head

   1121 *
   1094 /xmlrpc.php
    393 /
    156 /wp-content/uploads/2023/10/range-walkthrough.mp4
    121 /wp-content/uploads/2018/05/scholarship-program.pdf
    121 /wp-content/uploads/2017/08/newsletter-2017-08.pdf
    120 /wp-content/uploads/2017/07/bod-agenda.pdf

Three things, none of them visitors.

/xmlrpc.php was the most requested URL on the site, almost all of it from one AWS range in Singapore (47.128.36.0/24). That's a credential brute force. Every one of those POSTs boots WordPress and runs a password hash check, so it's also the most expensive URL on the site. Eight of them arriving together is a full house.

A scraper was downloading every PDF ever uploaded, about 120 times each, plus a 2023 range video 156 times. Big files over slow connections are the worst case for prefork, because the worker is pinned for the entire transfer.

The * line is Apache talking to itself. Those are the OPTIONS * wake-up connections the parent process sends to its children on a restart. 1,121 of them in one log file was the clue I didn't pick up on yet. More on that below.

server-status returned the WordPress homepage

Before changing anything I wanted the live scoreboard, which is what mod_status is for. It was enabled. The request went to localhost. And it came back with the site's <!doctype html> and a Google Analytics snippet.

WordPress did that. Its .htaccess sends every URL that isn't a real file or directory to index.php, and /server-status isn't a file. The rewrite runs before mod_status gets a say, WordPress sees an unknown slug, and you get a 200 with the homepage. The fix is a one-line exemption in the WordPress block:

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} !=/server-status
RewriteRule . /index.php [L]

I got this wrong on the first attempt, which is why I'm spelling it out. I put the condition right after RewriteEngine On. A RewriteCond applies only to the RewriteRule that immediately follows it, and on this site the next rule was WP Rocket's cache lookup, not the catch-all. The homepage came back again. The condition has to sit directly above the rule it's meant to guard.

The client's reasonable follow-up: isn't that exposing server-status to the internet now? It isn't. mod_status ships with Require local on that location, and that checks the real TCP peer address. I tried from outside with a forged Host: localhost and X-Forwarded-For: 127.0.0.1, and got a 403 and a 421. The .htaccess line only stops WordPress from hijacking the URL before Apache's own access check runs.

With that in place:

BusyWorkers: 1
IdleWorkers: 5
Scoreboard: _......W..___.._....
ParentServerConfigGeneration: 169

That last line is the one I'd been staring past.

169 restarts

ParentServerConfigGeneration counts graceful restarts since the parent process started. It had been up 18 days. 169 restarts is nine a day, which also explains the 1,121 self-pings in the access log.

The error log grouped them neatly:

      4 Oct 11 12:47
      4 Oct 11 03:03
      4 Oct 10 19:06
      4 Oct 10 01:15
      4 Oct 09 12:31

Four restarts, twice a day, at random times. That's the certbot timer, which runs twice daily at a randomised minute. And certbot's Apache plugin reloads Apache when it installs its challenge config and again when it removes it, so four restarts means two certificates being attempted on every run.

Certbot only attempts a renewal when a certificate is within 30 days of expiry. Twice a day, every day, for at least 18 days, means it was failing every time. certbot certificates made it embarrassing:

  Certificate Name: booking.example.com
    Expiry Date: 2024-07-21 (INVALID: EXPIRED)
  Certificate Name: dev.example.com
    Domains: example.com dev.example.com bod.example.com www.example.com
    Expiry Date: 2022-08-29 (INVALID: EXPIRED)

One certificate for a booking app that had moved to a different server two years earlier. Another from 2022 for a dev hostname and a webmail hostname that no longer existed in DNS at all. The live site had long since moved to a newer certificate that renewed fine. These two were leftovers, and every twelve hours certbot tried to renew them, failed the challenge, and gracefully restarted Apache on the way in and on the way out.

A graceful restart on prefork is supposed to be invisible. The parent tells each child to finish its current request and exit, then spawns fresh ones. On a pool of eight, with bots holding most of them, four of those in fifteen seconds is not invisible. It's a short outage, twice a day, on top of the slot exhaustion.

What changed

Everything here was done with a backup copy next to the original and a configtest before each reload.

MaxRequestWorkers  8  → 20
Timeout            300 → 60
KeepAliveTimeout   5  → 2

Twenty, not 150. Idle workers were 36 MB each and busy PHP workers run 60 to 80 MB, and MySQL lives on the same 2 GB. Twenty leaves room. The limit should come from the RAM, not from the default.

xmlrpc.php is denied at the Apache level, in a small file under conf-enabled, so it doesn't depend on a plugin or on .htaccess surviving the next permalink save. That breaks Jetpack and the WordPress mobile app, which this site doesn't use. If yours does, rate-limit it instead.

The four dead virtual hosts were disabled with a2dissite, and only then the two dead certificate lineages were deleted with certbot delete --cert-name. That order matters: the vhosts referenced the expired certificate files, and deleting the files first would have left Apache unable to pass its next config test. A certbot renew --dry-run afterwards came back clean for the three certificates that are actually in use.

The site went from a few timeouts a day to none, and certbot now touches Apache about once every two months, when it has a reason to.

Takeaway

"CPU ok, memory ok" rules out the machine. It doesn't rule out Apache, because prefork's capacity isn't a function of the hardware. It's a number in mpm_prefork.conf, and an idle box with a full worker pool looks exactly like a healthy one from the outside. Get the scoreboard working before anything else, and read the generation counter while you're there. A server that restarts itself nine times a day will tell you, but only if you ask.

Got a server that's idle and unreachable at the same time? Tell me what's going on and I'll take a look.