ddz.dev LLC

Symfony 8 ignores your #[Groups] attribute

2026-07-23T00:00:00.000Z

I upgraded a client's Symfony app to Symfony 8 recently. The upgrade itself went smoothly: dependencies resolved, the app booted, the test suite was green. A few days later a bug report came in that the invoice history was broken. Customers opened a PDF of an old invoice and got the letterhead, the logo, the table header, and then nothing. No line items, no totals. A beautifully rendered blank invoice.

The PDF is drawn client-side from a JSON endpoint, so I checked the API response first. The endpoint was returning HTTP 200 and a perfectly well-formed payload:

{
    "contents": [[], [], [], []]
}

Four line items, and every single one of them serialized as an empty object. The entity was fine. The database rows were fine. The controller was fine. The serializer just looked at each object, found nothing it considered worth serializing, and cheerfully returned {}.

The import that lies to you

The line items were annotated the way Symfony code has been annotated for years:

use Symfony\Component\Serializer\Annotation\Groups;

class InvoiceLineItem
{
    #[Groups(['invoice'])]
    private string $description;
}

Spot the bug? Neither did I, for longer than I'd like to admit. The #[Groups] attribute is right there. It compiles. Static analysis was quiet. The app had shipped hundreds of invoices with exactly this code.

The problem is the namespace. Symfony moved serializer attributes from Serializer\Annotation to Serializer\Attribute back in 6.4, and kept the old namespace around as a deprecation shim. Symfony 8 pulled the plug, and it did it in a particularly sneaky way. The shim file still exists, but the class inside it is declared like this:

if (false) {
    class Groups extends \Symfony\Component\Serializer\Attribute\Groups
    {
    }
}

The if (false) block means IDEs and static analysis tools still see a class there, but at runtime the class simply does not exist. Loading the file pulls in the new Attribute\Groups and nothing else.

Why PHP never says a word

Here's the part that turns a removed class into a silent failure instead of a fatal error. PHP attributes are lazy. #[Groups(['invoice'])] isn't executed when the file loads; it's stored as metadata, a class name and some arguments, and nothing tries to resolve that class name until someone calls reflection and asks for an instance.

Symfony's attribute loader walks the reflection metadata looking for its attribute classes, the ones in Serializer\Attribute. Your attribute names a class from Serializer\Annotation, which is not on that list and doesn't even exist. So the loader does the only reasonable thing it can do with an attribute it doesn't recognize: it skips it. That's correct behavior, by the way. Your entity might carry attributes from Doctrine, from a validation library, from your own code. The serializer can't throw on every attribute that isn't meant for it.

The result is a perfect storm of nobody-is-wrong. PHP is right not to error on an unresolved attribute class. The serializer is right to ignore attributes it doesn't own. The shim is right to exist so old code doesn't fatal on load. And you're left with an entity whose serialization groups have quietly evaporated, so normalize($item, null, ['groups' => 'invoice']) matches zero properties and hands you an empty object.

No exception. No deprecation notice, because the deprecation period is over. Not even a log line.

The fix

One line per file:

use Symfony\Component\Serializer\Attribute\Groups;

That's it. The attribute usage stays identical, only the import changes. Find every affected file with:

grep -rn 'Serializer\\Annotation' src/

Fix them all in one pass, because this fails per-class: one entity on the new namespace and one on the old gives you a response where half the graph is populated and half is empty objects, which is even more confusing to debug. Then clear the cache, since serializer metadata is cached and will happily keep serving the old, empty mapping:

php bin/console cache:clear

The invoices came back immediately, line items and all.

The rest of the minefield

Groups is not the only class that moved. The same shim treatment applies to the whole old namespace: SerializedName, Ignore, Context, MaxDepth, DiscriminatorMap. Every one of them will compile from the old import and do absolutely nothing. Ignore is the scary one, since a silently dropped #[Ignore] doesn't remove data from your response, it adds data you meant to hide. Empty invoices are embarrassing; leaking a field you explicitly marked as ignored is a different category of morning.

The same annotation-to-attribute migration happened in Doctrine and in the Validator component, so if your upgrade path skipped a major version or two, grep for those old namespaces while you're at it.

After I published this, a commenter on Reddit suggested Rector would have caught it, and the official upgrade guide does tip Rector right in the fix-your-deprecations step. That stung enough that I went and tested it: a scratch project with an entity reproducing the exact broken pattern (attribute syntax, imports from the old namespace) and every Symfony upgrade set from 6.0 through 8.1, plus the annotations-to-attributes set.

Zero changes. Rector reported the file clean.

The packaged sets convert docblock annotations to attributes. Code that's already on attribute syntax with the old imports sails through untouched. Rector can fix this, but only if you point it at the problem yourself:

use Rector\Renaming\Rector\Name\RenameClassRector;

->withConfiguredRule(RenameClassRector::class, [
    'Symfony\Component\Serializer\Annotation\Groups' => 'Symfony\Component\Serializer\Attribute\Groups',
    // ...same for Ignore, SerializedName, Context, MaxDepth, DiscriminatorMap
])

That works; I watched it rewrite my test entity. So the honest summary is worse than I thought when I started writing this section: static analysis is blind to this by design, and the refactoring tool everyone recommends passes it by default. The one command that reliably shows the truth is bin/console debug:serializer App\Entity\Whatever, which prints the metadata the serializer actually sees. My missing groups would have been plainly visible there.

Takeaway

After a Symfony major upgrade, green tests and a booting app don't prove your attributes are still being read. Anything that configures behavior through attributes can degrade into a no-op instead of an error, because unknown attributes are skipped by design. Grep for the old namespaces before you trust the output, and be suspicious of any API response that's the right shape but strangely empty. The serializer isn't broken. It's doing exactly what your imports told it to do, which is nothing.

Upgrading a Symfony app and hitting weirdness like this? Send me the details and I'll take a look.

The database was fine. The cloud around it wasn't.

2026-07-13T00:00:00.000Z

I deployed an application against Azure Database for PostgreSQL Flexible Server recently, and the database itself was never the problem. PostgreSQL was up, healthy, and happily accepting connections the entire time. The cloud wrapped around it was another story, and it cost me the better part of a day.

Every provider makes the same promise: "it's managed." What the promise quietly leaves out is the networking, the DNS, the TLS defaults, the firewall rules, and the three different portal blades you'll need to understand before your app gets to say hello to its own database.

The usual checklist ran out fast

The app kept failing to connect. Nothing exotic, just a timeout. The connection string looked fine: host, user, dbname, password, all correct. So I ran through the standard suspects. Firewall rules? Fine. Credentials? Correct. Is the server actually running? Yes, Azure swears it is.

That's normally where this kind of story ends. Someone fat-fingered a password, you fix it, you move on. Not this time. The checklist ran out and the timeout stayed.

The hostname that didn't exist

The server was configured for private access, which sounds like the simple option. It isn't. Azure doesn't just hand your database a private IP and call it a day. It expects you to understand Azure Private DNS Zones, because the server's hostname only resolves through a zone like privatelink.postgres.database.azure.com, and that zone has to be explicitly linked to your virtual network.

If the link is missing, the hostname doesn't resolve to a wrong address or a public one. It simply doesn't exist:

could not translate host name

At that point you're not debugging PostgreSQL anymore. You're debugging Azure's networking model, from inside an application that can only tell you "cannot connect to database." The fix was linking the Private DNS Zone to the VNet the app actually runs in. One checkbox, buried two portals away from anything labeled "PostgreSQL."

Thirty minutes for one line

With DNS sorted, the connection still failed, because Azure requires TLS and my client wasn't offering it. This is where PostgreSQL client libraries get to have opinions. Some try SSL first and fall back. Some don't try at all. Some negotiate, some fail on the spot. The server isn't broken and the client isn't broken; their defaults just don't agree.

The entire fix was one parameter:

sslmode=require

One line. Thirty minutes of investigation. That ratio is the signature of cloud debugging.

"Managed" means the complexity moved

Here's the part I actually find interesting. People assume a managed service means less complexity. It usually means the complexity relocated. Run PostgreSQL yourself and you own backups, upgrades, replication, and storage. Hand it to Azure and those genuinely disappear from your plate, which is great. But now you own VNets, private endpoints, DNS zones, firewall rules, and managed identities instead. The database becomes the easy part.

And this isn't an Azure problem. AWS does the same thing, Google Cloud does the same thing. Cloud providers are very good at hiding infrastructure. They're much less good at hiding distributed systems, and eventually every abstraction leaks: a DNS zone here, a security group there, a certificate that was supposed to be someone else's job.

The tradeoff is still often worth making. I'd rather debug a DNS zone link once than babysit replication forever. But pretending the complexity evaporated helps nobody, least of all the person staring at a connection timeout.

Takeaway

The SQL worked, the schema worked, authentication worked. Every single issue lived outside PostgreSQL, in services I didn't know I was operating until they failed. So now, whenever someone tells me "don't worry, it's managed," my next question is: "which part isn't?" That's where the next four hours usually go. The next time a managed service refuses to work, don't assume your application is broken. The bug may be sitting in a DNS zone you didn't know existed.

Fighting a cloud setup that refuses to connect? Send me the details and I'll take a look.

Tailscale SSH fails with "tailnet policy does not permit access"

2026-06-20T00:00:00.000Z

Tailscale is one of those tools that, when it works, makes you wonder what you were doing with your life beforehand. You install it on two machines, log in on both, and the private network that used to take an afternoon of firewall rules and key juggling just appears. Servers are reachable. SSH stops being a chore. Infrastructure becomes boring, which is the highest compliment I can pay it.

Then one day I ran tailscale ssh against a box I'd connected a hundred times, and got this:

tailnet policy does not permit access

The machine was online. Regular SSH still worked. tailscale ping came back clean. The node looked perfectly healthy from every angle I could think to check. And the error reads like a networking problem, so naturally I spent the first ten minutes debugging the network, which was fine the whole time.

Tailnet access and SSH access aren't the same thing

This is the bit that took me a minute to internalise. Being reachable on your tailnet and being allowed to tailscale ssh in are governed by two completely separate things. The first is your regular ACL connectivity. The second is a dedicated ssh section in your tailnet policy that nothing else touches.

So you can sit in a state where the box pings, MagicDNS resolves it, plain old ssh user@host over the tailnet works, and tailscale ssh user@host still slams the door. Everything points at the network because the network is the obvious suspect. The network is innocent.

The actual cause

In my case the policy simply had no SSH rules at all. Tailscale's default posture is to deny tailscale ssh unless a rule explicitly permits it, so an empty ssh section (or a missing one) means every attempt comes back as "tailnet policy does not permit access." It isn't refusing you because something's broken. It's refusing you because you never told it not to.

The fix is adding a rule to the ssh block in your tailnet policy:

{
  "ssh": [
    {
      "action": "accept",
      "src": ["autogroup:members"],
      "dst": ["autogroup:self"],
      "users": ["autogroup:nonroot"]
    }
  ]
}

That one says: any member of the tailnet can SSH into devices they own, as any non-root user. Adjust src, dst, and users to taste, but the shape matters more than the specifics. No rule, no access.

You also have to enable SSH on the target

There's a second trap waiting even after the policy is right. The machine you're connecting to has to be running with Tailscale SSH switched on, and it isn't on by default. You turn it on with:

sudo tailscale up --ssh

You can confirm the state with tailscale status. If SSH isn't enabled on the node itself, the policy can be flawless and you'll still get nowhere, because there's nothing on the other end listening for Tailscale to hand the connection to.

The root vs nonroot gotcha

This one's subtle enough that it deserves its own warning. Look again at that rule:

"users": ["autogroup:nonroot"]

It does exactly what it says. If you've been testing with:

tailscale ssh root@server

it'll keep failing, and you'll keep staring at a policy that looks correct, because by that policy root genuinely isn't allowed. Most examples you'll copy off the internet exclude root on purpose, which is the sensible default. It's also the easiest thing in the world to forget while you're three coffees deep trying to work out why a "working" config won't let you in.

What I check now, in order

After getting bitten by this, I have a short mental checklist that catches it almost every time. Is the node online? Does tailscale ping reach it? Is SSH actually enabled on the target with --ssh? Does the tailnet policy contain an ssh section that covers this connection? And is the user I'm logging in as allowed by that rule, root included? The answer is usually sitting in step three or step five, and the network at the top of the list is almost never the culprit.

Takeaway

When tailscale ssh says the tailnet policy doesn't permit access, resist the urge to debug the network. Connectivity and SSH authorisation are separate systems in Tailscale, and this error lives entirely in the second one. Enable SSH on the target, make sure your policy has an ssh rule that matches both the device and the login user, and the door opens. If the message instead said "you're connected, but the SSH policy denies this", I'd have got there a lot faster. Maybe one day it will.

Wrestling with a tailnet ACL or some other piece of weird infrastructure? Tell me what's going on and I'll take a look.

Modernization usually means rewriting everything wrong

2026-06-15T00:00:00.000Z

There's a particular meeting I've been in more than once. Someone opens a slide deck, a system that's been quietly running payroll for twenty years gets labeled "legacy", and a room full of people who have never read a line of its code start nodding along to words like modernization, cloud-native, and digital transformation.

That's usually the moment the trouble starts.

The old systems still running are usually the good ones

The genuinely terrible old systems mostly died years ago. The ones still alive today are the opposite. They're stable, and they're full of edge-case handling that nobody ever bothered to write down. They survived because they worked, not because they were pretty and not because they used a fashionable framework.

A COBOL system that's moved millions of dollars without a bad night in fifteen years isn't bad software. It just has the wrong haircut.

The rewrite question is the wrong question

The first question in the room is almost always "how do we rewrite this?". That's already a wrong turn. The question worth asking is "what are we actually trying to fix?", and those are not the same thing.

Sometimes the real pain is missing documentation. Sometimes it's onboarding that takes six months, or a system with no API to integrate against, or reporting that can't answer a simple question. None of that requires touching the core business logic. But rewriting feels good, especially to a team that didn't build the original and doesn't carry any of its scars.

"Nobody understands the old system"

I hear this one constantly. "The old code is impossible to understand." And yet payroll still runs, invoices still go out, the warehouse still syncs, and customers still get billed the right amount. Somebody understood it well enough for the business to survive two decades.

The real problem usually isn't the code. It's that the people who knew it retired, the documentation was neglected, and maintenance got starved for years because nothing was ever visibly on fire. That's an organizational failure wearing a technical costume. Rewriting the software doesn't fix the culture that let the knowledge walk out the door.

Rewrites are very good at deleting invisible logic

This is where it gets expensive. Old enterprise systems accumulate thousands of small business rules over time: a special price for one customer, an accounting exception for one jurisdiction, a regulatory quirk, a workaround discovered during some incident in 2011. Most of it lives nowhere. Not in the docs, not in a ticket, not in a spec. It lives in the running system.

When a team rewrites from scratch, they tend to rediscover those rules one production outage at a time, with the customer watching.

"Modern" and "simpler" are not the same word

There's a specific move I've watched play out more than once. Replace one stable monolith with a dozen microservices, a service mesh, an event bus, distributed tracing, four cloud environments and a pile of YAML, all to reproduce a process that used to run on a single box in a basement.

You haven't simplified anything. You've turned a maintenance job into a permanent platform team.

The strangler pattern is boring for a reason

The modernizations that actually work tend to avoid the big-bang rewrite entirely. You wrap the stable system in an API, replace one piece at a time, keep the proven logic running while you go, and shrink the risk on every step instead of betting the company on a cutover weekend.

It doesn't make a thrilling conference talk. It just keeps the business alive. You don't demolish the whole building because one floor needs rewiring.

The hard part was never the technology

Most of these projects don't fail because COBOL is scary. They fail on unclear ownership, optimistic timelines, missing domain knowledge, and management treating a migration as a way to reset organizational problems through software. That almost never works.

The technology is usually the most cooperative thing in the room.

Greenfield is fun. Businesses don't live there.

I get the appeal. Greenfield work is clean architecture, new tools, and nobody else's old decisions to inherit. But real companies come with history, integrations, regulators, customer expectations, and twenty years of accumulated logic.

Software engineering isn't architecture cosplay. The job isn't to build the prettiest system. It's to keep the business running while you make it better.

Sometimes the right move is almost nothing

This one makes people uncomfortable. Occasionally the best modernization plan is to fix the backups, add real monitoring, write down how the thing actually works, put the source under version control, and upgrade the infrastructure carefully, without touching the core at all.

Not every old system needs reinventing. Some of them just need the maintenance they were denied for ten years.

Takeaway

There's a real difference between software that looks old and software that's failing. A surprising amount of "modernization" is aesthetic discomfort dressed up as technical necessity.

Old systems can genuinely become dangerous. Unsupported hardware, real security holes, vendor lock-in, nobody left to hire. Those are worth acting on. But a full rewrite is usually the most expensive possible answer to any of them.

The most senior decision you can make is to notice that the old system isn't the enemy. The missing understanding around it is.

I help teams maintain, debug, and modernize OpenVMS and other legacy systems without the risky rewrite: COBOL, RMS, Rdb, and integration with modern infrastructure. If you're sitting on something critical that everyone's afraid to touch, that's the conversation worth having.

NAT Gateway is eating your AWS bill

2026-05-10T00:00:00.000Z

NAT Gateway costs $0.045 an hour just for existing. That's about $33 a month per gateway before a single byte passes through it, plus another $0.045 per gigabyte for every byte that does. Two of them across two availability zones, like every reference architecture suggests, and you're already at $66 a month before your app does anything.

This is the part of AWS that quietly eats bootstrapped startups. You read a few "production-ready" tutorials, you copy a sensible-looking diagram, and the next thing you have is private subnets, ECS on Fargate, RDS in two availability zones, an Application Load Balancer, NAT Gateways, and CloudWatch shoveling logs at every layer. It feels solid. It feels like the grown-up version of what you used to do on a $5 VPS.

Then the first real bill lands. Twelve users, almost no traffic, barely any revenue, and AWS already wants several hundred dollars a month for the privilege.

Most of that bill isn't your app. It's the plumbing around your app, and NAT Gateway is the biggest piece of it.

The architecture everyone copies

The pattern the tutorials hand you usually looks like this: public subnets for the load balancer, private subnets for the application containers, private subnets for the database, outbound internet routed through a NAT Gateway.

From a security angle it's fine. The application servers aren't directly exposed. The database is isolated. The diagram looks clean enough to put in a deck.

The issue is who that pattern was actually written for. AWS reference architectures are aimed at companies with compliance teams, funded startups with platform engineers, and internal systems where six-figure infra bills don't show up in anyone's personal life. They aren't written for one person trying to ship version one without burning rent money.

Copying the enterprise pattern at the prototype stage isn't being responsible. It's just expensive.

Private subnets aren't free internet

Here's the part the tutorials don't dwell on: a container in a private subnet can't reach the internet on its own. Every time it pulls a Docker image, hits Stripe, calls SendGrid, fetches an OAuth token, ships logs to a third party, or talks to S3 over the public endpoint, that traffic has to go somewhere. AWS routes it through the NAT Gateway, and now you're paying $0.045 a gigabyte for the privilege of letting your container reach Stripe.

Most of this is invisible while you're building. Your app might be quiet, but the infrastructure around it is chatty all day. Logs, metrics, package updates, container pulls, health checks, background workers, third-party API calls, all of it flowing through the same expensive pipe.

Why this hits bootstrapped people the hardest

NAT Gateway isn't bad. The problem is what it does to the early-stage P&L.

It introduces a fixed monthly infrastructure tax that exists before the product earns a single dollar. If you raised funding, that's annoying but invisible. If you're a solo founder paying out of personal savings while validating an idea, it's the difference between "I can run this for a year" and "I have to pull the plug in three months."

The shape of the bill ends up bizarre. Tiny database, minimal compute, almost no users, and yet the AWS invoice looks like you're running a medium-sized SaaS. I've seen setups where NAT Gateway alone cost more than the entire application compute it was supposedly serving.

That's a strange place to be when you're still trying to find out whether anyone wants the product.

The cross-AZ surprise

This is the part that catches people who thought they were being careful.

Common shape: ECS tasks in one AZ, NAT Gateway in another, usually because somebody set up the VPC with a single NAT Gateway "for cost reasons" and didn't realize the tasks would land elsewhere. Now every outbound request from those tasks pays cross-AZ data transfer on top of NAT processing.

AWS doesn't pop up a banner saying "congratulations, your tiny side project is now paying inter-AZ networking fees." You discover it later, staring at Cost Explorer and trying to work out why "outbound HTTPS to api.stripe.com" turned into a four-figure annual line item.

Most early products don't need private subnets yet

This is the part that makes people uncomfortable.

For a lot of early-stage products, the right answer is genuinely a single EC2 instance. Or App Runner. Or a small ECS service in a public subnet with a security group doing the actual exposure control. Or a Hetzner box that costs less per month than the NAT Gateway alone. The internet will not collapse because your MVP isn't wired up like a Fortune 500 banking platform.

What early infrastructure should optimize for is simplicity, cost, and how fast you can change things. Not how impressive the diagram looks on LinkedIn.

There are also middle paths nobody talks about. VPC endpoints for S3 and DynamoDB skip NAT entirely for that traffic, often for free. Interface endpoints handle a lot of AWS service calls. IPv6 egress-only gateways give you outbound internet for IPv6 without NAT pricing. Each of these is a small piece of work and a real line off your bill.

AWS isn't the villain

NAT Gateway exists for good reasons. At scale it simplifies networking, centralizes outbound access, and gives security teams a chokepoint they can audit. In a mature business the cost is rounding noise.

The mismatch is what kills you. Applying enterprise architecture to a product that's still searching for product-market fit is the same mistake as hiring a CFO before you have revenue. It looks like seriousness. It's actually just overhead.

Takeaway

There's a difference between architecture that scales technically and architecture that scales economically. Most early-stage AWS bills are paying for the first one before the business needs either.

Good engineering at the prototype stage isn't picking the most sophisticated infrastructure on offer. It's picking the smallest thing that does the job, and only adding the next layer once the business actually asks for it. Sometimes the most senior decision in the room is admitting you don't need the fancy setup yet.

If your AWS bill is way bigger than your product, send me the details and I'll take a look.

WordPress fleets on AWS are rotting from the database up

2026-04-27T00:00:00.000Z

Every dashboard was green and the site didn't load.

Web server CPU: bored. Memory: fine. Apache happily handling whatever Cloudflare forwarded its way. Pages took eight or ten seconds to start rendering, then stalled, then maybe came back. Like Wi-Fi at a busy coffee shop.

The database lived on its own RDS instance, and that was the part that confused everyone. It had been split off the EC2 box two years earlier specifically for reliability. MySQL on the same EC2 box as PHP kept saturating things during traffic spikes, so the operator did the responsible thing: peeled it off, put it on a gp2-backed db.t3 RDS, slept better at night. Problem solved.

Two years later, that exact instance was the one that went silent. The RDS storage burst balance had drained to zero somewhere around 9am, and from then on every query was waiting on disk IOPS the instance no longer had. Web server: bored. RDS: gasping. Dashboards: green, apart from one graph nobody was watching.

The fix from two years ago had quietly become the bug. That sentence is most of what's wrong with WordPress fleets on AWS, summarized.

How these fleets accumulate

An aging WordPress fleet on AWS is almost never something somebody designed. It's something that happened.

You start with one site on a small EC2. WordPress is light, one box handles it fine. Then you add a second site, then a third. The agency takes on a client whose old site just needs to stay online for another year. A WooCommerce install lands. A dev environment becomes a staging environment becomes a "we don't really know what this one does anymore." Different developers install different caching plugins. PHP versions drift between sites that share a box. Cron jobs pile up. Backups multiply, then overlap, then nobody's sure which ones still work. One client launches a campaign and melts shared resources for everyone else for forty minutes on a Tuesday morning.

None of these decisions was wrong on the day it was made. Together, they build a system nobody can confidently change.

The database is doing more than you think

Most WordPress performance complaints are really database complaints in a costume.

Pages look slow because PHP is sitting on its hands waiting for MySQL. Admin panels drag because half the popular plugins do queries that would embarrass a junior developer. Random timeouts show up because the storage burst balance on the RDS volume is gone and every query is now waiting on disk IOPS the instance simply doesn't have anymore.

That last one is the trap. AWS sells burstable resources, things like db.t3 instances and gp2 volumes, that perform beautifully right up until the moment they don't. The credits last for hours of moderate use, then run out, then performance falls off a cliff at exactly the wrong time. To the customer it looks random. To the operator it looks like a haunted machine.

Cheap infrastructure that collapses under normal usage isn't actually cheap. It's just billed differently.

Cloudflare hides a sick origin

Cloudflare in front of WordPress is genuinely useful. Edge caching, bot filtering, the SSL pipework, a cheap layer of armor against script kiddies. None of that is wrong.

What it also does, and what nobody mentions in the marketing, is hide how sick the origin really is. From the outside the site looks mostly fine because cached pages render in milliseconds. The logged-in users, the checkout flow, the editors hitting wp-admin, every API call, every cache miss. All of those are talking to the dying origin and getting the dying-origin experience.

Then traffic spikes, or the cache gets purged, or a campaign tag changes the URL pattern enough that nothing's in cache anymore, and the illusion drops. The thing was on fire the whole time. Cloudflare was just very good at the smoke screen.

The AWS bill is paying for caution, not performance

The bill on these setups grows in places that have very little to do with how the sites actually perform.

EC2 instances stay one or two sizes too big because nobody trusts downsizing without a load test that nobody has time to run. RDS instances carry years of inefficient queries the team has learned to live with. gp2 volumes sit there bursting and then not-bursting; nobody upgrades to gp3 because nobody knows the calculator off the top of their head. Snapshots accumulate at one per day per instance and nobody has audited the retention policy since 2021. There's an idle Application Load Balancer that used to front something. There's a NAT Gateway moving the tiniest amount of traffic at the steepest possible per-GB rate. Two of the environments are duplicates of each other "just in case." A third-party APM is collecting metrics from a service that was decommissioned eighteen months ago.

None of these line items will hurt you on their own. Together they're often a third of the bill.

You're not managing traffic, you're managing entropy

People think a fleet of fifty low-traffic WordPress sites is easier than one busy app. It isn't.

A busy app is one codebase, one deploy pipeline, one PHP version, one caching strategy, one set of cron jobs. You can know that app. Fifty WordPress sites are fifty plugin lists, fifty theme variants, fifty backup schedules, fifty sets of email quirks, fifty sets of client-specific surprises that nobody documented. A handful of them are running PHP versions you'd rather not name out loud. Two of them are quietly out of disk. One has been compromised for a month and is mining something.

You're not managing traffic. You're managing entropy.

That's why agencies feel exhausted even when no single site is "big." The exhaustion isn't proportional to traffic. It's proportional to the number of independent things that can go wrong overnight without warning.

What I'd actually do if I inherited one

If somebody handed me a tangled WordPress fleet on AWS tomorrow, I wouldn't start with Kubernetes, six new services, or any of the architecture cosplay that gets pitched at this stage. I'd start by knowing what's there.

First, an honest inventory. Which sites still make money? Which ones are abandoned? Which ones are legacy liabilities that nobody owns and nobody wants to admit they don't own? Most operators, when pressed, can't actually answer this. That's the real starting point.

Then, separation. The site that pays the rent shouldn't be sharing a database with a brochure site from 2018 that nobody's logged into in three years. Carve the important workloads off, even if it means duplicating a bit of infrastructure. Noisy neighbors are cheap to fix once you know who they are.

Then the database. Right-size the instances, move gp2 volumes to gp3, kill the queries that have been embarrassing themselves for years, fix the indexes that should have been there from the start. This usually moves the needle more than anything you can do to PHP.

Then standardize the runtime. One PHP version, one deploy pattern, one caching approach, one monitoring stack. Mixed environments create the kind of friction that quietly burns a senior engineer's afternoon every other day.

Then question every component that exists "because it sounded smart in 2022." If the load balancer is fronting one instance, it isn't load balancing. If the NAT Gateway exists for one outbound API call a day, it's a $30/month rental for a single phone call.

Finally, decide what should leave WordPress entirely. Some sites should become static. Some should be rebuilt against something more boring. Some should be quietly retired and replaced with a 301 redirect.

Takeaway

WordPress isn't the villain in any of these stories. AWS isn't either. The villain is the slow accumulation of decisions that were each individually reasonable, made by people who weren't there for the previous ones, on top of infrastructure that will quietly let you build whatever you want.

At some point the cheapest optimization isn't another instance upgrade. It's stepping back, looking at what you actually have, and putting it back into a shape somebody can understand.

Inherited a tangled WordPress fleet on AWS? Send me the details and I'll take a look.

Just want someone to take care of your WordPress hosting? Biro.works does a great job of exactly that.

The one setting that stops Microsoft Authenticator spam

2026-04-15T00:00:00.000Z

For months I was getting hammered with Microsoft Authenticator login prompts. Not one or two, but easily ten or more a day, always from places like Germany or the Netherlands, always labeled "Windows," and always at random times. At first it felt a bit concerning, then it just became incredibly annoying because I had to keep tapping Deny over and over again like some kind of human firewall.

My first assumption: compromised password

Naturally, I assumed the obvious: someone must have my password. So I changed it to something strong and completely new, logged out of all sessions, checked account activity, and generally did everything you're supposed to do when you think your account might be compromised. Everything looked clean, but the prompts kept coming as if nothing had changed.

Why password resets don't stop this attack

That's when it became clear that this wasn't a typical "someone got your password" situation. They don't actually need your password for this. All they need is your email address, and if that email is public (which mine is), they can keep triggering login attempts that push notifications to your Authenticator app. The whole idea is simple: spam enough requests and hope you eventually approve one by mistake, whether out of distraction or frustration.

Things I tried that didn't work

I went through all the usual fixes people suggest. I verified passwordless login was off. I removed and re-added Authenticator. I logged out of every session again just to be sure. I checked the sign-in logs, which turned out to be mostly useless because they only showed my successful logins and none of the garbage attempts coming from abroad. Nothing made a difference. The system was technically secure, but I was still being spammed constantly.

The fix that actually worked

What finally fixed it had nothing to do with passwords or Authenticator itself. I went into account settings, added a new email alias that nobody knows, made it the primary login, and removed (or at least disabled) my old email as a sign-in option.

The effect was immediate. The prompts stopped completely. No gradual decrease, no delay, just silence.

Why this works (and why the fix is so hidden)

The reason is pretty simple in hindsight. All those attempts were targeting my old, public email address. Once that email stopped being a valid login identifier, the attacker was effectively knocking on a door that no longer existed. The authentication process never even started, so there was nothing left to trigger on my phone.

Looking back, everything I tried before was focused on strengthening authentication, but the real problem was the entry point itself. As long as your public email is a valid login, someone can keep poking at it and triggering prompts, regardless of how strong your password is or whether passwordless is enabled.

Takeaway

So if you're dealing with the same thing, don't waste time cycling through password changes and app resets. The only thing that actually stopped it for me was changing the login alias to something private and removing the old one from sign-in. It's a small change, but it completely cuts off the attack vector.

It's also a bit frustrating that the system works this way, because instead of blocking the attacker upstream, it basically offloads the problem onto you and expects you to keep denying requests. But once you understand what's really being targeted, the fix becomes obvious. And thankfully, it works instantly.

Running into weird authentication or security issues? Send me your problem and I'll take a look.

Your server is running two PHP versions (and one of them is lying)

2026-03-24T00:00:00.000Z

I ran into a classic PHP mess the other day. A Symfony app was behaving perfectly fine from the command line (no warnings, no drama, nothing flagged as broken) and then the moment I loaded it in the browser, it blew up with this:

Undefined constant "XML_PI_NODE"

It's the kind of error that sends you chasing ghosts: something deep in the app, a flaky extension, a weird dependency mismatch buried in Composer. What it does not immediately scream is "your server is running two different PHP versions depending on how you access it." But that was exactly what was happening. CLI was on one PHP version, Apache was on another, and even though both were running on the same box against the same files, the two paths were serving code through completely different runtimes. From the terminal everything looked healthy. From the browser, things started breaking in confusing ways.

Why "PHP version" isn't really one thing

On Linux, "PHP version" is rarely a single value. The CLI binary can be on one version, the Apache module on another, and PHP-FPM on a third, all sitting quietly side by side on the same machine. Upgrade one of them and forget about the rest, and the box keeps limping along fine until something in the code hits behavior that differs between versions. That's usually when the weird errors start, and the most maddening part is that they don't show up consistently. The app isn't really "broken." It's only broken down one execution path.

The first thing I always check now

Any time PHP works in the terminal but not in the browser, I start with the same two commands. The first tells me what CLI is using:

php -v

That's half the picture. The other half is what the web server is actually running, and the quickest way to see it is still the oldest trick in the book: drop a temporary phpinfo() file into the web root:

<?php phpinfo();

Open it in the browser and compare the version it reports with what php -v gave you. If the two numbers don't match, there's nothing else to investigate yet. You've already found the problem, and the rest of the debugging is just pinning down which part of the stack is out of sync.

If you're running mod_php

If Apache is loading PHP directly through mod_php, check which module it currently has active:

apache2ctl -M | grep php

You can also look at what's installed on the system more broadly:

dpkg -l | grep libapache2-mod-php

Between those two outputs, it usually becomes obvious whether Apache is still wired up to an older PHP module that nobody remembered to disable after the upgrade.

If Apache is proxying to PHP-FPM

If you're on PHP-FPM instead, the Apache module isn't your problem at all. The socket path is. Look at your site config and find the line that points Apache at PHP-FPM. It's usually something like:

/run/php/php8.x-fpm.sock

That version number in the socket name really matters. It's very easy to install a newer FPM package thinking you've upgraded PHP everywhere, while Apache is still happily talking to the old socket.

Why the error message wasted my time

The most irritating thing about this whole class of bug is that PHP never fails in a way that points at the real problem. It doesn't say, "CLI is on 8.5, Apache is on 8.3, please sort your life out." It just throws side effects at you: a missing constant here, a removed function there, some extension behaving differently, random incompatibilities that look for all the world like application bugs but are really environment bugs. And if you don't know to look at the environment first, you can burn hours debugging the wrong layer.

That's exactly what happened to me here. It wasn't Symfony. It wasn't Composer. It wasn't the cache. It wasn't some mystical XML bug buried in a library. Apache was simply running a different PHP version than the terminal, and every other lead I chased was a dead end.

The actual fix

Once the mismatch is confirmed, the fix is almost embarrassingly simple: pick one PHP version and make every layer use it. If Apache is still loading an old mod_php, turn it off and turn the right one on:

a2dismod php8.3
a2enmod php8.5
systemctl restart apache2

If you're on PHP-FPM, install the version you actually want and update the site config so Apache points at the matching socket:

apt install php8.5-fpm

Then restart both services so the new pairing actually takes effect:

systemctl restart php8.5-fpm
systemctl restart apache2

After that, the weird error disappears and everything becomes boring again, which is exactly what you want from infrastructure.

How to keep this from happening again

The easiest way to avoid all of this is to be deliberate about your PHP setup. If you use PHP-FPM, use it everywhere and stop mixing it with mod_php. If you use mod_php, know which module is currently enabled and make sure it matches the CLI version. And if you keep multiple PHP versions installed on the same box (which is fine, and sometimes necessary), keep track of which service is pointing at which, because having multiple versions isn't really the problem. Forgetting that different parts of the stack can quietly drift apart from each other: that's the problem.

Takeaway

If PHP works from the terminal but not from the browser, don't dig into the app first. Start by comparing the PHP versions: check php -v, check phpinfo(), check your Apache module or your FPM socket path. There's a good chance the app is fine and the server is just the one lying to you.

Running into a weird PHP or Apache mismatch? Send me the details and I'll take a look.

Stop all running containers in Docker

2020-10-03T00:00:00.000Z

There's no docker stop all. The command you actually want is:

docker stop $(docker ps -q)

That's the whole trick. If you want to know why it works, what the fish shell version looks like, and when to reach for kill instead, read on.

Why you'd want this

Sometimes you start containers from your IDE and forget to stop them before switching projects. Or maybe you're done with your work (you lucky, you!) and don't want to waste memory or CPU cycles on services you aren't using. Either way, something is hogging the ports you need, eating your RAM and battery, and the IDE that started it all is long gone. Time for the terminal.

Let's see what we're up against. What does docker ps say?

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
9c07b6375e0e        my_project_tunnel   "/bin/sh -c 'rm -rf …"   4 weeks ago         Up 3 seconds        0.0.0.0:8062->8062/tcp                           my_project_tunnel_1
91b757beda05        my_project_web      "docker-php-entrypoi…"   4 weeks ago         Up 2 seconds        0.0.0.0:8000->80/tcp                             my_project_web_1
245f7e35f4f3        mysql:5.7           "docker-entrypoint.s…"   3 months ago        Up 3 seconds        0.0.0.0:3306->3306/tcp, 33060/tcp                my_project_db_1
77bc948bb148        mailhog/mailhog     "MailHog"                3 months ago        Up 3 seconds        0.0.0.0:1025->1025/tcp, 0.0.0.0:8025->8025/tcp   my_project_mailhog_1

Four containers. Docker's answer, out of the box, is four separate commands:

$ docker stop 9c07b6375e0e
$ docker stop 91b757beda05
$ docker stop 245f7e35f4f3
$ docker stop 77bc948bb148

Quite a mouthful. You can shave it down by typing only the first few characters of each container ID, but the real fix is to make the shell do the work: run docker ps, grab the container IDs, and feed them straight into docker stop.

Step 1: Get the running container IDs

Run docker ps with the -q flag and it returns only container IDs. Look!

$ docker ps -q

9c07b6375e0e
91b757beda05
245f7e35f4f3
77bc948bb148

Many tutorials will tell you to add the -a flag, making the full command docker ps -aq. Here, -a is redundant. According to the docker ps documentation, omitting -a returns only running containers, which is exactly what we need. Stopped containers don't need stopping.

Step 2: Feed them to docker stop

Almost every shell lets you run a command in a subshell, capture its output, and use it as arguments for another command. This syntax works in bash and zsh:

docker stop $(docker ps -q)

And this works in fish:

docker stop (docker ps -q)

You can use kill instead of stop for a faster shutdown. It saves a few seconds, but not every service appreciates being killed; databases in particular would rather get a polite stop and a chance to flush to disk. If the containers are throwaway dev services, kill away.

Why Docker?

2020-10-01T00:00:00.000Z

Docker is an awesome tool for web app development. Back in the stone age (i.e. pre-2010), if you needed a specific version of PHP with a specific set of extensions and a specific version of MySQL, your dev environment had to be... well, pretty specific.

Developers had to adjust their working environment to whatever the current project needed. That was a cumbersome task, so most of us settled for "close enough": the PHP version would be somewhere in the ballpark of what was required, there might be a few extra extensions lying around, and most of the time it was fine. Except when it wasn't.

The next logical step was to create isolated environments used only for development. Some sort of virtual machine, so to speak. Actually, that's exactly what they're called. The process is very easy: download a Linux ISO, spin up a VM, install a fresh Linux on it, install Apache, PHP, and every extension and config you need, set up the network share and the virtual network interface, and within a week or two you're good to go. Rinse and repeat for every project you're working on. You might understand why this wasn't anyone's favorite option.

Enter ~~Docker~~ Vagrant. This tool was a great helper in the early 10s. Instead of messing with virtual machines by hand, you described what you wanted the VM to look like, and from then on simple up/down/destroy commands did the respective things to it. Vagrant was essential for replicating production on a development machine, but it still carried the overhead of running two (or more) entire operating systems at the same time. Not great for performance, battery life, or your carbon footprint.

Enter Docker. Depending on the host OS, it introduces practically no overhead while still letting you run anything on anything. You can give every service its own container and have them talk to each other over an internal network, expose any port or map it to any other port, and bring the entire app environment up or down using only one command.

That's the whole pitch: production parity without the week of VM setup. Once you've worked that way for a while, going back feels like hand-cranking a car.