aws_instance must be replaced, because Amazon published a new AMI

All I wanted was an SMTP user. Self-hosted Mattermost can't send password-reset emails without one, SES needs an IAM user for SMTP credentials, so I added three resources to the Terraform: aws_iam_user, aws_iam_user_policy, aws_iam_access_key. About as boring as infrastructure changes get. I ran terraform plan expecting "3 to add" and got this instead:

Plan: 7 to add, 2 to change, 4 to destroy.

Four to destroy. In a stack where I'd added three things and removed nothing. The list:

  # aws_iam_access_key.ses_smtp will be created
  # aws_iam_role_policy.github_deploy will be updated in-place
  # aws_iam_user.ses_smtp will be created
  # aws_iam_user_policy.ses_smtp will be created
  # aws_instance.host must be replaced
  # aws_lb_target_group_attachment.ingestion must be replaced
  # aws_lb_target_group_attachment.mattermost must be replaced
  # aws_s3_object.compose will be updated in-place
  # aws_volume_attachment.data must be replaced

aws_instance.host is the production box. The one EC2 instance that runs the whole platform: the durable workflow engine, the API, the dashboard, and the Mattermost I was trying to fix. Terraform wanted to tear it down and build a new one, and then re-attach the data volume and the load balancer targets to whatever came out the other side.

The one line that explains it

terraform show on the saved plan, grep for forces replacement:

  # aws_instance.host must be replaced
-/+ resource "aws_instance" "host" {
      ~ ami = "ami-007b20f2c1234c2c9" -> "ami-0e3942cfb1b8de381" # forces replacement

That's it. The AMI changed. I hadn't touched it, but then I'd never pinned it either. The instance gets its image from a data source that looks like every AMI data source in every tutorial ever written:

data "aws_ami" "al2023" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["al2023-ami-*-x86_64"]
  }
}

resource "aws_instance" "host" {
  ami           = data.aws_ami.al2023.id
  instance_type = var.instance_type
  # ...
}

most_recent = true means exactly what it says: every time Terraform refreshes, it asks AWS for the newest Amazon Linux 2023 image that matches the filter. In May, when the host was first created, that was ami-007b20f2c1234c2c9. Amazon publishes a new AL2023 AMI every few weeks. By September, "most recent" pointed at ami-0e3942cfb1b8de381, and since the AWS provider can't swap the image under an existing instance, a different ami means destroy and recreate.

The three other "must be replaced" lines were just collateral. They reference aws_instance.host.id, the new instance would have a new id, so they'd all be rebuilt too. And the aws_iam_role_policy.github_deploy "will be updated in-place" in that list? Same story, one level removed: the deploy role's policy embeds the instance ARN for SSM, and that ARN was suddenly "known after apply". (The aws_s3_object.compose update is the only innocent one. That's the compose file picking up the new SMTP settings.) One drifted AMI, five resources churning.

What would have actually happened

I like to be honest about blast radius in these posts, so here's what terraform apply would have done had I typed yes on autopilot.

The instance gets terminated. Its root disk goes with it: Docker images, the rendered .env, the Tailscale node identity, the SSM agent registration. A fresh instance boots from the new AMI and cloud-init runs the bootstrap script from scratch. The data would've survived, since the data volume is a separate aws_ebs_volume and Postgres for Mattermost and the engine's state live there. Everything else starts over: re-mount the volume, re-join the tailnet as a new machine, re-pull every image. The GitHub Actions deploy workflow has the instance id as a repository variable, so the next push would've tried to deploy to a host that no longer exists. Somewhere between five and thirty minutes of downtime, and an afternoon of "why is the deploy failing" on top.

For an SMTP user.

The really uncomfortable part is that this had nothing to do with my change. Any terraform apply after roughly June would have produced the same plan. Adding a tag. Fixing a typo in a description. The trap was armed the day I wrote most_recent = true and it just needed me to run apply once, for any reason, after Amazon's next image release.

The fix

You have two sane options. Pin the AMI id in a variable and lose the data source entirely, or keep the data source for new environments and tell the instance to stop tracking it. I went with the second:

resource "aws_instance" "host" {
  ami           = data.aws_ami.al2023.id
  instance_type = var.instance_type
  # ...

  # data.aws_ami.al2023 is `most_recent`, so every new AL2023 release would
  # otherwise make the next apply *replace* the prod host (and re-run
  # cloud-init). OS updates come via dnf on the host; a deliberate AMI bump is
  # done with `terraform apply -replace=aws_instance.host`.
  lifecycle {
    ignore_changes = [ami]
  }
}

A fresh terraform apply on a brand new environment still picks up the latest image, because ignore_changes only applies once the resource exists in state. The existing host stays on the image it was born with, and OS patches arrive the way they do on any long-lived box: dnf update on the host, not a rebuild. When I want the new image, the escape hatch is right there in the comment. -replace makes the rebuild an explicit decision instead of a side effect.

Re-plan:

  # aws_iam_access_key.ses_smtp will be created
  # aws_iam_user.ses_smtp will be created
  # aws_iam_user_policy.ses_smtp will be created
  # aws_s3_object.compose will be updated in-place
Plan: 3 to add, 1 to change, 0 to destroy.

Zero to destroy. The github_deploy policy churn disappeared too, since the instance id was stable again. Apply, done, Mattermost got its SMTP user, and I got a new item on the list of things I check before typing yes.

Takeaway

most_recent = true on an AMI data source is a perfectly good way to pick an image for a server you're about to create. It's a terrible way to describe a server that already exists, because "the latest image" is a moving target and Terraform's only way to move an instance onto it is to destroy the instance. If your aws_instance is a pet (one box, persistent disk, a hostname other things depend on), add ignore_changes = [ami] the same day you write the data source. And read the destroy count on every plan, even the boring ones. Especially the boring ones.

Got a plan that wants to destroy more than you changed? Send me the details and I'll take a look.

• Aws
• Terraform
• Infrastructure
• Debugging