Restate's default node name is your container hostname

I hit this one while cleaning up after a testing session. Local dev stack, nothing exotic: Postgres, Restate (the durable workflow engine), and a mail catcher, all in one compose file. I ran docker compose down, did some other work, ran docker compose up -d, and two of the three containers came up. Restate exited with code 1.

The log was one line:

[RT0002] invalid configuration: node-name is required: The working directory
'/restate-data' contains data from node 'ed5910dcd293' but the default node
name is '97418545ae39'. This would ignore the existing data. To avoid
accidental misconfigurations, you have to explicitly specify the node name to
tell Restate to resume from the existing data or to start a fresh node.

Nothing in my config mentioned a node name. I'd never set one. So where did ed5910dcd293 come from, and why was the "default" suddenly different?

Two hostnames walk into a volume

Both of those strings are Docker container ids. That's the whole bug.

Restate persists its state (journals, in-flight workflow invocations, the works) into a data directory, and stamps that data with the node's name. When you don't configure a name, it falls back to the machine's hostname. Inside a Docker container, the hostname is the container id.

Now walk through what compose does. docker compose down doesn't just stop the container, it removes it. The named volume with /restate-data survives, that's the point of volumes. The next docker compose up -d creates a brand new container with a brand new id, which means a brand new hostname, which means a brand new default node name. Restate boots, looks at the volume, and sees data belonging to ed5910dcd293 while believing it is 97418545ae39. Rather than quietly starting fresh and orphaning your workflow state, it refuses to start at all.

Which is, honestly, the correct behavior. For a database of durable executions, "I found somebody else's data, I'll just ignore it" would be a much worse default than a crash. But it does mean that every Restate-in-compose setup without a pinned node name is a time bomb with a very specific trigger: the first time the container gets recreated. Not docker compose restart, that keeps the container and its hostname. Recreation. A down/up cycle, a compose file edit, an image upgrade. Any of those, and your engine won't come back.

The fix

Give Restate a stable name. In compose that's one environment variable:

  restate:
    image: restatedev/restate:1.6.2
    environment:
      # Restate derives its default node name from the container hostname,
      # which changes on every recreation. Pin it or the next `up` fails
      # with RT0002 and refuses to resume /restate-data.
      RESTATE_NODE_NAME: "ed5910dcd293"
    volumes:
      - restatedata:/restate-data

The value matters if you have existing data you want to keep: use the id from the error message, the one the volume already belongs to. It's ugly, but the error literally hands it to you. If the volume is disposable (or you're setting this up fresh), pick any stable name you like, restate-local reads a lot better than a fossilized container id.

If you'd rather start clean instead, remove the volume and let it initialize under the new pinned name:

docker compose down
docker volume rm <project>_restatedata
docker compose up -d

The embarrassing part

While fixing the local compose file I opened our production one, ready to apply the same change there. It was already fixed. Pinned node name, complete with a comment explaining exactly this failure mode. Past me had hit the same wall on the prod host, solved it properly, written it down, and then never carried the fix back to the local file sitting two directories away.

The local stack survived for months anyway, because I almost never ran docker compose down. The containers just stayed up between sessions, keeping their hostname, keeping the peace. The bug was there the whole time, waiting for the one day I decided to tidy up.

Takeaway

If a stateful service derives its identity from the hostname and runs in a container, its identity is a container id, and container ids don't survive recreation. Pin the name explicitly on day one, in every compose file, not just the one that already burned you. And when an error message names two mysterious hex strings, check whether they're both just hostnames: half the mystery usually evaporates right there.

Fighting a container that won't come back up? Send me the details and I'll take a look.

• Docker