An operator on biro.works pinged me: "Can't approve the only item in the approvals section. This keeps happening when Restate is updated." The second sentence turned out to be the whole bug. It took me a while to believe it.

The setup

The agent platform runs on self-hosted Restate (restatedev/restate:1.6.2, TypeScript SDK 1.14.4). Human approval gates are built on ctx.awakeable(): the workflow suspends until someone clicks Approve or Reject in a Next.js dashboard. The click hits an ingestion service, which calls POST /restate/awakeables/{id}/resolve on the Restate ingress. The approval row in Postgres is flipped from pending by the workflow itself, in a finalize-approval durable step that runs after await awakeable.promise.

Everything lives on one EC2 host under docker compose. Each deploy builds images in GitHub Actions, restarts compose over SSM, and runs a one-shot container that re-registers the engine with Restate:

curl -X POST http://restate:9070/deployments \
  -H 'content-type: application/json' \
  -d '{"uri":"http://engine:9080","force":true}'

Remember that force.

A resolve that succeeded into the void

There were no browser errors to look at. The buttons are server actions, so everything happens server-side. The ingestion logs were the first interesting thing, because they said the resolve had worked:

{"level":"info","msg":"approval resolved","app":"biro","route":"approvals","id":10,"approved":true}

And yet the row stayed pending. Clicking again produced the same success line and the same stuck row.

I traced the chain end to end: dashboard action, ingestion route, resolveAwakeable(), Restate ingress. Nothing swallowed an error. The resolve genuinely came back 2xx.

So I asked Restate what it thought was running. The admin API has a SQL endpoint for that:

curl -X POST http://localhost:9070/query \
  -H 'content-type: application/json' \
  -H 'accept: application/json' \
  -d '{"query":"SELECT id, target, status FROM sys_invocation"}'

(Without the accept header you get Apache Arrow binary, which is not what you want in a terminal.)

The suspended leadIntake invocation that owned awakeable sign_18TUbPLapcZ4ZWEs9aL2ERrI3UOkbGFKzAAAAEQ wasn't suspended. It wasn't failed either. There was no row at all. Only a handful of recently completed projectScoping runs.

That's the bug in one sentence: resolving an awakeable whose invocation is dead succeeds, and nothing anywhere tells you. The only writer of the approval row was gone, so pending was forever.

The flag that reads like a no-op

Why was the invocation gone? Here's what the Restate docs say about the force flag on deployment registration:

If true, it overrides any existing deployment using the same uri. This can lead inflight invocations to an unrecoverable error state and implies breaking.

We force-registered on every deploy. Four deploys that day, and after each one, whatever had been sitting at an approval gate was gone. The deployment id didn't even change between them, which is what made it look harmless. The in-flight invocations didn't survive it anyway.

"This keeps happening when Restate is updated." The operator had it right in the first message.

The fix, in three layers

Stop forcing. The register one-shot now tries {"uri":"http://engine:9080"} without force first. When the service contract hasn't changed (same services, same handlers, new code), Restate treats that as a no-op and the suspended invocations are left alone. A retry loop (10 tries, 3 seconds apart) covers the engine still booting. force is the fallback for real contract changes, and it logs loudly when it fires. The next deploy logged registered instead of force-registered, and the gates lived.

One caveat, because it's a different failure with a different symptom: new code that changes the order of durable steps inside a handler can still break a suspended invocation when it replays, force or not. That shows up as a journal mismatch error on the invocation, not as a row that quietly disappears.

Check for a pulse before resolving. The gate now stores its owning invocation id in the approvals row. That's ctx.request().id, which the SDK typings describe as constant across retries and suspensions. Before resolving, the ingestion service runs SELECT status FROM sys_invocation WHERE id = '<inv_...>' against the admin API. No row, or completed while the row is still pending, means the approval gets marked expired and the caller gets HTTP 410 with a hint to re-trigger. There's still a small window between the check and the resolve, but a workflow that died an hour ago no longer gets a green success message.

Sweep the orphans. A five-minute cron expires any pending gate older than 20 minutes whose invocation is gone, and posts a Mattermost alert listing what it expired. Twenty minutes is past the 15-minute soft-gate auto-approve window, so a gate that's still inside its window is left alone. Rows from before the invocation id existed can't be verified at all, so after the threshold they're expired too. That's a tradeoff I accepted, and it healed the original stuck approval on the first tick:

{"level":"warn","msg":"approval expired","id":10,"why":"no invocation id (pre-tracking gate) and workflow never resolved it"}

The UI got some honesty as well. The Approve and Reject buttons were fire-and-forget: await fetch(...) with no look at res.ok. They're useActionState buttons now, with a pending state, a success confirmation, or the 410 "gate expired" message inline.

What I'm keeping from this

An awakeable is a bearer token into a workflow, and resolving it tells you nothing about whether anyone is still holding the other end. If your UI's source of truth is only ever updated by the workflow, verify the invocation is alive before telling the operator it's done. sys_invocation through the admin /query endpoint is the tool for that.

force: true on deployment registration looks like an idempotency flag. It's a dev-mode tool. The docs' "unrecoverable error state" means exactly what it says, even when the deployment id stays the same.

Same-URI redeploys with an unchanged contract don't need force at all. You only pay for in-flight invocations on genuine contract changes, and for a single mutable endpoint that price is unavoidable. So build the janitor (expire, alert, re-trigger hint) instead of pretending it can't happen.

And check res.ok. Fire-and-forget fetch in server actions produced three separate "the button silently does nothing" bugs in that codebase in one week.

Two smaller traps

  • The admin /query endpoint returns Arrow binary unless you send accept: application/json.
  • Its DataFusion SQL wants single-quoted string literals. $$dollar$$ quoting gets you a ParserError.

Fighting a stuck Restate workflow? Send me your problem and I'll take a look.