Never let the LLM write the subject line

I'm building biro.works, a website studio where AI agents handle the first line of everything: a lead emails us, an agent drafts the reply, a human approves it, the reply goes out. The pipeline worked. Leads got answers within minutes, the tone was right, the pricing came straight from the catalog.

Then I looked at a test conversation from the client's side, in Gmail. The client had sent "Need a website for my bakery." Our reply arrived as "Your new bakery website with biro.works". A brand new conversation, sitting in the inbox with no connection to the message it answered. Reply to that, get another new conversation. Three emails in, the thread was confetti.

Nothing was broken in any log. The emails sent fine, delivered fine, read fine. They just refused to be a conversation.

Two bugs wearing one symptom

The stack: SES receives mail for our reply domain, dumps the raw message to S3, a Lambda parses it and POSTs a normalized payload to the ingestion service, which kicks off a durable workflow that drafts and (after approval) sends the reply. The inbound payload had everything: sender, subject, body, and the RFC822 Message-ID.

Bug one: the workflow never received the subject or the Message-ID. The webhook dutifully stored both in the database, then started the workflow with just name, email, and body. So the reply went out with no In-Reply-To and no References header at all. Gmail had literally nothing to attach it to.

Bug two is the one worth a blog post. The reply draft comes from Claude as a JSON object, and the schema I gave it looked like this:

{ "replySubject": string, "replyBody": string, ... }

See the problem? I asked the model for a subject line. The model, being a model, wrote one. A good one, even. Friendly, specific, on-brand. And since the prompt only contained the thread bodies, it had never seen the original subject it was supposed to preserve. It wasn't hallucinating. It was answering the exact question I put in the schema.

Gmail wants both

Here's the part that makes this worth writing down. I fixed the headers first and assumed I was done, because everything you skim about email threading talks about References and In-Reply-To. That's the RFC 5322 story, and clients like Thunderbird or Mutt live by it.

Gmail doesn't. Gmail threads on the reference headers and the subject line. Same References, different subject: new conversation. Google says so themselves, in the fine print of their threading documentation: correct References and In-Reply-To headers, and then, as its own separate requirement, "The Subject headers must match."

So a reply with perfect headers and a creative subject still splits the thread for every Gmail and Google Workspace user, which for a small-business audience is roughly everyone. The model wasn't allowed to be creative here. Nobody had told it that, because nobody had told me that.

The fix

The subject of a reply is not content. It's protocol. So it moved out of the model's hands and into a ten-line function:

/** Strip any stack of reply/forward prefixes ("Re:", "RE:", "Fwd:", ...). */
function stripReplyPrefixes(subject: string): string {
  return subject.replace(/^(\s*(re|fwd?|aw|sv)\s*(\[\d+\])?\s*:\s*)+/i, "").trim();
}

export function threadReplySubject(original: string | null | undefined, fallback: string): string {
  const stripped = original ? stripReplyPrefixes(original) : "";
  return stripped ? `Re: ${stripped}` : fallback;
}

The prefix stripping matters more than it looks. Clients stack prefixes (Re: Re: Re:), localize them (AW: from German Outlooks, SV: from Scandinavian ones), or number them (Re[2]:). Normalize to a single Re: and Gmail is happy.

The model still returns replySubject, but it's demoted to a fallback: it's only used when a lead arrives through the web form, where there's no email thread to preserve and someone has to write a subject for the first message. For anything that arrived as an email, the subject is derived, end of discussion.

And the Message-ID now rides through the whole pipeline into the send. With SES v2 that's the Headers field on the simple content type (supported since well before my @aws-sdk/client-sesv2 3.1057):

Content: {
  Simple: {
    Subject: { Data: subject },
    Body: { Text: { Data: text }, Html: { Data: html } },
    Headers: [
      { Name: "In-Reply-To", Value: inboundMessageId },
      { Name: "References", Value: inboundMessageId },
    ],
  },
},

One more detail that would have bitten later: the approval gate. A human reviews every outgoing reply before it sends, and the review screen was showing the model's subject, not the one that would actually go out. The subject is now computed before the gate, so the operator approves the real email. If a value gets rewritten after approval, the approval didn't cover it.

Takeaway

When you put an LLM inside a pipeline, every field in your output schema is a small delegation of authority. replyBody was a reasonable thing to delegate. replySubject wasn't, because the subject line of a reply isn't prose, it's a protocol field with one correct value, and asking a text generator for it guarantees you'll get text instead of the value. The model did nothing wrong. The schema did.

Building an email pipeline around an LLM? Send me the details and I'll take a look.

• Email
• Llm
• Aws