Why We Freeze Email Payloads on Cloudflare Queues

We gave every email an idempotency key. Before 29 September 2026, if we lost the provider’s response and then changed the email layout, we would retry with different bytes under the same key. Resend would answer with a 409, and we would mark the email as failed, even though the first attempt might have reached the recipient. We now store the first attempt’s message fields before sending and reuse them on retry when the payload store is available. The key protects a retry only when the request carries the same content and arrives within the provider’s idempotency window.

Around that, we use a transactional outbox on Cloudflare Queues: we record the intent in the database, publish the outbox reference with its message kind, version and optional tracing metadata, and use an atomic claim to admit a delivery while its claim lasts.

We built it for Evenhall, where a partner company’s managers see what their employees did through the programmes of The Educational Equality Institute (TEEI), a nonprofit, and turn it into a report they can file. TEEI’s free programmes connect people from Ukraine, and women from Latin America and the Caribbean, with volunteers who help them practise a language and build a career. Evenhall is a React Router 8 application on a Cloudflare Worker, with a Turso (libSQL) database and Resend for email, and we send email when a manager invites employees, a company applies or someone asks for a demo. Each of those emails crosses three systems that cannot share one transaction: our database, the queue and the email provider. The overview of how TEEI runs on Cloudflare Workers has a short section on its queues.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

One invitation, from a manager’s click to the hours in a report

The invitation is the email this note keeps returning to, and the step-by-step table further down follows one through a deploy. It is also where an employee’s volunteering with TEEI starts.

A manager at a partner company invites colleagues from Evenhall by email. The email says what they will do (take part in the TEEI programmes the company supports) and how their time is counted: hours are measured from session attendance, so there is nothing to log. After accepting, each colleague creates a volunteer account in TEEI Language or TEEI Mentorship. In TEEI Language, volunteers set the times they are free and learners book sessions inside them. Hours from the sessions they hold are what the company’s report in Evenhall counts. How corporate volunteering works at TEEI follows that journey from the employee’s side.

That is why the failures below matter to us. If the invitation arrives twice, the first thing a colleague sees from the programme is a duplicate. If it reached the inbox but is recorded as failed, our record is wrong about whether someone was asked to start.

What the invitation starts looks like the work of Heidi B., a volunteer language partner in TEEI Language who comes to every session with a topic ready.

Evenhall is not TEEI’s only sender on Resend. Of the emails TEEI Language and TEEI Mentorship logged in the seven days to 01:30 UTC on 17 October 2026, 4,077 were recorded as sent through Resend, among them session reminders and booking confirmations, and 4,054 had a delivery receipt from Resend. Those platforms have a send path of their own, which this note does not describe.

Three ways to send an email twice, or not at all

The queue delivers at least once. Cloudflare’s delivery guarantees say that a message, “in rare occasions, may be delivered more than once”. If we send once for every delivery, a duplicate delivery can produce a duplicate email.

The database and the queue cannot share a transaction. Publishing first can leave a message without its database record; writing first can leave a committed record whose message was never published. That is the problem the transactional outbox pattern exists to solve.

The provider’s answer can be lost. Resend may accept an email and the response may never reach us. On retry, we need the provider to reuse an accepted request or send once if the first request never landed. Resend’s idempotency keys are built for that, on two conditions: the retry must carry the same payload, and it must arrive within the provider’s window, 24 hours in the documentation we read on 15 October 2026.

These are separate failure boundaries. An outbox transaction records an obligation; a relay makes that obligation eligible to run; a claim controls concurrent attempts; provider idempotency reconciles repeated requests; a receipt tells us about delivery after acceptance. None of those steps substitutes for another.

The transaction below is a demo-request example. Our email families do not all have the same atomic write and recovery coverage, so this is not a claim of eventual delivery or exactly-once effects across every flow. We describe the guarantees at the boundary that implements them.

The row is the truth; the message points to it

We keep contact details in the database and publish this queue envelope:

Verbatim excerpt.

/**
 * Deliberately contains no contact details. Queue payloads are visible in
 * Cloudflare operations tooling, so the consumer resolves all PII from Turso.
 */
export type CsrEmailQueueMessage = Readonly<{
  kind: "csr-email-outbox";
  telemetry?: { sentryTrace?: string; requestId?: string };
  outboxId: string;
  version: 1;
}>;

The row holds the delivery record: the recipient, the template and its version, the idempotency key, the provider’s message ID once there is one, the attempt count, the last error and a status. Three database guarantees sit under it. A unique idempotency key prevents a second row under the same key, and the conditional UPDATE further down prevents two concurrent claims. A partial unique index prevents one provider message ID from identifying two rows.

Keeping the message this small keeps personal data out of the queue, so nothing in Cloudflare’s queue tooling or metrics shows who was emailed. It also leaves the row as the one record of what has been sent: when a duplicate or late message arrives after the row has settled, we read the row and do nothing. Before sending, we read the outbox row from the database. Our Workers are built around round trips to one database, and why they run near the database rather than near the user is a note of its own.

Writing the row in the same transaction

The outbox only helps if we write the row in the same transaction as the change that caused the email. For a demo request, we record the lead, store the submitted form and insert the outbox rows in one write batch, and we publish only after the batch commits:

Adapted for publication.

await db.batch(
  [
    upsertLead,          // INSERT ... ON CONFLICT(email) DO UPDATE
    insertSubmission,    // INSERT INTO formSubmissions ...
    ...buildEmailOutboxInsertStatements({ entries: outboxEntries, formId, now }),
  ],
  "write",
);

// Only after the commit: publish one envelope per outbox row.
const enqueueResult = await enqueueEmailOutbox(env, outboxEntries);
if (!enqueueResult.queued) {
  logger.warn("Demo contact email outbox remains pending", {
    errorCode: enqueueResult.errorCode,
    formId,
  });
}

We run the statements in one transaction using libSQL’s batch in "write" mode, so the lead, the form and the email intent exist together or not at all. We insert into the outbox with ON CONFLICT(idempotency_key) DO NOTHING, so replaying it cannot create a second row. We still await the transaction and publication attempt in the HTTP request; we move only the provider call off it.

We handle company applications differently. When we decline an application or ask for more information, we record the decision and insert an email intent in one transaction, with INSERT OR IGNORE so that a repeat does nothing. We retry acquiring a busy write transaction a few times, before running any application statements; after that, an ambiguous commit error goes back to the caller. The helper’s comment says why: “Never replay an ambiguous commit/network failure as a fresh transaction.” In the code as of 15 October 2026, a relay that runs on a schedule then writes the outbox row, its stored message input and the intent’s queued marker in one transaction, and only then publishes; in the same pass, it publishes rows that were committed and never published.

Publishing is the step that is allowed to fail. In enqueueEmailOutbox we call sendBatch and then set the rows to enqueued, three tries with a short doubling pause. We update that state only while a row has no provider message ID and is still pending, enqueued or retry, so a late update cannot overwrite a claim or a settled status.

If all three tries fail, or the queue binding is missing, we record the eligible rows as pending with the reason and return { queued: false }; only a failure to write that state reaches the caller. We may have published the message before the state update failed, so we cannot infer that nothing reached the queue. We may therefore publish a row twice, on a second try or from a relay; we must handle duplicate deliveries in the consumer anyway.

The consumer: one decision per message

In our configuration, Cloudflare can deliver several email messages in one invocation, and we handle them one at a time. Two rules from Cloudflare’s batching documentation shape the loop. If one message fails and nothing in the batch has been acknowledged, “the entire batch is retried”; when a message receives both retry() and ack(), “The first method call wins in all cases.” So we decide each message on its own, exactly once, and never call retryAll():

Adapted for publication.

export async function consumeEmailQueue(batch: MessageBatch<EmailQueueMessage>, env: Env) {
  const isDeadLetterQueue = batch.queue === env.EMAIL_DLQ_NAME;

  for (const message of batch.messages) {
    if (!isQueueMessage(message.body)) {
      // Invalid envelope: acknowledge without reading or updating an outbox row.
      message.ack();
      continue;
    }
    try {
      if (!isClaimedOutboxId(message.body.outboxId)) {
        // The envelope carries an ID, but a matching row may not exist.
        // Mark an eligible row failed first; if this write throws, the message retries.
        await markUnclaimedOutboxRow(env, message.body.outboxId);
        message.ack();
        continue;
      }
      if (isDeadLetterQueue) {
        await markDeadLetter(env, message.body);
        message.ack();
        continue;
      }
      const result = await processOutboxJob(env, message.body, message.attempts);
      if (result.kind === "retry") message.retry({ delaySeconds: result.delaySeconds });
      else message.ack();
    } catch (error) {
      captureError(error, { attempt: message.attempts });
      message.retry({ delaySeconds: retryDelaySeconds(message.attempts) });
    }
  }
}

We ask two questions at the top, and they look like one: is this message ours at all, and which kind of email does its ID belong to? We acknowledge an invalid envelope without touching the database. For a valid envelope whose ID no family recognises, we mark an eligible row failed before we acknowledge. That order comes from the incident further down.

We consume the dead-letter queue in the same Worker. We pass exhausted deliveries to the relevant email-family handler, then acknowledge after it returns. One Worker for both queues keeps routing simple, and means one broken deploy affects both. The main queue retries a message a bounded number of times before it moves to the dead-letter queue, and we set each delay ourselves with retry({ delaySeconds }), as in the loop above; the queues have no retry delay of their own configured.

message.attempts starts at 1, which the delay in the loop relies on. On our exponential retry paths the delay doubles with each attempt and stops growing at a cap, so a message’s retries can spread over a few hours and a temporary provider outage has time to clear. We have not tested this against a real outage. Other paths use shorter, fixed delays.

We also cap how many consumer invocations may run at once, while Cloudflare recommends leaving that setting unset so that the consumer can scale. The reason for our setting is not written down, and we have not measured whether it ever binds.

How we claim a row with one UPDATE

Before sending anything, we claim the row. We use one conditional UPDATE to decide which delivery may send, here as it stood on 15 October 2026:

Adapted for publication.

async function claimOutbox(
  env: Record<string, unknown>,
  outboxId: string,
): Promise<boolean> {
  const db = getClient(env);
  const now = new Date();
  const claimExpiresAt = new Date(now.getTime() + CLAIM_WINDOW_MS).toISOString();
  const result = await db.execute({
    sql: `UPDATE csr_email_outbox
          SET status = 'processing',
              attempt_count = attempt_count + 1,
              claim_expires_at = ?,
              next_attempt_at = NULL,
              updated_at = ?
          WHERE id = ?
            AND provider_message_id IS NULL
            AND (
              status IN ('pending', 'enqueued', 'retry')
              OR (
                status = 'processing'
                AND (claim_expires_at IS NULL OR claim_expires_at <= ?)
              )
            )`,
    args: [claimExpiresAt, now.toISOString(), outboxId, now.toISOString()],
  });
  return result.rowsAffected === 1;
}

We use these conditions to prevent the failures above. Once we have stored the provider’s message ID, the row cannot be claimed again, however many times its message arrives. Only a waiting row can be claimed, and pending is on the list, so a row whose publication was never recorded can still be sent. The expiry means a Worker that died in the middle of a send does not hold the row for ever.

When two deliveries compete for a claim, we let both run the conditional UPDATE; only one changes the row. We test eligibility and change the row in the same statement. A SELECT before it could show both deliveries a pending row, and it would reserve the row for neither.

The claim alone does not make an email go out once. We admit one delivery through the conditional UPDATE while its claim is valid. After the claim expires, another delivery may take over even if the first is still waiting for the provider. We use the idempotency key to reconcile attempts whose outcome we could not record, including a lost response or an earlier request still waiting when its claim expired.

Freezing the payload before the first attempt

Every send carries the idempotency key stored on its outbox row. Resend’s documentation, read on 15 October 2026, says what that buys: for 24 hours, the same request with the same key gets “the same response, without actually sending the email again”. It also documents the condition, as a 409 with the code invalid_idempotent_request: “this idempotency key has already been used on a request that had a different payload.”

Before the fix, we sent a newly rendered message on every attempt, which is the same request only while the rendering never changes. The sequence that breaks it is a lost response followed by a deploy. The first attempt reaches Resend, Resend accepts it, the answer never arrives, and the row goes to retry. Before the retry, we deploy a new version of the shared email layout.

On retry, we render the new layout, present different bytes under a key Resend already holds and get a 409. Before the fix, we then wrote the row as failed, which says that nothing was sent, while the first attempt may well have reached the recipient. Our campaign emails could meet the same 409 without any deploy, because each render of a campaign email produces slightly different bytes.

The fix landed for report emails on the morning of 29 September 2026 and for invitations, demo requests, applications and campaigns two hours later. We documented the hazard in the fix’s description without recording an observed case, so we do not know how often it happened.

Here is the right-hand timeline for one invitation like the one at the top, step by step, as our model test runs it.

Before the fix, step 4 sent fields B and step 5 was a 409, which we wrote as failed. The test checks the result on both sides of the provider call: the second request has the same fingerprint as the first, its HTML has no trace of the new layout, and the model has sent one email. That fingerprint covers the fields we hand to the provider wrapper (sender, recipient, reply-to address, subject, HTML, text and tags), before tag cleaning and the environment tag; it is not a fingerprint of the final HTTP request, and it leaves out the key and any attachment. It does not assert the key on its own; in the model, only the same key can return the original message ID. The same test runs for demo requests, applications and campaigns, and a separate one for report emails.

We freeze like this:

Adapted for publication.

/**
 * Call after the consumer's checks on the row, immediately before the
 * provider wrapper. Returns the payload of the FIRST attempt when one was
 * stored, and otherwise stores `rendered` and returns what was stored (which is
 * another attempt's payload if it froze first).
 */
export async function freezeOrReplayOutboxMessage(db, outboxId, rendered) {
  // The store holds one attachment. No email family sends more; one that did would keep rendering.
  if ((rendered.attachments?.length ?? 0) > 1 || !(await frozenPayloadStoreAvailable(db))) {
    return { freezeAvailable: false, message: rendered, replayed: false };
  }
  const frozen = await loadFrozenPayload(db, outboxId);
  if (frozen) {
    return { freezeAvailable: true, message: withFrozenPayload(rendered, frozen), replayed: true };
  }
  const stored = await freezePayload(db, outboxId, toFrozenPayload(rendered));
  return { freezeAvailable: true, message: withFrozenPayload(rendered, stored), replayed: false };
}

We freeze the message fields we hand to the provider wrapper: sender, reply-to address, subject, HTML, text, tags and an attachment. We keep the recipient, the idempotency key and the audience from the message we prepare for each email family; freezing does not replace them.

We freeze after the claim and row checks, immediately before the provider call. After INSERT OR IGNORE, we read back and validate the stored row. We use the fields stored first, even if they came from another attempt, and throw rather than send a partial payload.

The guarantee depends on the frozen-payload store being available, and before relying on replay we check that the store has every column the reader needs. Replaying stored fields does not extend the provider’s 24 hours. Our model tests cover sequential retries across a layout change, and they show the later attempt reusing the first attempt’s stored fields.

The claim lifetime and the provider window solve different problems. Claim expiry lets work recover after a consumer disappears; it is not evidence that the earlier provider request failed. The provider window lets an otherwise identical request return its earlier answer; it is not a permanent uniqueness constraint. Once that window has elapsed, reusing the same key cannot by itself justify another send. That is why an ambiguous result needs a durable record and a reconciliation decision, rather than a new key that merely makes the next request easier to accept.

We verify the replayed fields with a model; we do not verify the final HTTP body, where we still clean the tags and add the environment tag in the wrapper on every attempt. The stored payload contains personal content, so in the send path we delete it when the provider accepts the email or returns a final result.

What the provider’s answer means

We handle acceptance, an idempotency conflict and other rejections in separate branches. On acceptance, we record the provider’s message ID on the row under the same provider_message_id IS NULL guard as the claim. A 409 with invalid_idempotent_request can still arrive, for example from a row first attempted before the freeze existed. We write it as dead_letter with unknown_outcome, because it proves that an earlier request with the key reached the provider and says nothing about whether that email went out.

That dead_letter is a row status in our database, not Cloudflare’s dead-letter queue. A 409 with concurrent_idempotent_requests means another request with the key is still in progress, and we retry it. Everything else is either retryable (429, any 5xx, a transport failure, an answer without a message ID) or final.

Acceptance is not delivery. We deduplicate matched receipts with ON CONFLICT(provider_event_id) DO NOTHING, and every later statement for that receipt acts only while it is unprocessed (processed_at IS NULL). We advance the outbox row only when the event is newer by julianday time, or equally new with a status that ranks at least as high as the stored one. After a bounce or complaint, we add the address to the suppression list we check before the next send. We commit the receipt, the guarded updates, any suppression and the processed marker in one write batch, so no processed marker lands ahead of the effects it records.

The email that disappeared behind an acknowledgement

Between 7 and 21 September 2026, report deliveries started as a test send, or as a backfill of a missed run, reached the queue and were never sent. Other families were not affected. On 21 September we caught one in production: we started to process the message, logged an error and acknowledged it in 0 milliseconds. Fourteen hours later the outbox row was still enqueued, with zero attempts.

The first cause was that we built IDs containing a character our recogniser rejected, so we could not tell which family the message belonged to. The second cause is the one that made it last two weeks. We answered an unrecognised ID by logging it and acknowledging the message, and nothing on that branch touched the database. By acknowledging, we deleted the message but left the row enqueued, as if work were still in flight.

In the words of the fix, that is what turned a one-character bug “into a two-week ‘the queue never delivers’”. We fixed it twice on the same day. We now recognise the IDs we build, and we now ask the two questions separately in the consumer, as in the loop above. For a valid envelope whose ID no family recognises, we mark an eligible row failed with outbox_id_unclaimed inside the try, so if that write fails the message is retried rather than destroyed while the row still looks alive.

In the regression test, we run the real consumer loop against an in-memory libSQL database built from the same migration. We test that the row reaches failed, duplicate delivery, rows with a provider message ID and the dead-letter path. If we cannot write the failed status, we verify that we retry and leave the row enqueued. Against the consumer as it was before the fix, three of its first five tests failed.

Queue metrics would not have shown this, because an acknowledged message is deleted from the queue whether or not we did any work for it. For the 30 days to 15 October 2026 at 10:00 UTC, Cloudflare’s sampled analytics estimate that the production email queue had 41 messages written, 41 read and 41 deleted after success, and none moved to the dead-letter queue. We did not measure the queue’s numbers for the incident itself, and the analytics we read have no breakdown by email family. Check queue metrics for message movement, and the outbox row and provider receipts for whether an email went out.

Two smaller incidents, both fixed, involved the provider. On staging on 10 September 2026, Resend rejected application emails because one tag value contained a character outside the set it allows in tags; we now clean tag names and values at the provider boundary. On 28 September 2026, in production we answered 503 to 363 webhook receipts that belonged to staging QA mail. Every email now carries a tag that names its environment, and we skip the other environment’s receipts in each webhook.

When this is the wrong pattern

When you can tolerate loss before publication. If a notification needs no atomic database write and its loss costs nothing, publish to the queue and accept at least once.

When you have no relay. We can let publication fail safely only if we later publish the rows left pending. Without a sweep, we record an email nobody sends. Build the relay with the outbox, not after it.

Our application relay and report recovery are specific implementations, not a universal sweep for the table. Recovery, exhausted-delivery handling and frozen-content cleanup need their own coverage for each email family. We have open work in those areas. The consumer and payload-freezing examples should not be read as evidence that those wider obligations are complete.

When nobody reads unknown_outcome. Writing a 409 as an unknown outcome moves the decision to a person. Someone has to investigate those rows to establish whether the provider sent the email.

When the payload cannot be stored. The frozen payload is personal content in your database for as long as a retry may need it. If you cannot keep it there, you need a rendering that is byte-stable across deploys, which is harder than it sounds. Plan a separate clean-up for rows that settle outside the send path, through a webhook or an unsubscribe.

Verify this

Reproduce the provider’s rule. In your own project, send the same Idempotency-Key twice with two different subjects. Resend’s documentation says the second request gets a 409 with invalid_idempotent_request, while the same payload would get the original response.

Test your consumer against a model of that rule. Give the model two answers: the same key and payload return the original message ID without sending, and the same key with a different payload returns a 409. Let it accept your first attempt and lose the response, change your template, then run your consumer again on the same outbox row.

Assert three things: both requests carry the same fingerprint, the second attempt is accepted with the original message ID, and the model sent one message. A consumer that sends newly rendered fields on retry fails the first two, because its second request is refused; counting sent messages alone would not catch it. Fingerprint the final provider request, after tag cleaning and the environment tag, rather than only the replayed fields.

Extend that test with failures on either side of each acknowledgement. Stop after the domain transaction but before publication, then ask whether a relay finds the row. Let the provider accept but lose its response, then ask whether the same request is replayed. Fail the database write that records acceptance, then ask whether another attempt can reconcile the provider’s answer. Finally, make a status write fail before ack() and verify that the message is retried. Run these against synthetic recipients and a provider model: they test the protocol without sending email. A passing queue invocation alone cannot answer any of these questions.

What we draw on. We read Evenhall’s code as it stood on 15 October 2026, when all 67 tests in the six email test files passed at 10:48 UTC in 1.82 seconds. The queue figures come from Cloudflare’s analytics for the 30 days to 10:00 UTC that day. We did not read any email or participant records from Evenhall’s production database, so the article gives no counts of invitations, applications or delivered emails from Evenhall. The count for TEEI Language and TEEI Mentorship comes from TEEI’s daily email measurement, which counts the rows of a copy of that platform’s email log by provider and status, without reading addresses or content. The phone screens were captured on 17 October 2026 from Evenhall’s public demo and the public tour of TEEI Language, and the invitation was rendered from Evenhall’s template with the demo’s sample names; nothing was sent. On 17 October 2026 we also read the schema of Evenhall’s production database and its migration ledger, and found the frozen-payload store in place there. The same day, we ran the six email test files again together with the report-email test; all 93 tests passed, including the model tests behind the step-by-step table.

These records are TEEI’s own. They are not public, and we do not publish an endpoint for them.