Durable Object alarms: stale decisions, moving deadlines

We found that we could erase a learner’s new ring while checking whether another had timed out. After reading a map from a Durable Object, we awaited the database and wrote back an older copy. Even with a fresh read, we could still apply a stale decision. In the same object, we kept reminders alive by measuring age from a deadline we moved at every check, until their history grew and, on 6 August 2026, writes failed in production. We now check each decision against the ring we evaluated and measure age from a fixed anchor. During review, we found the race; we have not measured whether it affected a real booking.

A ring starts when a learner on TEEI Language taps a volunteer who is available now. The volunteer’s screen rings for two and a half minutes, as configured on 15 October 2026, and if nobody answers, we cancel the session and tell the waiting learner. We keep that time with an alarm on a Durable Object. Our overview of how TEEI runs its platforms on Cloudflare Workers tells the short version of both mistakes.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

One object, one alarm, two maps

TEEI Language and TEEI Mentorship are one codebase deployed as two Workers, and each Worker has one LivePresence Durable Object per environment. The class is SQLite-backed and uses the key-value API for exactly two values. rings maps each booking to the ring in progress, with a claimedAt time, an expiresAt derived from it, an outcome that starts as 'pending', and display fields such as learnerName. rescueWatches holds reminders that wake the object when a participant in a scheduled session has not arrived. Each map is stored as one value, shared by every booking in it.

A Durable Object has one alarm: “Each Durable Object is able to schedule a single alarm at a time by calling setAlarm()”, and a second call overrides the first. We share it between both maps: we choose the earliest candidate and replace a future alarm only with an earlier one, so re-arming one kind of work cannot postpone the other. During alarm(), we read null from getAlarm() unless we have scheduled another alarm since the handler began, because an alarm counts as set only until it starts. With the values as of 15 October 2026, we give an expired pending ring a retry candidate five seconds from now.

The alarm has “guaranteed at-least-once execution”, and “Only one instance of alarm() will ever run at a given time per Durable Object instance”. One instance of the alarm is not one event at a time, though. The state API says: “A Durable Object can run an alarm concurrently with another request”.

Storage-only work keeps the merge together

While a storage operation runs, the object’s input gate, as the glossary defines it, defers every other event until the object has stopped executing JavaScript and is no longer waiting for storage. So after a storage read completes, synchronous code continues without another event interleaving. Rules of Durable Objects puts the boundary in one line: “Input gates only protect during storage operations. Non-storage I/O like fetch() or writing to R2 allows other requests to interleave, which can cause race conditions”.

The same read, change and write is therefore safe in one handler and unsafe in another. When a learner claims a volunteer, the claim handler reads the rings map, adds the new ring and writes the map back, and every await in it is a storage operation. The alarm did the same thing with a database call in the middle.

How the alarm lost a ring

Before 12 September 2026, the alarm read the rings map once. For every pending ring past its expiry, it asked the database to settle the ring and then tried to deliver the missed-call notification to the volunteer early. Then it wrote the whole map back. The input gate is open across both calls, and one object serves the whole environment, so anything another request wrote to the rings map during them was erased by the alarm’s older copy.

Two shapes were possible. A ring claimed for a different booking while the alarm waited vanished. The claim had armed an alarm, but only the alarm cancels an unanswered booking on timeout, so without the stored ring, a later alarm run could not time the booking out. The other shape was the same booking: a ring answered while the alarm waited could be overwritten with the alarm’s older decision.

We did not find this in production. It came out of a review of our test model on 11 September 2026, was confirmed in the code, and was fixed the next day.

A fresh read is not enough on its own

The obvious fix is to re-read the map just before writing it. That fixes the first shape: a ring claimed for another booking is in the fresh copy, so it survives. It does not fix the second. The alarm’s decision for the ring it evaluated would still be applied to whatever that key now holds, and a newer stored answered outcome would be overwritten with the alarm’s stale timeout decision.

The author of the fix checked this by putting the bug back two ways. With the whole-map write restored, two tests failed. With the fresh read kept and only the generation check removed, one test failed: the one in which the ring is answered during the alarm’s await. In the commit’s words, the check is “independently load-bearing: a fresh read alone would have passed the cross-key case while still losing a durable answer.” We did not repeat the author’s September runs.

Decisions carry the generation they were taken from

We changed how we commit the alarm’s decisions. Instead of editing the map read before the external await, we collect decisions, each paired with the ring we evaluated, and commit them against the map storage holds by then.

Adapted for publication. Shortened: the write-freeze check and the rescue-watch drain at the top, logging, error reporting, messages to open sockets and the early notification delivery are removed, and the pruning comment is rewritten to say only what the code enforces. Nothing is renamed.

override async alarm(): Promise<void> {
  const rings = await this.readRings();
  const now = Date.now();
  // bookingId -> what was evaluated, and what to write (null deletes). Never applied blindly.
  const decisions = new Map<string, { evaluated: RingState; next: RingState | null }>();
  for (const ring of Object.values(rings)) {
    if (ring.outcome === 'pending' && now >= ring.expiresAt) {
      let authority;
      try {
        authority = await settleSnapUnansweredCancellation(this.env, ring.bookingId, 'timeout');
      } catch (err) {
        continue;
      }
      if (authority === 'pending') continue;
      if (authority === 'inactive') {
        decisions.set(ring.bookingId, { evaluated: ring, next: null });
        continue;
      }
      const next = applyRingOutcome(ring, authority);
      if (!next) continue;
      decisions.set(ring.bookingId, { evaluated: ring, next });
    }
  }
  // Prune terminal rings more than an hour after their claim. Deletion is also a decision:
  // apply it only if the stored generation still matches the ring evaluated here.
  for (const [id, ring] of Object.entries(rings)) {
    if (ring.outcome !== 'pending' && now - ring.claimedAt > 60 * 60 * 1000) {
      decisions.set(id, { evaluated: ring, next: null });
    }
  }
  await this.commitRingDecisions(decisions);
  await this.armAlarm();
}

The alarm does not choose an outcome; the database’s answer is the authority. 'timeout' includes a cancellation committed by an earlier attempt whose response was lost, 'answered' or 'declined' means something else got there first, and 'inactive' means the booking is no longer a live ring. If the call throws or answers 'pending', we record no decision; if the ring is still pending when we re-arm, it contributes the five-second retry candidate. We catch the failure ourselves rather than lean on Cloudflare’s retries, which only an exception escaping alarm() triggers.

With no decisions, we return without reading. Otherwise we read the map once and write it at most once; every await is a storage operation:

Verbatim excerpt.

  private async commitRingDecisions(
    decisions: Map<string, { evaluated: RingState; next: RingState | null }>,
  ): Promise<void> {
    if (decisions.size === 0) return;
    const fresh = await this.readRings();
    let changed = false;
    for (const [bookingId, { evaluated, next }] of decisions) {
      if (!ringGenerationMatches(fresh[bookingId], evaluated)) continue;
      if (next === null) delete fresh[bookingId];
      else fresh[bookingId] = next;
      changed = true;
    }
    if (changed) await this.writeRings(fresh);
  }

We still read and write the whole map. The input gate keeps the read, synchronous validation and write together; non-storage I/O would reopen the window after validation. We reject stale decisions through the generation check and preserve other entries through the fresh map.

What “still the same ring” means is one function:

Verbatim excerpt.

export function ringGenerationMatches(
  fresh: RingState | undefined | null,
  evaluated: RingState,
): boolean {
  if (!fresh) return false;
  return fresh.claimedAt === evaluated.claimedAt
    && fresh.expiresAt === evaluated.expiresAt
    && fresh.outcome === evaluated.outcome;
}

Cloudflare’s advice here is “optimistic locking (check-and-set) patterns: read a version number before the external call, then verify it has not changed before writing.” We use claimedAt, expiresAt and outcome instead: a claim sets both times once and nothing rewrites them, and applyRingOutcome changes the outcome only from pending. Because we derive expiry from claim time, it adds an invariant check, not an independent identifier; detecting a replacement still requires a different claim timestamp. A unit test pins learnerName outside the generation, so a display change is never read as a new ring. We write a ring derived from the evaluated copy, so an excluded field must stay unchanged or keep its fresh value. If your records change in place, use a version that every write increments.

Why not block concurrency

blockConcurrencyWhile() would have closed the window completely, and Cloudflare advises using it “sparingly” because “it significantly reduces throughput”. Our object is one per environment and holds every live socket, so the author of the fix ruled it out: blocking would hold back every socket message for as long as the database call and the notification delivery take. In our judgement, the choice turns on which events can wait: with an object per room, document or user, we still block that object’s other events for the external call.

Settling a ring twice is safe

Cloudflare also asks for idempotent alarm handlers: “In rare cases, alarms may fire more than once.” We settle the booking in the database first and update the ring in the object’s storage afterwards, in two separate commits. The database call cancels only a booking that is still accepted and unanswered, and “The write-once cancellation timestamp lets an alarm retry distinguish ‘my prior cancel committed’ from an unrelated terminal state.” When the update changes nothing or its response is lost, we read the persisted marker and outcome instead of inferring failure, with no compensating write. If the object fails between the commits and the ring remains pending, a later alarm asks again; the generation check applies the answer only to the state we evaluated.

We record the missed-call notification as a durable job in the same database transaction as the cancellation. The insert may do nothing on a retry, so a separate statement checks that the persisted timeout has the matching job before the batch can commit. The alarm only attempts an early delivery; a failure there is left for the scheduled job. What we claim here covers settling a ring and that notification job.

A test double that shared a reference

We missed the race because of the fake storage our tests ran against. Its put cloned the value, but its get returned the stored object itself. When a test called the claim handler from inside the alarm’s await, both handlers held the same object, so the new ring appeared in the alarm’s map by reference. In memory there was only one map, so nothing was ever lost. A control test written earlier to catch this very defect passed against the broken product code on that harness; we could have mistaken that pass for proof of a fix. So we repaired the harness first:

Adapted for publication. The comment is shortened to its first sentence, and the declaration of a second store above the excerpt is removed.

  const storage = {
    // CLONE ON READ, not just on write.
    get: vi.fn(async (key: string) => (
      key === 'rings' ? structuredClone(rings) : structuredClone(otherKeys[key]) as unknown
    )),
    put: vi.fn(async (key: string, value: unknown) => {
      if (key === 'rings') rings = structuredClone(value) as Record<string, RingState>;
      else otherKeys[key] = structuredClone(value);
    }),

We model each read as an independent copy; we have not verified that behaviour in the real runtime. Cloudflare’s storage documentation says that put accepts “any type supported by the structured clone algorithm”, not that each get returns a new copy.

With the harness repaired and the product code unchanged, two new tests failed. With the fix, in the author’s run on 12 September 2026, 208 tests passed, and the 48 presence tests that existed before passed unchanged against the cloning harness. The first new test claims a second booking from inside the mocked database call, exactly where the input gate would be open:

Adapted for publication. Three fields of the claim that name its participants are removed.

  it('keeps a ring claimed while the alarm was awaiting the database', async () => {
    const run = harness(expiredRing());
    h.settle.mockImplementationOnce(async () => {
      await notify(run, {
        kind: 'claimed',
        bookingId: 'second-booking',
        claimedAt: 2_900,
      });
      return 'timeout';
    });

    await run.presence.alarm();

    expect(run.rings()['second-booking']).toBeDefined();
    expect(run.rings()['second-booking']?.outcome).toBe('pending');
    // The alarm's own decision still lands for the ring it actually evaluated.
    expect(run.rings()['snap-booking']?.outcome).toBe('timeout');
  });

The second delivers an answer inside the await while the mocked database still says 'timeout'. Real database authority would have said 'answered', so the test stages a conflict production would not produce; it pins the rule that a decision never undoes a newer outcome. Both run in Node against hand-written fakes, under a model in which non-storage awaits let other events in; they do not exercise the real gate or real storage. Cloudflare’s testing guide for Durable Objects describes runInDurableObject() and runDurableObjectAlarm() in the Workers pool for Vitest, which run against the real runtime; we have not moved these tests there.

A watch that never grew old

The second mistake was a rule about time. We create a watch when the first person arrives in the room for a booking, with its first boundary as dueAtMs. At that boundary, the alarm calls code that reads the booking and who is present from the database and alerts the missing participant if needed. With the configuration as of 15 October 2026, we then record the boundary as evaluated and advance the watch five minutes to the next one, and we measure retirement from its first boundary, with a one-hour threshold.

Until 6 August 2026, the retirement rule compared the current time with dueAtMs, the boundary being processed, and only then stored the next boundary. When the alarm fired on time, the current time was that boundary, so the difference was zero at every run. A drain more than an hour late would have retired the watch; on-time drains kept it alive, adding one number to its list every five minutes for as long as the object lived.

On 6 August 2026, the object’s production writes began failing with SQLITE_TOOBIG on a Worker version unchanged for five days; as the fix’s author put it, “the data grew.” All watches shared one stored value, and “since pruning also needs a write, nothing could shrink it back.” We did not measure the stored value’s size, identify the limit it hit or establish why the errors stopped. Cloudflare’s limits page, as documented on 15 October 2026, gives 2 MB for a key and value combined on SQLite-backed storage and 128 KiB per value on the older key-value backend; our sources do not establish which applied. The errors stopped before the fix was committed, and more than an hour after the commit our notes still recorded it as not deployed, so we do not credit the fix with ending the incident.

A fixed anchor, and a cap applied on read

We stamp anchorMs once, from the first boundary the watch is armed for, and never rewrite it. If a second person opens the room and pulls the boundary earlier, the watch takes the earlier boundary and keeps its anchor: in the commit’s words, “an earlier boundary from a second arrival wins the alert but buys no extra life.” Retirement measures age from the anchor:

Adapted for publication. This is the end of one pass of a loop over the watches. A comment about how the next boundary is chosen and the write after the loop are removed, and one phrase in the remaining comment is made more precise.

      const evaluated = [...watch.evaluatedDueAtMs, watch.dueAtMs].slice(-MAX_EVALUATED_BOUNDARIES);
      const nextDueAtMs = watch.dueAtMs + 5 * 60_000;
      // Age is measured from the fixed anchor, NEVER from dueAtMs, which steps forward on
      // every drain. Against a moving target the difference stays near zero for as long as the
      // alarm keeps firing on time, so the watch never retires and its list never stops growing.
      if (nowMs - watch.anchorMs > RESCUE_WATCH_TTL_MS) delete watches[bookingId];
      else {
        watches[bookingId] = {
          bookingId,
          anchorMs: watch.anchorMs,
          dueAtMs: nextDueAtMs,
          evaluatedDueAtMs: evaluated,
        };
      }

The rings never had this problem. A finished ring becomes eligible for pruning once more than an hour has passed since claimedAt, which is written once; we schedule no separate timer for it and prune it when a later alarm run reaches cleanup. The watches measured from a field that existed to move. When a record has to expire, ask whether your own code ever changes the timestamp you measure from. If it does, the record can stay young forever.

The anchor bounds how long a watch lives, not how many boundaries it collects. A separate cap does that, also for watches already stored. On read, before the alarm processes a watch’s due boundary, we keep the last 24 numeric entries of its history (the cap as of 15 October 2026), in evaluation order, not necessarily timestamp order. We still fetch the complete stored map and filter each complete history first, so we bound the retained history, not the amount read.

Adapted for publication. The two comments are rewritten to say only what the code enforces.

      const evaluated = watch.evaluatedDueAtMs.filter((at) => typeof at === 'number');
      watches[bookingId] = {
        bookingId: watch.bookingId,
        // Reconstruct a missing anchor before truncating the stored history. The first
        // entry is the fallback, not necessarily the minimum.
        anchorMs: typeof watch.anchorMs === 'number' ? watch.anchorMs : (evaluated[0] ?? watch.dueAtMs),
        dueAtMs: watch.dueAtMs,
        // Bound each retained history independently of retirement.
        evaluatedDueAtMs: evaluated.slice(-MAX_EVALUATED_BOUNDARIES),
      };

A watch written before anchors existed takes the first numeric entry of its untrimmed history, or its current boundary if that history is empty, so it needs no migration; taken from the trimmed tail, the anchor would be a later boundary. A watch that had been growing for days gets an anchor days old and retires the next time the alarm processes its due boundary; the reconstructed anchor exists only in memory until a later write persists it. No surviving watch is written back with more than 24 boundaries. In the commit’s words, “trimming on READ is what lets a deploy recover the object instead of the stored value having to be purged by hand”; that is the design and what the tests show. Neither step guarantees on its own that the combined map fits under the limit, and we have not shown recovery against a real size rejection, because our fake storage refuses nothing.

Drain until it is gone

The test that guards the anchor runs the alarm at every boundary, exactly on time, until the watch disappears:

Adapted for publication. An assertion on the stored anchor and the comment above the loop are removed.

  it('retires a watch an hour after the boundary it was first armed for', async () => {
    const anchor = T0 + 3 * 60 * 1000;
    vi.spyOn(Date, 'now').mockReturnValue(T0);
    const lp = harness();
    await lp.arm('booking-1', anchor);

    let drains = 0;
    let longestList = 0;
    while (lp.watches()['booking-1'] && drains < 100) {
      vi.spyOn(Date, 'now').mockReturnValue(lp.watches()['booking-1'].dueAtMs);
      await lp.presence.alarm();
      drains += 1;
      const still = lp.watches()['booking-1'];
      if (still) longestList = Math.max(longestList, still.evaluatedDueAtMs.length);
    }

    expect(lp.watches()['booking-1']).toBeUndefined();
    // TTL / STEP boundaries plus the drain that crosses the deadline.
    expect(drains).toBe(TTL_MS / STEP_MS + 2);
    // The anchored TTL is what bounds the list; the read-side ceiling never has to bite.
    expect(longestList).toBeLessThan(MAX_EVALUATED);
  });

In our on-time test of 15 October 2026, we retire the watch on run fourteen, 65 minutes after its anchor, with at most thirteen boundaries retained. A short list is not evidence of expiry: a cap can bound the history while leaving the watch alive indefinitely, so the test checks that the watch disappears and the exact number of drains. With the old comparison, the 100-drain guard would stop the loop while the watch is still present, so the absence assertion would fail. The author put the old comparison back on 6 August 2026 and saw two of the four new tests fail; we did not repeat that run. Two more tests start from a 5,000-boundary fixture, not a production measurement, and check that the first drain clears it and that a surviving watch writes small.

When this is the wrong pattern

In our judgement, a skipped decision has to be cheap to take again. We can ask the database again for a ring that remains pending after we skip a stale decision, and a ring someone else settled needs nothing from us. If your decision has already caused a side effect outside the object that cannot be repeated or undone, that side effect has to be idempotent or recorded with the decision.

One value per map is the whole map every time. For rings, each change reads and writes the whole map. Keeping a map as one value is also why one record’s growth could stop every write in the object. We considered one storage key per booking and have not done it. Our thresholds limit each entry, not the number of entries; if yours cannot be bounded that way, or there can be many, separate keys or the object’s SQL tables are the better shape.

Two jobs on one alarm share their failures. The timeout and the watches wake on the same alarm, and the watches run first. So a malformed watch value “must degrade to ‘no watches’, never throw and take the ring timeout down with it”, and a failure in a watch’s work is caught, logged and left for the scheduled job. After a failed drain we still advance or retire the watch; the evaluated list records attempts, and delivery state in the database decides what is retried. If your second job can fail in ways you do not control, give it its own object.

Verify this

Two reproductions you can run in your own project:

• The test double. In a Vitest fake for ctx.storage, make get return the stored object and call a second handler from inside a mocked await. The race test passes against the broken code. Make get return structuredClone of the stored value, and it fails until the code is fixed.
• The moving deadline. Compare now with a deadline you step forward on each call, run it exactly on time at each step, and count the steps until it retires. It never does. Then measure from a fixed anchor.

What we ran for this note

On 15 October 2026 we ran four selected test files for this state machine: presence, rescue watches, write freeze and the protocol unit tests. All 52 tests passed in 19.8 seconds on Vitest 4.1.11, on fakes in Node. The constants above are from the source revision tested that day; results from the original authors’ runs are marked as theirs.

Real-time availability with Durable Objects describes the same object from the booking side. The missed-call notification follows the shape of our email outbox behind Cloudflare Queues: the decision and the duty to tell someone are written together.