Engineering noteResearch & Engineering
Part of: Cloudflare Workers
Durable Objects for Rings, SQL for Outcomes
We had recorded the volunteer's answer, but still removed the learner from the lesson. Here is where live signals end and durable outcomes begin.
The programme
On this page
We could take a learner out of a lesson after the volunteer had answered. We had recorded the answer in SQL, but asked our Durable Object for the outcome after we had dropped the finished ring. With the socket connected, we did not poll either. We now read the booking first and poll while the learner waits, whether or not the socket is connected. We still use one hibernating Durable Object per environment to hold WebSockets, send rings and start timeouts. We keep settled outcomes in SQL.
The rule behind it took us two failures to learn: we can keep a socket connected after forgetting a finished ring, so we use the Durable Object for signals and pending timeouts and read settled outcomes from SQL. This extends the live-plane section of how TEEI runs its platforms on Cloudflare Workers.
Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.
8 September 2026: the answer was already in the booking
A learner on TEEI Language sees that a volunteer is available now and taps their name. We ring the volunteer’s connected screen, and the volunteer answers, declines or lets it ring out. The learner waits in her room until she hears which.
Until 8 September 2026, the object was the only place we asked about the learner’s ring from her room. We replay a ring from the object only while it is pending, drop a finished ring after a while to keep storage small, and answer “unknown” for a ring we no longer hold. We had also gated the learner’s polling fallback behind liveSocketConnected(), so we did not poll at all for a learner with a working socket. After a learner reloaded, we could no longer show her that the volunteer had answered. We kept waiting for the answer, then removed her from the lesson.
We measured it in production on 8 September 2026. Of 102 times a learner or volunteer attended an “Available now” session, 51 ended early. Of 2,322 attendances in scheduled sessions measured the same way, 14 did. We have not repeated that measurement since the fix, so we cannot yet give an after figure. We had delivered the ring itself; what failed was our reading of its outcome after the object had dropped it.
Since 16 July 2026 we had recorded in the booking’s own columns whether a ring was answered, and whether it was settled as declined or timed out. The volunteer’s answer was written there before they could join the video meeting. We already had the answer in the booking. We had asked the wrong place.
The fix has two halves. In the route the learner’s room polls, we now read the booking’s columns first and return a terminal outcome from them at once, never overridden by the object. For a booking still pending, we next check whether the other person is already in the video room, which counts as answered. We ask the object only when neither has a terminal outcome, and for a pending booking we answer “pending” instead of “unknown”.
As of 15 October 2026, we request the learner’s ring status in the room once when it opens, because an event that fired while the page was loading is gone, and then every 10 seconds while she waits, whether or not her socket is connected. We make that background poll with a fetch we catch locally rather than with fetcher.load, because a failed fetcher request can raise the room’s route error boundary, and a failed poll should mean “keep waiting”, not an error screen. We anchor her local deadline to the original claim, so a reload cannot start a fresh waiting period. At the deadline we ask the server once more; if it still has no terminal answer, we retire the waiting banner and leave the call alone. We never infer a cancellation from silence.
We initialise local ring state from the loader, but revalidation changes loader data without rerunning the useState initialiser, so we copy a loader-delivered decline or timeout into local state explicitly. We added ten regression tests with the fix; one of them says the route “answers from the columns when the live plane has forgotten the ring”.
Two planes
TEEI Language and TEEI Mentorship are one React Router codebase deployed as two Workers, and each Worker has its own LivePresence Durable Object namespace. We address one object per environment through a fixed name.
The durable plane is Turso, our libSQL database. We keep there the windows in which volunteers are available, the bookings and, in the booking’s columns, the outcome of every ring. The real-time plane is the Durable Object, and we keep two kinds of state in it. The open WebSockets, their attachments and the presence we derive from them are connection state: a deploy drops them, and we rebuild them from reconnecting browsers. In the object’s own durable storage we keep two small values, the open rings with their deadlines and a set of rescue watches, and we set one alarm.
We send change signals through the object and keep content out of them. live_changed tells a learner’s page that the list of available volunteers may have changed; it carries no payload and no roster, and the page reloads only what it is authorised to read. When someone receives a message, we send the recipient’s sockets the conversation id, the kind of activity and the sender’s role; for a new notification, at most its type code. We never send message or notification content over the socket: we fetch it through the page’s own loader, on the same authorised route we would use without a socket. A ring carries more: the booking it belongs to, who is calling and when the ring expires.
As of 15 October 2026, this is the pattern on the learner’s home page:
Verbatim excerpt.
useLiveSocket(Boolean(liveRailEligible), (message) => {
if (message.type === 'live_changed') refreshLiveRail('socket');
});
useEffect(() => {
if (!liveRailEligible) return;
const timer = setInterval(() => {
if (!liveSocketConnected()) refreshLiveRail('poll');
}, 60_000);
return () => clearInterval(timer);
}, [liveRailEligible, refreshLiveRail]);
Whether we refresh after a socket signal or through the fallback poll, we read the list of available volunteers from the database through the loader, which is why we run the Workers near the database, not the user. A signal lost while the socket stays connected can leave the list stale, and a learner may then tap a volunteer who has just gone. That costs her a “no longer available” message, never a double booking, because we check the claim against the offer in the same transaction that creates the booking.
Presence follows the same rule. We announce a volunteer as online when we accept their first socket and offline when we close their last; another tab changes nothing. We have never broadcast learners’ own presence, and the counts we send for a group session’s door or a Practice Now queue are numbers without names, the same principle behind a public live activity feed without participant identities.
Write first, then signal
A learner’s claim starts in the database. When a learner claims an available volunteer, we send Turso one batch that runs as a single transaction: we create the booking only if the volunteer’s offer is still open at the generation the learner saw, close the offer and queue the volunteer’s notices. A conditional insert can succeed while inserting nothing, and the consume update can rightly change no row when a trigger has already consumed the offer, so we end the batch with statements that raise an error unless the booking, the consumed offer and the queued notices agree, and that error aborts the whole batch. When two learners tap the same volunteer, one generation of the offer wins and we tell the other learner it is gone.
A remote transaction can commit while its HTTP response is lost. We do not retry the write then. We look up the booking that won that generation of the offer and return it, and on a retried request we resend only the claimed signal to the object. The notices were committed with the booking and are never queued twice. While the ring remains in the object’s storage, we replay a repeated claimed signal only while it is pending and unexpired, without changing its stored deadline or terminal outcome.
The booking commits before we attempt a bounded notification to the object. A failed notification does not roll back the booking: as checked on 15 October 2026, the request waits up to approximately two seconds for that attempt, and if it fails, we log the failure and keep the booking.
One object, hibernating, and how we rebuild it
We keep idle connections open without keeping the object active through three parts of the WebSocket Hibernation API. We accept each socket with this.ctx.acceptWebSocket(socket, tags), so that finding every socket of a role, or one volunteer’s sockets, is a tag lookup with getWebSockets(tag), never a walk through memory. We write a small attachment per socket with serializeAttachment and read it back with deserializeAttachment. And we register a ping and pong pair with setWebSocketAutoResponse, so we let the runtime answer the browser’s heartbeat without waking the object.
When an event wakes a hibernated object, we start again in the constructor with empty memory, and nothing needs to be in it: we take counts from getWebSockets and the attachments, read the open rings from storage, and derive presence from which sockets are open.
Over 30 UTC days, 15 September to 14 October 2026 inclusive (from 00:00 UTC on 15 September up to, but not including, 00:00 UTC on 15 October), the TEEI Language object was active for 38.1 of 720 hours, 5.29 % of the time, while holding up to 152 WebSockets at once. We derived the unit of activeTime from billed duration. And we have not run the same traffic without hibernation, so we cannot say how much of the 5.29 % is hibernation and how much is an object with little to do.
Tags are fixed when we accept a socket, but some of what we need to know changes while the socket lives. Which group session’s door a learner is standing at is a property of the open page, not of the account. We receive that from the browser and keep it in the attachment, which we can rewrite. We rebuild the attachment field by field rather than spreading the old one, because when someone leaves a door without entering another, omitting the old field is the update. For the room’s waiting message, we count distinct other accounts: until 1 October 2026 a learner with two tabs open saw her own second tab announced as someone waiting for her. And we keep watching apart from standing, so a page that only wants a count hears it without being counted.
Hibernation and eviction keep healthy sockets and their attachments. A deploy does not: “Code updates disconnect all WebSockets.” We rebuild socket state from each browser’s page. When the browser’s socket opens again, we claim its door again from the page, then its Practice Now queue or watch, then the doors it watches, because “a reconnect starts from a socket the DO has never heard of”. We regain presence as volunteers’ sockets reconnect. We rebuild socket-derived state from reconnecting clients and reload displayed data from Turso. Pending rings are different: we retain them in durable storage because the alarm needs them.
The one ring the browser cannot re-announce is one it has not heard about yet, such as a volunteer opening a fresh tab from a push notification. Before 11 August 2026 we read the open rings from storage before answering the volunteer’s upgrade request, so a storage read that timed out would have turned a valid handshake into HTTP 500. We have no record of that causing a production incident. We now answer 101 first, deliver the pending ring afterwards inside ctx.waitUntil, and, if that read fails, close the socket with code 1013, “try again later”, so we reconnect from the page and poll instead of trusting a socket we know is missing a ring.
The alarm and the input gate
In the code checked on 15 October 2026, we give a ring four states in the object, pending, answered, declined and timeout, and a deadline of two and a half minutes, the code’s “would a human still be waiting” threshold. We commit the learner’s claim in Turso and then send the object claimed. We write the ring into storage, point the alarm just past its expiry and ring the volunteer. If the volunteer answers, we record the answer in the booking before they join the video meeting, and only then signal the object. If they decline, we first attempt the guarded cancellation in the database, then send the UI signal. If the alarm fires first, we ask the database to settle the ring as unanswered.
We never let the alarm write “timeout” on its own authority. We use the same settlement function for a decline and an alarm timeout: a guarded UPDATE that changes the booking only if nothing else has settled it, and, if nothing changed or the response was lost, a read of the booking. In the function’s own words, “The read below is the authority; never issue a compensating write.” If the answer reached Turso a moment before the alarm, our update changes nothing, we read “answered”, and we tell both sides the call was answered. When we settle a timeout we also queue the volunteer’s missed-call notice, an in-app notification mirrored to Web Push, in the same transaction. We attempt delivery from the alarm and retry undelivered notices in a scheduled job. Email follows the same shape, a row committed with the change and delivered afterwards, in reliable email with Cloudflare Queues and a transactional outbox.
In the object, terminal outcomes are first-writer-wins: whichever terminal signal reaches it first. We settle the race between an answer and a cancellation with guarded database writes, so the object’s local outcome is not guaranteed to match the database’s. That is one more reason we read outcomes from the booking.
Committing decisions, not snapshots
We could overwrite newer ring state while waiting for the database. With a Durable Object’s input gate, we process no new events while awaiting the object’s storage; while awaiting a database fetch, we can process other events. Until 12 September 2026, the alarm read the map of rings, awaited the database and the missed-call notice for each expired ring, then wrote the whole map back. A ring claimed during one of those awaits could be overwritten by the alarm’s older map, losing its timeout with it, and in the same window a recorded answer could be overwritten by a stale timeout decision. We did not find this in production. It came out of a review of our test model on 11 September 2026, was confirmed in the code and fixed the next day; we have not measured whether it affected a real booking.
We now treat (claimedAt, expiresAt, outcome) as the ring’s version. We re-read storage before committing and apply a decision only where those three values still match; a changed display name is not a new version:
Adapted for publication. Comments shortened and the method dedented for display; the logic is unchanged.
// Is `fresh` still the same GENERATION of this ring as the copy a decision was taken from?
export function ringGenerationMatches(
fresh: RingState | undefined | null,
evaluated: RingState,
): boolean {
if (!fresh) return false;
return fresh.claimedAt === evaluated.claimedAt
&& fresh.expiresAt === evaluated.expiresAt
&& fresh.outcome === evaluated.outcome;
}
// Only storage reads and writes are awaited here, which the input gate holds exclusively.
private async commitRingDecisions(
decisions: Map<string, { evaluated: RingState; next: RingState | null }>,
): Promise<void> {
if (decisions.size === 0) return;
const fresh = await this.readRings();
let changed = false;
for (const [bookingId, { evaluated, next }] of decisions) {
if (!ringGenerationMatches(fresh[bookingId], evaluated)) continue;
if (next === null) delete fresh[bookingId];
else fresh[bookingId] = next;
changed = true;
}
if (changed) await this.writeRings(fresh);
}
Re-reading protects the unrelated entries in the map; comparing the evaluated generation protects the entry we were deciding about. In the 12 September 2026 controls we tested them separately. Restoring the stale whole-map write lost both an unrelated claim and an answer to the ring being settled. Keeping the fresh read but removing the generation comparison preserved the unrelated claim and still overwrote the answer. The guarded SQL write protects the booking, and the generation comparison protects the object’s stored copy.
We did not wrap the alarm in blockConcurrencyWhile. On one object that holds every socket in the environment, that would have held up every socket message for as long as the database took to answer. The guarding test is named for the behaviour, “does not overwrite a ring answered while the alarm was awaiting the database”, and it only fails against the old code because our revised fake returns detached copies, so the test can expose stale writes. Those tests do not establish object-identity behaviour in the production runtime. Our earlier fake returned stored objects by reference, so we mistook a memory mutation for a storage write and passed the test with the bug still present.
One alarm, two jobs
A Durable Object has one alarm, and we share ours between ring timeouts and rescue watches, with which we wake the object at the moments a scheduled session might need help, rather than waiting for the next scheduled run. We move the alarm earlier, never later, unless the stored one is already in the past. If we cannot settle an expired ring because the database is unavailable, we retry after a fixed interval. Retrying immediately would turn the outage into an alarm loop.
The shared alarm also shared a failure. We had measured a rescue watch’s age from a boundary we moved forward on every check, so watches never retired and their history kept growing. On 6 August 2026, the object’s production writes failed with SQLITE_TOOBIG, the alarm’s among them. We corrected the growth defect: we now measure retirement from a fixed anchor and cap each retained history on read. That does not by itself bound the combined map’s size. The recorded errors had already stopped before that fix was deployed, so we do not attribute the incident’s recovery to it. The regression test drains until the watch disappears; one drain would pass with the old rule. We also skip malformed entries, because throwing here would take ring timeouts down too.
During database maintenance we settle nothing, re-arm the alarm a minute later and return normally, as checked on 15 October 2026. Cloudflare’s alarm() documentation, checked that day, guarantees at-least-once execution with up to six retries; an alarm that keeps throwing runs out of them and loses its timeout.
What the object keeps durable
With outcomes in the booking, we can lose socket-derived state without changing a settled booking, and lost rescue watches only delay the rescue work until the scheduled job runs. Pending rings are the state we keep durable: we persist them with their deadlines, preserve the alarm’s work and compare versions before applying its decisions.
If your timeouts carry money or a promise to a person, persist their deadlines and test that they settle with no socket connected. A scheduled job that settles any pending claim older than its deadline keeps the object the fast path rather than the only path.
Reading the object’s analytics
Cloudflare’s metrics and analytics for Durable Objects need care with WebSockets. A WebSocket connection appears in durableObjects once, when it closes, and two disconnect statuses, clientDisconnected and responseStreamDisconnected, dominate what is marked as an error. In our sampled analytics for the same 30 UTC days, 61,243 of the 62,245 TEEI Language invocations marked as errors were http invocations with one of those two statuses. The average wall time of a clientDisconnected call was about 23 minutes, consistent with socket lifetimes rather than short requests. An alert on this object’s raw error rate would mostly be measuring sockets that closed.
Alarm invocations are not a count of rings either: rings, rescue boundaries, retries and deferred runs all set the alarm. We have not measured how many rings were answered, declined or timed out, nor the time from a claim to the volunteer’s screen ringing.
Where this design stops
Authority. If you build a collaborative document, a game or a chat whose message history lives in the object, make the object the authority, use its storage and back it up. Our design works because every outcome that matters already had a home in a relational database the rest of the platform reads.
Size. The documented ceiling is 32,768 WebSockets per object, well above the 152 we saw at most in the 30 UTC days we measured. We must also account for CPU and memory, which may limit the practical number first. Before choosing one object for a larger audience, measure CPU, memory and broadcast costs at your expected peak load. Consider objects scoped to rooms or programmes; splitting a global object later changes every place that counts.
Recovery. We rebuild state by resending the browser’s page state after every reconnect. For clients without that behaviour, we need to persist more state and design recovery for it.
Verify this
- Hibernation in your own account. Deploy Cloudflare’s WebSocket hibernation server example, add
setWebSocketAutoResponsein the constructor, keep a few tabs connected and idle for an hour, then comparesum { activeTime }withmax { activeWebsocketConnections }indurableObjectsPeriodicGroups. Store a value withserializeAttachment, send a message later, confirm the constructor ran again and read the value back. - The input-gate window. In your own object, have
alarm()read a stored map and pause at a controlledfetch. Add a key through a request and wait for that request to finish before releasing the fetch. A stale whole-map write loses the key; a fresh read preserves it. Repeat by changing the evaluated entry: preserving it also requires comparing its generation. Make test storage return detached values, for example withstructuredClone.
On 15 October 2026 we ran the ten test files that cover the object, the protocol, the notifier, the ring-status route, the presence route and the Worker’s socket handling: 131 tests, all passing. They run in Vitest with fakes, not against workerd, which is why the fake’s storage semantics matter so much. The analytics, the tests and the 8 September measurement come from our own account, repository and production logs, and we keep the queries and raw output with the source of this note.
Frequently asked questions
Should a Durable Object be the source of truth for real-time state?
Not in our design. We keep every booking outcome in the database and use the Durable Object to hold WebSockets, send “something changed” signals and start ring timeouts through its alarm, keeping pending rings and their deadlines in its own durable storage. We read settled ring outcomes from the booking first.
How do you keep a Durable Object alarm from overwriting state written while it waited?
When we await a database request in the alarm, we can process other events before that request returns. We record each decision with the version of the ring it evaluated, re-read storage when we commit, and apply the decision only where storage still holds that version.
Why does Cloudflare report so many Durable Object errors for WebSockets?
A WebSocket connection appears in the invocations dataset once, when it closes. In our sampled analytics for the 30 UTC days from 15 September to 14 October 2026 inclusive, 61,243 of the 62,245 TEEI Language invocations marked as errors were http invocations with one of the two disconnect statuses.