How Practice Companion Uses ElevenLabs

Practice Companion is the structured practice in TEEI Language between conversations with people. A learner keeps meeting their volunteer one to one, and other learners in group sessions. On the days in between, the companion gives them four rooms around one desk: Write, Listen, Say it and the level conversation. It remembers their level, the situation they are practising for and the lines they have chosen to keep, and every room has a way back to a person. The companion adds structure and continuity between the sessions with people; those sessions stay the centre of the programme.

This note is the speech layer under two of those rooms: how a spoken turn in Say it travels from the learner’s microphone to the reply they hear, and how every dialogue in Listen was voiced once, before publication. Both rooms use ElevenLabs, in two different ways. Say it is a live conversation on the ElevenLabs Speech Engine: ElevenLabs recognises the learner’s speech, decides when their turn has ended and speaks the reply, while our own server decides what the reply is. Listen is text to speech: we voiced each dialogue once through the ElevenLabs API, encoded it once, and the room plays the file. No model runs in the Listen room at all.

Three results from building it:

• 0.77 to 0.89 s from the end of the learner’s speech to the answer on a rule turn in Say it, over nine turns in our integration spike on 11 September 2026.
• 24 Listen dialogues, voiced with ElevenLabs on 10 September 2026, encoded once and served from Cloudflare R2.
• 325 spoken lines in those dialogues, 25.3 minutes of audio, counted from the committed catalogue on 15 October 2026.

Practice Companion is not open to learners yet. Before the rooms open, they have to pass L2SAFE, the benchmark TEEI Research built to test whether a model understands and answers a question as safely in learner language as in fluent language. We are also tuning prompts and model settings so that learners get the best possible experience, finishing the production setup of the voice path, and deploying the level judgement behind its certification gate.

Our note on Practice Companion’s rooms describes all four rooms, how they are built on AWS and how the companion reads a learner’s level. We wrote this one for a developer who wants to see the setup, and for the people at ElevenLabs who built the products it runs on. The code it quotes is ours, from the TEEI Language platform and from our AWS infrastructure code, as it stood on 15 October 2026. It describes what is built; the paragraph above explains what remains before the rooms open.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

At a glance

The rest of this note is about the two rooms that use ElevenLabs.

Say it: the session comes before the token

A conversation in Say it starts in our Worker, not at ElevenLabs. When the learner opens a situation and chooses to speak, the browser asks our token route for a session. The route checks the learner’s access, writes the session row to our database, and only then mints the conversation token from ElevenLabs. The header comment on that route says why the order matters:

/**
 * ...
 * The session row exists BEFORE the token: the runtime binds the provider conversation to
 * the account through this row, never through anything the browser says. If live voice is
 * switched off (no engine id, no key) the same call opens a written session, so the room
 * works without a microphone and the switch is one env var away.
 */

The mint is one HTTPS call from the Worker to ElevenLabs, carrying the key, asking for a WebRTC conversation token for our Speech Engine resource; it is the same call the official SDK makes. The browser receives the token and the conversation id, never the key. If the mint fails, the same call opens a written session instead, so the room still works, in writing. On a reconnect the route mints a new token for the same session and rebinds the new provider conversation to it, with the state version it expects, so the runtime can tell a stale binding from a live one.

This order is the whole access model. When the Speech Engine later opens a socket to our runtime for that conversation, the runtime already knows which account it belongs to, because the row was written before the browser could connect. A conversation id that nobody minted is closed at init, and so is one whose session has ended.

Say it: in the browser

The browser’s job is small. It loads the ElevenLabs client, opens the conversation with the token, and reports status, mode, messages and errors to the room. The import lives in a module of its own with a .client suffix, which React Router keeps out of the server bundle: on the server every export is undefined, in the browser it is the real dynamic import. That split came from a measurement. With the import inline, the WebRTC SDK sat inside our Worker bundle where no request could ever run it, at a cost we measured on 11 September 2026 as 168 KB of gzipped server code, enough to fail the bundle step for every Language and Mentorship release. Now the browser fetches the SDK only when the learner chooses to speak.

export async function startVoice(input: { token: string; opening: string }, cb: VoiceCallbacks): Promise<VoiceSession> {
  // The SDK import lives in a client-only module so the server bundle never carries it [...].
  const { Conversation } = await loadVoiceSdk();
  let conversation: Awaited<ReturnType<typeof Conversation.startSession>>;
  try {
    conversation = await Conversation.startSession({
      conversationToken: input.token,
      overrides: { agent: { firstMessage: input.opening } },
      workletPaths: { rawAudioProcessor: rawAudioProcessorUrl },
      onStatusChange: (s) => cb.onStatus(s.status),
      onModeChange: (m) => cb.onMode(m.mode),
      onMessage: (m) => cb.onMessage({ role: m.role === 'user' ? 'user' : 'agent', text: m.message }),
      onDisconnect: (d) => cb.onDisconnect((d as { reason?: string })?.reason ?? 'unknown'),
      onError: (e) => cb.onError(String(e)),
    });
  } catch (error) {
    const name = (error as { name?: string })?.name;
    if (name === 'NotAllowedError' || name === 'PermissionDeniedError' || name === 'SecurityError') throw new MicrophoneDenied();
    throw error;
  }
  return {
    control: (token) => {
      if (CONTROL_TOKENS.includes(token)) conversation.sendUserMessage(`[${token}]`);
    },
    correct: (index, meant) => conversation.sendUserMessage(correctionText(index, meant)),
    levels: () => ({ output: clamp(conversation.getOutputVolume()), input: clamp(conversation.getInputVolume()) }),
    setPaused: (paused) => {
      conversation.setMicMuted(paused);
      conversation.setVolume({ volume: paused ? 0 : 1 });
    },
    end: () => conversation.endSession(),
  };
}

Three details in that excerpt carry a decision each. The opening line of the situation, the receptionist’s first words, is passed as the first message, so the learner hears it at once and the runtime holds the same text as the line it has already said. The worklet path is ours: the SDK measures what the learner hears in an AudioWorklet and, left to itself, loads that worklet from a blob URL and then a data URL, both of which the page’s script policy refuses, so we serve the worklet from our own origin like any other built asset. And the two level meters the room shows, the learner’s own voice and the receptionist’s, are the SDK’s own input and output volume, read when the room asks for them.

The worklet took two fixes, both found on a staging build and invisible on the development server. The first served it from our origin, on 2 October 2026. The second, the next day, had to make the build emit it as a file: with a plain ?url import the build inlined the small worklet as a data URL again and the level bars stayed flat. A test now builds the client and checks that the worklet comes out as a file.

Say it: what ElevenLabs does

The Speech Engine is ElevenLabs’ product for exactly this split. In its documentation’s words it “adds voice capabilities to any chat agent”, where ElevenLabs “handles speech-to-text and text-to-speech” and “your server provides the LLM logic”. Each connection to our server is one conversation. The Speech Engine recognises the learner’s speech, decides when their turn has ended, delivers the transcript to our runtime, and speaks what the runtime sends back, cutting itself off when the learner starts talking again.

The Speech Engine resource itself, the thing the token names, is created once per environment by an operator running a script, never by the runtime. Its settings are the ones we measured in the September spike plus the privacy posture the room requires: the first message may be overridden per conversation, the language is English, and the privacy block is explicit.

overrides: { firstMessage: true },
// ...
language: 'en',
// ...
privacy: { recordVoice: false, deleteAudio: true, deleteTranscriptAndPii: true, retentionDays: 1 },

No voice recording, audio deleted, transcripts and personal data deleted, one day of retention. The API accepted those settings on 11 September 2026. The resource also carries a voice, a turn-taking setting and the address of our runtime; those belong to the operator and are not in this note.

Say it: the runtime

What the Speech Engine connects to is a small Node.js 22 service we call the practice-voice runtime. It runs on AWS Fargate in Europe (Ireland) behind an Application Load Balancer, and it is a separate service for one reason: we run the SDK’s own integration as ElevenLabs ships it. engine.attach listens for the WebSocket upgrade on a Node.js HTTP server, and the SDK sends a reply only in answer to a transcript. Cloudflare Workers and Durable Objects do hold long-lived WebSocket connections, so the limit is not Workers. Running the integration anywhere other than a Node.js HTTP server, a Durable Object included, would have meant writing it ourselves on the SDK’s lower-level verifyRequest and createSession. A Lambda cannot hold the conversation open. The runtime imports the official @elevenlabs/elevenlabs-js SDK, fetches its Speech Engine resource with speechEngine.get, and attaches it to its HTTP server with engine.attach. Connection authentication is the SDK’s own: every incoming connection carries a header signed by ElevenLabs, which the SDK verifies with the API key, and the option that would switch that check off is never set.

On init the runtime looks the conversation id up in our database and refuses, in this order: when the service is at its capacity, when the id was never minted or the session has already ended, when the session is older than its ceiling, and when the session is the learner’s own situation and no guardrail is configured. Each refusal is one log line with a reason, and closes that session. Every step after the lookup that can fail runs inside a try, so one handler’s failure closes its own session and nothing else. Before production, a review found that without it, one failure would have reached other sessions: on Node 22 an unhandled rejection stops the process, and every conversation on it. A last handler now logs any rejection that still escapes, and the process stays up.

Say it: one turn

Every learner turn reaches the runtime as the whole transcript so far, and the runtime identifies the newest line by its position and its words. That identity was the first thing the spike taught us. ElevenLabs re-delivers the conversation history when it revises its segmentation: on 11 September 2026 “No, I working morning.” arrived, and 1.1 s later “No, I working morning. Can after three?” arrived at the same position. So a re-delivery of an answered turn gets silence, and a longer text at an answered position is a revision: the runtime restores the checkpoint taken before that position and applies the turn again. Positions are counted once for the whole session, across reconnects, because the provider’s count restarts on every bind.

The line is masked as it enters the engine, before the rules read it or anything stores it, so that any email address or phone, card or bank account number it finds is a placeholder from there on, and the rest of the line stays readable for the exercise. Then the rules read it. A fixed situation in Say it is a typed contract: the facts the server holds (the receptionist has two slots), a table of lines, and a step function. The contract classifies most turns by rule, without a call to anyone, and the same rule decides, before anything is spoken, whether a model will be needed at all.

When no rule can read the line, the runtime says a bridge line at once (“Let me see.”) and asks Claude Haiku 4.5 on Amazon Bedrock, in Europe, for the intent and nothing else: a Converse call with a short reply budget and temperature 0, parsed as an intent and at most two times in HH:MM form, and dropped if the contract does not declare that intent. In a fixed situation the model never writes a line the learner hears. The resolved line, from the contract’s table, follows the bridge in the same response:

    // The bridge line is spoken at once; the resolved line follows as the same response.
    async function* bridgedResponse() {
      yield `${contract!.bridge} `;
      const outcome = await learnerTurn(contract!, draft, { index, text, revise }, understand, signal);
      if (signal.aborted || outcome.kind === 'duplicate') return;
      const ok = await commit(outcome.text, [...outcome.events, { kind: 'sent', index, text: outcome.text, bridged: true }], outcome.ended);
      if (!ok) return;
      yield outcome.text;
      if (outcome.ended) scheduleClose(session, conversationId);
    }
    try {
      await speak(bridgedResponse(), contract.bridge);
    } catch (error) {
      if ((error as { name?: string })?.name !== 'AbortError') throw error;
    }

Commit comes before speech. The line is written to the session, with its events, and only then yielded to the Speech Engine; a turn whose signal was aborted before commit commits nothing, and the caller restores the checkpoint.

Times are the one place where the transcript’s spelling and the rules’ spelling had to be reconciled. Speech-to-text writes a time in more forms than a person types it: “half-past ten”, “10.30” and “10:30am” are the same time as “half past ten”, “10:30” and “10:30 am”. A small, idempotent function brings every time into the form the rules know before the rules see it, and the same function runs before the bridge decision, so the two never disagree about whether a model is needed.

/** "10:30am", "4pm", "1:30p.m.": the space before the half of the day she did not say out loud. */
const GLUED_HALF_OF_DAY = /(\d)(?=(?:am|pm|a\.m\.|p\.m\.)(?![\p{L}\p{N}]))/giu;

/**
 * Her line with every time in the plain form the rules know. Idempotent, and the identity on a
 * line with no time written in another form.
 */
export function plainTimes(text: string): string {
  return text.replace(HYPHEN_BETWEEN_CLOCK_WORDS, '$1 ').replace(DOTTED_TIME, '$1:$2').replace(GLUED_HALF_OF_DAY, '$1 ');
}

The learner’s own situation is the exception, and it is where the model speaks. When the learner practises their own words (“Tell my landlord the heating is broken”), there is no line table to pick from, so the model writes the other person’s line itself, behind the same Bedrock guardrail the writing room uses, and the runtime refuses to open such a session at all if no guardrail is configured. The reply is a whole line by default, committed to the session before it is spoken. A sentence-by-sentence streaming adapter exists behind a switch that is off by default, and the help and simpler lines always come whole.

Say it: one channel

The runtime speaks only in reply to a transcript, which is how the SDK is built: a reply sent before the learner’s first transcript is dropped. So the four controls the room offers (repeat, simpler, help, again) and “What I heard” put right travel as text turns over the same ElevenLabs connection, in square brackets, and the engine reads them before anything else.

export const CONTROL_TOKENS: readonly ControlToken[] = ['repeat', 'simpler', 'help', 'again'];

/** A control travels as a learner turn on the voice path: "[help]". Never spoken by her. */
export const CONTROL_PATTERN = /^\[(repeat|simpler|help|again)\]$/i;
export function controlFromText(text: string): ControlToken | null {
  const m = CONTROL_PATTERN.exec(text.trim());
  return m ? (m[1].toLowerCase() as ControlToken) : null;
}

/** "What I heard" put right, sent the same way: "[correct:3] Thursday is good." */
export const CORRECTION_PATTERN = /^\[correct:(\d{1,3})\]\s+(.+)$/is;
export function correctionFromText(text: string): { index: number; meant: string } | null {
  const m = CORRECTION_PATTERN.exec(text.trim());
  return m ? { index: Number(m[1]), meant: m[2].trim() } : null;
}
export function correctionText(index: number, meant: string): string {
  return `[correct:${index}] ${meant.trim()}`;
}

A control is a turn with a position like any other, and the bridge decision is false for it, so a control never waits for a model. A correction retracts the turn it names and restores the checkpoint before it; the learner’s corrected words become the line at that position.

Turn-taking is the one part of the chain we could not settle with a setting. The Speech Engine ends a turn at the first full stop: “I have pain in my back.” went out as a turn 1.8 to 2.3 s into a line of 3.2 s, and the patient turn-eagerness setting changed nothing (9 of 9 lines split on 4 October 2026, against 18 of 18 on the normal setting). The room answered the first half, the learner kept talking, the provider cut the answer, and the second half arrived as a turn of its own, where it read as a line nobody could answer. Two mechanisms meet that. The SDK aborts the previous handler’s signal on every new transcript, so only a turn that had not been committed and sent yet was actually cut short, and the runtime logs that. And the runtime folds: the provider’s history shows how much of the cut answer the learner heard, and when that is short, their newest words are the rest of the line before, joined and run at the earlier position as a revision.

The threshold is two words. Over the 23 cuts in the 17 synthetic rounds of 3 and 4 October 2026, the bridge word not counted, the learner had heard 0 to 2 words of the cut answer in the 20 cuts that came mid-line, and 3 or 4 in the 3 that came after a new line had begun. That is a synthetic learner, which is why the two log events, one for a cut and one for a fold, carry the count for real ones.

Say it: measured

The numbers in the figure come from the integration spike of 11 September 2026: headless Chromium with a WebAudio microphone playing synthesised learner turns, a home connection in Norway, a tunnel to a local runtime, and ElevenLabs’ standard environment. From the end of the learner’s speech, a rule turn was answered in 0.77 to 0.89 s over nine turns. A model turn with the bridge line was answered in 0.80 s, with the resolved line about 1.2 s later, measured once; the same kind of turn without the bridge took 1.95 to 2.11 s, measured twice. A control turn, end to end, took 0.42 to 0.65 s over eight. The provider’s own span, logged in the spike as speech end to transcript, was 0.77 to 0.88 s, which is most of a rule turn: the rule itself makes no call to anyone. The intent classifier alone, on Bedrock in Europe with fifteen calls per model, put Claude Haiku 4.5 at a median of 1,268 ms and a p95 of 1,351 ms; Nova 2 Lite was faster but wrong on one of three phrasings, and Sonnet 4.6 slower, so Haiku reads intent in Say it, while the written rooms run on Sonnet. The end-to-end run on the repository’s own runtime the same day gave 0.84 s on two of three turns and 1.23 s on the first. The conclusion we drew from that day: the median target of under 1.5 s held only with rules first and a bridge line on model turns, and every response we observed that day, on every path, came in under 3 s. With a model turn measured once and twice, that is what we saw, not a p95.

Two things the figure does not show. The segment from our reply leaving the runtime to the first audible sound in the browser is never measured alone; the conversational spans in the figure start at speech end, the classifier was timed in a separate run, and we do not subtract one measurement from another to invent a third. And the learner’s own situation, where the model writes the whole line, measured 1.84 s median and 2.06 s at the 90th percentile for the whole line over 234 calls behind the guardrail on 29 September 2026, which a listener hears as a pause. That is why the streaming adapter exists. It stays off by default.

Say it: audio and text

ElevenLabs hears the learner. Their audio goes from the browser to the Speech Engine, which turns it into the transcript, and the transcript reaches our runtime as unmasked text; masking can only start there. What ElevenLabs keeps is set on the resource: no voice recording, audio deleted, and transcripts and personal data deleted after one day. ElevenLabs also offers a Zero Retention Mode for enterprise customers, which deletes most request and response data as soon as a request completes; our integration does not turn it on. Our runtime receives no audio, only text. Every transcript event carries the conversation so far, and before masking the runtime reads the learner’s lines in it as said, for three things: to tell a re-delivery of an answered line from new words, to tell a revised line from the one already answered, and to join the two halves of a line the turn-taking cut. Two different lines can mask to the same text, so only the words as said can tell them apart. For those comparisons the runtime keeps the last line it committed at each position in memory while the conversation’s socket is open, and drops it when the socket closes; it is never logged or written to a database. The line is then masked as it enters the engine, and the masked line is what the engine, the rules and the model read, and the only form of it TEEI stores. Bedrock receives only the masked line.

Each part of Say it has its own location. Our runtime runs on AWS Fargate in Ireland, and our database is in Ireland too: that is where the masked text TEEI stores is kept. The runtime calls Bedrock from Ireland through the EU inference profile for Claude Haiku 4.5, which can process a request in any of the EU regions the profile covers. Our integration reaches ElevenLabs through its standard environment, not one of its data residency environments. ElevenLabs’ data residency documentation says that customer data in the standard environment is stored in the U.S. Storage in the EU, India or Singapore is an Enterprise option with an isolated environment of its own. Even there, processing may take place outside the storage location, for support and content moderation among other things; for EU residency, the documentation says processing can be kept in the EU by adding Zero Retention Mode and using the API, apart from some optional integrations that need processing elsewhere. The documentation names no processing region for the standard environment, so this note names none.

What we do store about a conversation sits in three tables keyed to the learner’s account: the session row with its checkpointed state and an optimistic version, so a lost race is refused and never merged; an append-only log of the events that explain each line, including the transport events (connected, disconnected, sent, aborted, playback confirmed, audio dropped, mode changed, kept, session ended); and the lines the learner chose to keep. Beside them, the spoken turns that move a situation on or ask to hear a line again are also written into the evidence layer the writing room uses, as the same masked text, for the learning history described under Next. A log line in the runtime carries positions, counts and kinds, never the learner’s words. When a Bedrock call fails, the log carries the kind of the error, read from the error’s name and code only and never its text, because an SDK message can carry a resource identifier, an account id or the text the call was made with. Deleting the learner’s account deletes their rows in the three tables and their evidence rows by account id.

Listen: a recording made once

Listen has no model in the room: nothing is computed while the learner listens. Each dialogue is written and reviewed by people first, then voiced once by an operator running a script against the ElevenLabs text to speech API, and the room plays the result. The catalogue is a committed JSON file behind a zod contract: a level from A1 to B2 with a word budget per level, lines spoken by one or two roles (a receptionist and a caller, a landlord and a viewer, an interviewer and a candidate), three questions with three options each and the line that holds the evidence, a short glossary in English and Ukrainian, and an audio block that only the publishing script may write.

The synthesis script asks for one line per request, with the model eleven_v3 and raw 16-bit mono PCM at 44.1 kHz back, which the API answers with on the Pro plan (measured 10 September 2026: HTTP 200, audio/pcm). Raw samples are the point. The start of every line in the finished recording is then a byte count rather than a probe, and the whole dialogue can be encoded once:

const BYTES_PER_SECOND = SAMPLE_RATE_HZ * BYTES_PER_SAMPLE;

/** Whole milliseconds of 16-bit mono PCM at the request sample rate. */
export function millisecondsFromPcmBytes(byteLength) {
  if (!Number.isInteger(byteLength) || byteLength < 0) {
    throw new Error('PCM byte length must be a non-negative integer');
  }
  if (byteLength % BYTES_PER_SAMPLE !== 0) {
    throw new Error(`PCM byte length ${byteLength} is not whole 16-bit samples`);
  }
  return Math.round((byteLength / BYTES_PER_SECOND) * 1000);
}

/** Start of each line in the joined item, from the byte length of every line before it. */
export function offsetsFromPcmByteLengths(byteLengths) {
  let cumulativeBytes = 0;
  return byteLengths.map((byteLength) => {
    const offset = millisecondsFromPcmBytes(cumulativeBytes);
    cumulativeBytes += byteLength;
    millisecondsFromPcmBytes(byteLength);
    return offset;
  });
}

The joined samples go through ffmpeg once (ffmpeg -y -f s16le -ar 44100 -ac 1 -i item.pcm -c:a libmp3lame -b:a 64k item.mp3), which is how the published file gets clean timestamps; joining per-line mp3 files with a stream copy had produced non-monotonic timestamp warnings. The script passes the previous and next line as context to the models that accept it, and never sends those fields to eleven_v3, which answered them with HTTP 400 on 10 September 2026; a model that rejects them at run time gets the same request once more without them. A 429 is retried, honouring the retry-after header when it is a number of seconds. The character cost of every line is read from the header the API returns with the audio, or counted from the text when that header is missing, and summed into the catalogue entry; an error message never carries the key.

Batch 1, counted from the committed catalogue on 15 October 2026, is 24 dialogues, six per level, 325 lines and 25.3 minutes of audio, generated between 09:47 and 10:02 UTC on 10 September 2026 for 14,525 credits. We voice them within the ElevenLabs Impact Program.

Listen: publishing and the room

The publishing script uploads the mp3 to Cloudflare R2 under a content-addressed name, downloads it again, compares the digest, and only then patches the audio block into the catalogue, byte-stable for every untouched item. An item is published when, and only when, it carries that block. One field in the block is named for a tool’s sake: it is file, not key, because an object name beside a hex digest reads as a credential to a secret scanner.

In the room, the player is an ordinary audio element pointing at our own route, with metadata preloaded. The route checks the learner’s access and the programme’s capability, then serves the object from R2 with Range support: a HEAD request answers with the size, a Range request is passed to R2 as it is, an unsatisfiable range gets a 416 with the size, and the body goes out as 206 or 200 with audio/mpeg, an ETag and a private cache. Range is what lets the room replay a line: the evidence line of every question is a time offset from the catalogue, shown as minutes and seconds, and seeking to it is a byte range.

The room draws no picture of the machine. A diode on the speaker line and a level meter driven by the recording’s own audio say that something is playing, and the credit under the player says by whom: “Voice by ElevenLabs”, in English, Norwegian, Polish, Ukrainian, Spanish and Portuguese, and nothing more. There is no persona, no face and no name for the voice.

Running it

The runtime is one Dockerfile in three stages on a Node 22 base image, running as an unprivileged user with a health check, and one esbuild step that bundles the server to a single ES module for Node 22. The infrastructure is one CDK stack in our AWS code, instantiated twice, for staging on 22 September 2026 and for production on 26 September 2026, sharing nothing: each stage has its own Speech Engine resource, its own secrets, log group, image repository and alarms, and a test asserts that the production template never references a staging resource.

The stack’s header explains its two shapes. The service is “a long-lived socket behind an ALB and not a Lambda”, because the Speech Engine connects to it and a Lambda cannot hold that conversation open. And the VPC has public subnets and no NAT gateway, because the service’s only egress is to public endpoints (ECR, CloudWatch Logs, Secrets Manager, Bedrock, ElevenLabs, Turso); what closes the door is the security group, which admits the load balancer and nothing else. The task role may invoke one model, the EU inference profile for Claude Haiku 4.5, and apply one guardrail, and nothing wider. Alarms per stage cover healthy tasks below the desired count, target 5xx, unhealthy targets, and the learner’s own situation failing repeatedly, read from a log metric filter; a monthly budget per stage notifies by email. The stack’s tests pin all of it, in their own names: “grants the task role bedrock:InvokeModel on that one model and nothing wider”, “refuses a wrong region, a foreign certificate, a non-wss URL or a floating tag”. The first deployment of a stage stands up empty, so nothing pulls an image that does not exist yet; we learnt that on the day staging was created.

What we traded, and when this pattern is wrong

• The runtime is a service of its own because we run the SDK’s integration as shipped, and it attaches to a Node.js HTTP server. Workers and Durable Objects hold long-lived WebSockets too, so moving it there is a choice, not a wall; it means writing the integration yourself on verifyRequest and createSession. If your web tier already runs on a Node.js HTTP server, you do not need a separate service.
• Controls as text turns cost one position each, and keep a single channel. A side channel to the runtime would have given the room a second way to speak, and the runtime has none.
• Whole lines by default. Streaming would recover part of the 1.84 s median measured on 29 September 2026 in the learner’s own situation, but it means a line is spoken before it is committed. We kept commit before speech.
• Rules first, model second, is what met the median target in the 11 September 2026 spike. If your lines must come from a model, the bridge line is the one tool the measurements left us, and you should measure your own slowest turns before promising a median.
• Turn-taking is the provider’s. We did not solve it; we folded around it, with a threshold tuned on a synthetic learner until the logs from real ones say otherwise.
• Test the built artefact, not the development server. Both worklet fixes were invisible on the development server and showed only on a staging build.
• When this pattern is wrong: if the model should write every line, the contract engine is overhead. If you cannot run a long-lived socket server, you cannot use the Speech Engine this way. If you must keep audio, the privacy posture here is not yours.

Next: what we will build on ElevenLabs

Everything above is built today. This section is what we will build next, and none of it is in the product yet. Three ElevenLabs APIs we do not use today come in: Text to Dialogue with timestamps and Forced Alignment for Listen, and Scribe v2 Realtime for separate speaking exercises. The Speech Engine integration stays as described above, and the work on it is turn-taking.

From Listen to Say it

A learner who has just heard a dialogue in Listen will open the same scenario in Say it, take one of its roles and answer aloud. What carries over is the situation and the role, the learner’s level, the dialogue’s vocabulary and what the dialogue was written to teach. ElevenLabs will do the work it does in Say it today, the learner’s speech in and the other person’s voice out; which line comes next stays with TEEI’s scenario contract and its rules. Today the two rooms do not share scenarios: no Listen dialogue has a Say it situation of its own yet, so each one that leads into a conversation needs one, written and reviewed like the dialogue.

Listen: one request per dialogue, with timestamps

Today every Listen line is a request of its own, and the start of every line is a byte count in the joined PCM, as shown above. That method stays the record of how the first 24 dialogues were made. New dialogues will be voiced with Text to Dialogue with timestamps, POST /v1/text-to-dialogue/with-timestamps, which voices a whole dialogue with several voices in one request. The pipeline will run in eight steps:

1. The dialogue is written and reviewed in TEEI, as now.
2. Its lines and their voices go to Text to Dialogue together, as one ordered list of inputs.
3. ElevenLabs generates one continuous dialogue in those voices.
4. The response carries the audio, a list of voice_segments and, when present, an alignment and a normalized_alignment.
5. TEEI validates the audio and the timing before anything is kept.
6. The timing is written to the dialogue’s catalogue entry.
7. The file is published to Cloudflare R2: uploaded, read back and checked, as now.
8. Listen uses the timestamps to move between lines and to keep the text in step with the audio.

Each voice segment names the input line it belongs to in dialogue_input_index, with its voice_id, start_time_seconds and end_time_seconds, so the first segment of each line gives that line’s start, which is what the byte counts give today. Validation checks that every line has a segment, that segments follow the line order and that they fall inside the file. The two alignments are per character and may be null, so word boundaries are derived from the character times and checked, and a dialogue without a usable alignment gets one from Forced Alignment (below). The API reference advises keeping a request to about 2,000 characters of text, so a longer dialogue is voiced in parts, and each part’s timestamps are shifted by the length of the audio before it. We have not run this pipeline on the catalogue yet, so this note makes no claim about its calls, its cost or how it sounds.

Listen: word timing for the recordings we have

Forced Alignment, POST /v1/forced-alignment, takes an audio file and its text and returns when each word and character is spoken. We will run it on the 24 recordings already published, which have line starts but no word timings; on any new dialogue whose alignment is missing or fails validation; and whenever timing has to be made again without voicing the dialogue again. With word timings in the catalogue, Listen will highlight words as they are spoken, jump to a word and repeat a phrase. Forced Alignment does not tell speakers apart, and ElevenLabs’ documentation warns that diarised text gives unexpected results, so the text goes in without speaker labels, and who says each line stays in the catalogue, apart from the timing. All of it happens before publication: the room still plays a file from R2, and no model runs during playback.

Speaking exercises with Scribe v2 Realtime

Scribe v2 Realtime is ElevenLabs’ streaming speech to text. With timestamps switched on, a committed transcript can carry its words with the start and end of each one (event reference), and TEEI will check those times before it uses them. We will use it in targeted speaking exercises of their own, not inside a Say it conversation: each exercise opens its own audio stream to Scribe, with a single-use token our Worker mints, the same pattern as Say it. From the word times, TEEI will read pace, pauses, hesitation, repeated attempts and how evenly a sentence flows, and combine that with its own rules about what a learner at that level should practise next. Our Say it integration makes no call of its own to the Scribe Realtime API, and nothing in this design connects the two. A transcript that is right is not a pronunciation score, so feedback on pronunciation will come only where a measurement has been validated for it.

Say it: the conversation itself

Turn-taking is still the provider’s, and the fold above works around it rather than solving it. The next work builds on what the runtime already has: the checkpoint before every position, the revision of an answered position, the abort signal on a new transcript, and the fold. It covers a learner who stops mid-sentence and goes on, a learner who speaks while the room is still talking, a transcript revised after the runtime has answered, and partial lines joined into the one the learner meant. Each change is measured against the baselines in this note: the response times of 11 September 2026 and the 23 cuts in the synthetic rounds of 3 and 4 October 2026. The measures are response time and how much of a cut answer the learner heard, which the runtime already logs for every cut and fold. Real learners will give that count a baseline the synthetic learner could not.

One learning history across the rooms

Say it already writes the spoken turns that move a situation on or ask to hear a line again into the same evidence layer as Write, as a channel of their own. Listen’s results will join them, and with the level conversation they make the learner’s one history. From it the companion will choose vocabulary and phrases, the next dialogues to hear, the next situations to speak in and their difficulty, and when to bring back something the learner found hard. Three things stay apart in that history: practice, progress the learner can show, and the formal reading of their level from the level conversation. Listen’s questions stay unscored: the learner sees whether an answer matches the dialogue, and that never becomes a grade. Conversations with volunteers one to one, group sessions and the teacher-led TEEI Language Courses go on beside the companion. For the learner it becomes one path: hear a situation, speak in it, write about it, and meet it again when it is due.

Verify this

The connection check our runtime relies on is the SDK’s, and the SDK’s README documents it. From @elevenlabs/elevenlabs-js 2.67.0: “Both SpeechEngine.Server and speechEngine.attach() automatically verify the X-Elevenlabs-Speech-Engine-Authorization header on every incoming connection, rejecting any requests that were not signed by ElevenLabs.” The attach example in the same README, abbreviated here with the example’s language-model call replaced by a comment, is the shape of our runtime:

// Attach Speech Engine to existing server
await elevenlabs.speechEngine.attach("seng_123", httpServer, "/api/speech-engine/ws", {
    // ...
    onInit(conversationId) {
        console.log("session started:", conversationId);
    },

    async onTranscript(transcript, signal, session) {
        // Pass the transcript to your LLM: signal auto-aborts if the user interrupts.
        // ... the example's language-model call, which produces `response`, is omitted here.
        // In our runtime, the rules read the line first, and a model only when no rule can.
        session.sendResponse(response);
    },

    onClose(session) {
        console.log("session ended:", session.conversationId);
    },
});

The offset arithmetic in the Listen section is pure, and you can check it on paper: 44,100 samples per second at 2 bytes per sample is 88,200 bytes per second, so a line of 176,400 bytes lasts 2,000 ms and the next line starts there.

• Read the ElevenLabs Speech Engine documentation, which also covers how a server authenticates the connections it receives.
• Read the text to speech API reference: the output_format list carries both pcm_44100 and mp3_44100_64, and it states that PCM at 44.1 kHz needs the Pro tier or above.
• Read about ElevenLabs’ models, including eleven_v3, and about the Impact Program.
• Read our note on Practice Companion’s rooms, which this note sits under, and TEEI Language, the programme the companion belongs to.
• Read how Amazon Chime SDK carries TEEI’s conversations, the other audio path on the same platform, where the learner talks with a person.
• If you are building with TEEI, talk to us and we will walk you through the code.