AI Writing-Level Estimate: What TEEI Tested and Learned

TEEI Language gives learners an estimated writing level based on eight written answers. The result includes examples from their own writing and suggestions for practice. Learners remain free to choose a different practice level.

Building that experience required us to examine what happens between a learner writing a sentence and a model assigning it a level. A quote could be copied correctly while the judgement attached to it was wrong. The same conversation could receive different levels on repeated readings. A sentence of help beside an answer could influence the result. Models from different providers could disagree widely and still share the same mistakes.

This is the record of that work, from the first design in August 2026 to the release on 25 October and the delivery fix the following morning. It covers the instrument, eleven design decisions revised during development, the September evaluation, a comparison of 22 models, the move to eight answers, the background assessment job and the October evaluation of the released configuration, L1.

The evidence has a clear boundary. All the evaluation conversations in this record were constructed and labelled by Claude agents. No human-rating round has been completed. L1 also failed one of our internal release criteria and was released under an explicit exception. The figures below describe those tests. They do not establish an accuracy rate among learners.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

What the learner gets

The room is called Check your English. It asks the learner to answer eight short questions and explains that the result will be an estimated writing level with suggestions for what to work on next.

Each question takes a free-text answer. The learner can skip a question or ask for one of three neutral clarifications: a few sentences are enough, an example from their own life is suitable, and there is no right or wrong answer. The conversation does not correct the learner while they are writing.

When an answer contains fewer than five words, the room asks once: “Could you say a bit more, in a sentence or two?” The learner can add a sentence or send the answer as it is. This small piece of wording became one of the experiments described below.

After the eighth answer, the learner presses Read my level. A wait screen appears and explains that the assessment can take a minute or two. The work continues as a background job, so the learner can close the page and return for the result.

The result is headed “Estimated writing level”. It explains its basis: “Based on your written answers in this conversation. This AI estimate is a starting point for practice.” Below the level, What you can do shows observations supported by the learner’s sentences. “Worth working on next, from what you wrote” gives up to three suggestions, also tied to their writing.

An About this estimate note says that the method has not yet been validated against teacher ratings. The saved result is labelled as an AI estimate wherever it is shown. A difference between two such estimates is not presented as established progress.

When there is too little usable writing, the learner receives no band, including no default A1. The message asks for a little more writing in English, and the learner can start again.

The estimate helps set a starting point for practice. It does not lock the learner to that level. A learner with an A2 estimate can choose B1 practice, and the saved estimate remains A2. Their choice is used for group practice and Practice Now. A seat already booked in a group session keeps its booked level. This separates the recorded observation from the learner’s decision about what to practise.

The level conversation opened on 25 October 2026, separately from the other Practice Companion rooms.

What we tested, and what the numbers mean

We investigated several different questions. Can the system complete the assessment? Does its result match the label assigned to a constructed example? Does it give a similar result when the same example is assessed again? Are its quotations actually present in the answer? Can it handle short writing, another language, misleading surface features and instructions embedded in an answer?

Agreement with teachers is a separate question. The intended labels in these tests were written by the agents that constructed the conversations. They are useful development references, but they are not independent human judgements about learners.

Three units recur throughout this note:

Five assessments of the same conversation are five repeats of one case. They help expose instability; they do not create five independent learner examples. Some assessments stop at a writing check before any model call, so the number of assessments multiplied by three is not necessarily the number of model readings.

The exact-agreement metric also has a specific denominator. It compares the returned outcome with the case’s primary reference band on eligible, answered rows that were not refused. Expected insufficient outcomes and accepted guardrail refusals are reported separately. In the model comparison, missing usable outcomes can reduce the denominator. We therefore show the fractions and missing outcomes, rather than treating every percentage as an overall success rate.

Three configurations, not one continuous score

The first and last percentages are not a controlled before-and-after comparison. Both the scorer and the evaluation sets changed. The September result describes the earlier configuration.

Throughout this note, an internal release evaluation means our own technical checks on these constructed cases. It is not an external certification of CEFR accuracy.

Designing the conversation

The first design in August called for a conversation whose conclusions could be traced to the learner’s own sentences. It would provide a band and useful observations, without a points score, a confidence percentage or a certificate. In September we chose to develop that conversation as the instrument around which level estimates and future work on progression would be built.

TEEI also has a separate multiple-choice web assessment. Its implementation uses six routing questions followed by eight questions in the selected track, for fourteen in total. Repeated attempts draw on a fixed question bank, so familiarity can affect the result. The selected track can change, which is why it would be inaccurate to say that every repeated attempt necessarily asks the same fourteen questions.

The writing conversation asks the learner to use language: describe a routine, explain what happened, ask for help or support an opinion. That gives the assessor evidence of written production and interaction. It also makes the assessment sensitive to the task, the wording of the question, the amount of writing and the judgement applied to it.

One role asks; another assesses

The interviewer words the next question. It does not assign a level. The scorer receives the completed conversation and assesses it. It does not choose the next task.

The interviewer uses Claude Sonnet 4.6 at temperature 0.3. A forced tool returns only a question, with an output budget of 200 tokens. The response must pass checks for one question and a maximum of 240 characters. If generation, validation or the call itself fails, the task’s canonical wording is used.

This fallback was exercised during the production walkthrough: one wording call timed out after ten seconds, the planned question was displayed, and the conversation continued. The model can vary the wording without becoming the only way to ask the question.

Eight answers from a versioned task plan

The plan, ltp-4, begins with three core tasks. Code then chooses three support or three stretch tasks according to the mean length of those first answers. At an average of at least 25 words, it selects the stretch branch; otherwise it selects support. Two closing tasks follow.

Length is a simple routing signal in this design. It does not assign a band, and it does not establish that the two branches are equally difficult. All learners are offered eight question turns, but they are not necessarily offered the same tasks or wording.

The final column describes the intended accessibility of the question. It is not the level a learner receives for answering it.

Each task has A and B variants. The first conversation selects a variant randomly and records it. A later conversation selects the other variant relative to the learner’s last scored conversation. Random selection avoids making the first variant predictable from the account identifier alone, which is useful for later blinded review.

The variants were designed to be comparable. Their empirical equivalence has not been established. The same limitation applies to the support and stretch branches and to generated variations in question wording.

Five criteria drawn from the CEFR Companion Volume

The rubric paraphrases five areas from the CEFR Companion Volume: written interaction, task and thematic development, vocabulary and structural range, grammatical accuracy, and coherence and cohesion.

Its intended range is A1 to C1. C2 was excluded from this short instrument. That design decision does not establish that eight answers provide adequate evidence at every other band; the instrument’s coverage still needs empirical examination.

The rubric changed twice during the work.

On 24 September, rubric-w1 became rubric-w2 after an A1 case was rated A2 on every criterion in all five repeats. The earlier descriptors allowed a connector or an isolated correct sentence to carry too much weight. The revised wording made the distinction between isolated forms and sustained control more explicit.

On 24 October, rubric-w2 became rubric-w3. Its B2 written-interaction line had referred to relating an answer to what the learner was asked. That was too easy to satisfy: short B1 answers could stay on topic. The replacement followed the Companion Volume’s overall written-interaction descriptor: “Can express news and views effectively in writing, and relate to those of others.”

Another rule came from an early B1 case with many surface errors. The scorer placed it at A2, and the prompt gained the instruction “Range is not accuracy.” A learner can attempt a broader range of language while making errors in it. The two observations need to remain distinguishable.

A correct quote can accompany an incorrect judgement

For each criterion, the scorer returns a band or not observed, a reason and evidence consisting of a turn number and a quotation. Its structured response also contains the overall outcome, observations about what the learner can do and suggestions for practice. It contains no confidence number.

Each quote is checked against the learner’s answer to the named turn. Only whitespace is normalised. A quote that is absent is removed. A criterion left without valid evidence becomes not observed. Suggestions and positive observations without valid quotes are removed too.

This happens twice: first in the AWS function that calls the model, then in the application against the stored answers. The second check grounds the returned evidence in the record the application actually holds.

The checks also constrain the assessment:

• Fewer than two observed criteria produces an insufficient outcome.
• An overall band outside the lowest-to-highest range of the observed criteria produces an insufficient outcome.
• Skipped turns cannot provide evidence.

These rules establish where the quoted text came from. They do not prove that the quote supports the assigned band. That distinction became visible when repeated calls returned different levels with valid quotations, and later when Ukrainian sentences were used as evidence for English criteria.

We also tested deriving the overall band from the lower median of the criterion bands. On 31 cases assessed five times, that alternative produced seven high readings on designated challenge cases, compared with four when the model selected a band within the observed criterion range. We kept the latter rule. This was a comparison within that test setup, not a general finding that one aggregation rule is always preferable.

When there is not enough writing

The prompt already said that a word or two was not enough for a level estimate. On 24 September, answers such as “Krakow. / I study. Engineering. / Yes. Very.” still received A1 judgements across the criteria.

We added a writing floor in code: at least two answers containing five words or more, and at least fifteen words in total. Below that floor, the system returns insufficient without asking the model. Above it, the scorer can still decide that there is not enough suitable evidence.

These thresholds are a practical minimum for attempting an assessment. They do not establish that fifteen words are sufficient to measure a person’s writing proficiency. Since 20 October the application has checked the same floor before making the AWS request, avoiding an assessment attempt that the backend would reject immediately.

Content guardrails and the evaluation gate

The learner’s answers are sent as guarded content for Amazon Bedrock’s content checks. If any of the three readings is refused by that guardrail, the combined assessment is refused and no level is saved. A staging test containing an instruction to award C2 exercised this refusal path.

The instruction to treat learner answers as material to assess, rather than instructions to follow, also appears in the scorer’s system prompt. One successful injection test demonstrates that particular outcome; it does not establish universal resistance to instructions embedded in writing.

The content guardrail is separate from the internal release-evaluation gate discussed later. The former checks content during calls. The latter checks whether the scorer being deployed matches an approved evaluation record. The October release exception concerned the evaluation gate.

September: eleven decisions revised during development

Before the September scorer passed its internal release evaluation, we revised the dataset, rubric, aggregation, timeouts and version tracking. Some changes replaced an early design; others corrected an implementation assumption.

Repetition exposed variation inside the judgement

The single-reading experiment showed that the same conversation could receive different bands across calls, even when all quotes were valid. At temperature zero, variation remained in five of 31 cases.

A closer look at one case showed that the disagreement began within the criteria. One reading assigned B1 to four of the five criteria and returned B1 overall. Other readings assigned A2 throughout. Changing only the final arithmetic would not address that disagreement.

We used three separate calls at temperature 0.1. The calls share a model, rubric and prompt, so they can share systematic errors. Combining them was a response to measured variation, not a claim that they act as three independent assessors.

Why the lowest reading was retained

We first tried the median of three. On the exploratory run of 45 conversations with five repeats, the stored readings allowed us to compare several rules under the same conditions.

The median failed the internal criterion against high readings on designated challenge cases. A case with A2 as its reference could return B1, B1 and A2; the median would retain B1. An earlier estimate based on resampling had suggested that the median would never inflate those cases. The subsequent run contradicted it, and the claim was withdrawn.

The implemented rule first checks whether enough readings can be used. If as many readings report insufficient as assign a band, or more, it returns insufficient. Otherwise it keeps the lowest band among those that assigned one. At least two usable readings are required by the assessment path. A content-guardrail refusal is handled separately and refuses the whole assessment.

The retained result is one complete reading, including its own criterion judgements and evidence. We do not take a level from one call and construct a supporting explanation from another. Disagreement among the readings is recorded.

This rule favours avoiding high placement at a boundary, at the cost of more low placement. It is a product and evaluation choice with a measurable tradeoff. Repetition also cannot repair a case that all three calls consistently rate above its reference.

The 25 September evaluation

The earlier configuration passed a run covering 60 constructed conversations, assessed five times each, for 300 assessment outcomes. Assessments below the writing floor return without a model call, so the number of calls cannot be obtained by multiplying all 300 outcomes by three.

The development set contained 31 cases. A fresh regression set contained fourteen. A further holdout contained fifteen, with its design specified before the examples were written: five typical cases, four designed to tempt a higher reading, four designed to tempt a lower one and two mechanical checks. It was frozen before its first run, and its hash was recorded.

Exact agreement is the evaluation record’s metric on eligible band rows, not a percentage of all 300 outcomes. Expected insufficient outcomes and accepted refusals follow separate rules. No designated challenge case was read too high in the combined results. The record reports that 99.8% of returned evidence quotes matched the answers after whitespace normalisation.

Fourteen holdout cases had a reference band; the fifteenth was intended to produce insufficient. All fourteen matched their reference band in every repeat, and the remaining case produced the intended insufficient outcome. Those deliberately constructed examples are too small and too different from a sample of learners to establish accuracy in use.

The recorded thresholds and labels were not changed to make the run pass, and no case was dropped. That matters for interpreting the internal result. It does not make the labels independent of the team that constructed them.

A separate analysis of the same readings reported an increase in low readings on the development set from 2.8% with a single reading to 5.2% with the lowest-of-three rule, mainly on an error-heavy B1 case and a six-turn B2 case. That analysis used a denominator that was not preserved and reported a different development exact rate from the approval record. We retain it as part of the research history; the primary result above is the approval record’s figure.

The recorded usage was approximately 1.34 million input tokens and 0.98 million output tokens. A separate timing check reported 34 readings of longer development conversations, with a median of 17.2 seconds and a 90th percentile of 27.1 seconds. These timings describe that check, not a latency guarantee for all learners.

What an approved evaluation record covers

The runner writes an approval record only after a full qualifying run against a published guardrail version, with enough repeats, all required sets and an unchanged scorer fingerprint. Each set must pass its own thresholds. Errors and inappropriate refusals fail the run as well as the seven numerical criteria listed in the appendix.

The deployment check compares the candidate with that record: scorer fingerprint, covered files, model, pinned guardrail version and guardrail policy. The fingerprint covers the scoring code and rubric. The interviewer is outside that fingerprint; it has a separate task and canonical-wording fallback.

This approval records tested behaviour on the constructed examples. It does not certify agreement with teachers. Once a held-out set has been examined and used to choose changes, it also ceases to provide fresh, untouched evidence for later selection. Its original result remains a historical result, and later uses must be described as reuse.

October: comparing models under the same task setup

On 16 October we compared newer models, less expensive models and combinations of models against the September Sonnet 4.6 configuration. The experiment reused the same 60 constructed conversations, with three assessment repeats and three model readings per assessment.

Of 31 model identifiers attempted, 22 produced result sets. One of those, Fable 5.1, was incomplete because API credit ran out. The other attempted identifiers either were unavailable to our account, failed the response schema during preparation or encountered rate limits before a usable full run. These access outcomes say nothing about the models’ assessment ability.

Eighteen of the 22 models were accessed through Amazon Bedrock. Haiku 5.5, Sonnet 5.5, Opus 5.5 and Fable 5.1 were accessed through the Anthropic API. Kimi K3 used a global Bedrock profile that could route outside the EU. Other models used different regional routes, including Stockholm. These were synthetic conversations; the routes should not be read as a description of where the released product sends learner writing.

A shared task still needs model-specific integration

The comparison used the historical prompt and rubric, with documented adaptations for the interfaces. Some models did not support prompt caching or the same strict-schema setting. Some needed different tool-choice handling, larger output allowances or no temperature parameter. For models served outside Ireland, the experiment called the Ireland-based guardrail separately where required.

Each adaptation was checked before the main run. Nevertheless, the experiment compared practical integrations under a shared task specification. It was not a claim that every model had received its own best possible prompt and inference setup.

There was also a known development advantage for the baseline. The prompt and intended labels had been written and adjusted around Sonnet 4.6 by Claude agents. A result on that material cannot be treated as a neutral ranking of model intelligence or CEFR expertise.

The experiment’s denominators

Each complete attempt scheduled 180 assessment rows: 60 cases, repeated three times. Three cases were below the writing floor, leaving 171 assessments with 513 recorded reading attempts, including attempts refused by the guardrail. The reconciliation checked the stored rows and their identities, including failed and refused attempts.

The baseline’s exact-band comparison used 72 development rows, 39 regression rows and 42 holdout rows. Other models sometimes returned no usable outcome on one of these rows, reducing their exact-rate denominator. The full appendix therefore reports missing outcomes as well as exact fractions. A model that cannot return a result has still failed that part of the task, even when the conditional exact rate omits the missing result.

Three models passed all three sets

The costs are historical estimates from recorded token usage and the prices used in the experiment. They include the recorded cache behaviour and exclude separate ApplyGuardrail charges and the rest of the application’s infrastructure. They are not current price quotations or total costs for a successful learner journey.

Kimi K2.5 was a credible cheaper candidate under these conditions: its estimated model usage cost was about 21% of the baseline’s. Its regression and holdout agreement were lower. DeepSeek V3.2 also passed the internal criteria, but 52% of its model-backed assessments exceeded the then-current 28-second request budget.

The tested newer Claude configurations did not outperform the historical baseline across the full evaluation criteria. Opus 5.5, for example, produced two high readings on designated holdout challenge cases and failed that set. Other newer models had their own combinations of disagreement, unusable output and failed criteria. Their complete results are in the appendix.

This supported keeping the existing setup at that time. It did not show that a newer model could never perform better with another prompt, protocol, dataset or deployment constraint.

Fable’s incomplete run

Fable 5.1 had 159 failed reading attempts when API credit was exhausted. There were 118 of the 171 model-backed assessments for which all three calls received responses. We compared it with the baseline on exactly those same assessment keys, and excluded it from model rankings and panels.

The assessment count includes cases handled under non-band rules, which explains why it differs from the exact-band denominator. This matched-subset result is informative about the completed portion, but cannot substitute for the missing part of the run.

What happened when models formed a panel

We combined stored readings into panels of three or five models from different vendors, using six aggregation rules. The broader comparison produced 274 three-vendor and 571 five-vendor combinations. The Bedrock-only analysis used twelve eligible models and produced 172 and 316 combinations respectively. Those are different search pools and should not be mixed when quoting their results.

The best selected three-model panel in the Bedrock analysis combined DeepSeek V3.2, Opus 4.6 and Pixtral. Each model contributed its lowest-of-three result, and the panel took the median across those three results. It achieved 100% exact agreement on each of the three existing sets.

That was a higher score than the baseline on those sets. It was also the best result selected from 172 combinations using the development data. The sets had already been used in the research programme. We did not establish that the gain would persist on new learner material.

The same panel’s estimated model usage cost was $0.2145 per assessment, about 2.9 times the baseline’s. Its median assessment time was 28.3 seconds, with 52% taking more than 28 seconds. At the time, that exceeded the synchronous request budget too often.

The typical panel performed less well. Across the 172 Bedrock three-model panels, the median exact rates were 0.889, 0.897 and 0.929 for development, regression and reused holdout, compared with 0.917, 0.974 and 1.000 for the Sonnet 4.6 baseline.

Different providers could share the same mistakes

All twelve panel-eligible Bedrock models produced the same outcome on 44 of 171 model-backed rows, or 26%. They all agreed on none of the 21 A1 assessment rows.

Disagreement alone did not tell us which model was correct. In the Bedrock analysis, pairs from different vendors shared a median of 47% of their errors, compared with 21% for pairs from the same vendor. The broader analysis of the top ten models reported 42% and 25% respectively.

These descriptive measures need care. The shared-error calculation divided the overlap by the smaller error set, and the same-vendor groups contained few pairs. They do not establish that models from the same vendor are generally more independent. They show that vendor diversity did not prevent correlated errors on these examples.

A panel that defers disagreement is a different product

One configuration used Opus 4.6, Kimi K2.5 and GLM-5 and deferred any disagreement for human review. It deferred 12%, 14% and 18% of rows across the three sets. On the rows it decided automatically, exact agreement was 100%, 97% and 100%, with no designated challenge case rated too high.

Those percentages cover the selected subset the models agreed to decide. The deferred cases were not evaluated by people in this experiment. The result points to a possible workflow involving human judgement; it is not the measured performance of a completed human-and-model service.

Why the later background job changes the latency argument

The 28-second constraint belonged to the architecture used on 16 October. The subsequent background job allowed substantially longer assessment work. Reanalysis of the stored panel timings found that the selected panel’s 99th percentile was 51.9 seconds, with no assessment exceeding a minute in that sample.

That removes the old request ceiling as a sufficient reason to rule out that panel. It does not remove the higher cost, the shared errors or the need to establish an improvement on independent material. We retained the single-model setup; the evidence did not yet justify a production panel change.

The conversation itself changed the assessment

The level conversation opened on staging on 15 October with six questions. Four days later, a walkthrough ended with “Nothing was read”.

The system had not crashed. It returned in 215 milliseconds because the answers were below the writing floor and no model was called. The interface had told the learner to write as much or as little as they liked. It had not made the need for sentences clear, and the same final message covered several different reasons for no level.

We changed both the interaction and the task-plan version.

Conversations started under an earlier plan retain their own rules. The plan version is part of the stored assessment’s comparison key, so an ltp-4 assessment is not treated as the same instrument as an ltp-3 assessment.

The help text changed the result

We examined one synthetic conversation with six very short answers and two longer answers of 25 and 28 words. The transcript contained a help line beside a short answer: “A word or two is too little to read a level from.”

That line was visible to the scorer as part of the conversation context. We tested five versions, four assessments per version and three model readings per assessment, for 60 model calls. Two versions also adjusted the last answers to fit their questions more closely.

The new wording produced a band in all eight assessments across its two variants. The original wording was associated with insufficient readings in both versions that included it. This is a small diagnostic experiment on one conversation, with some answer content changing between variants. It establishes sensitivity in this test; it does not estimate how often help text affects learners or reveal the model’s internal reasoning.

It was enough to change our wording. “Could you say a bit more, in a sentence or two?” asks for more writing without declaring that the current answer is unsuitable for assessment. We changed the help text while leaving the scorer’s prompt unchanged for that experiment.

The broader lesson is that the material surrounding an answer can affect a judgement. A task plan and a scoring prompt are only part of the instrument if the scorer also receives clarification text and other conversation context.

Keeping the writing flow usable on a phone

On 23 October we inspected the writing flow at a 390 by 844 pixel viewport. Submitting an answer revalidated the page loaders, briefly moved the counter backwards and removed the input field for more than a second.

We changed the flow so that non-final answers no longer trigger that full reload. A conflict caused by the server having moved on triggers one refresh. A dropped connection keeps the learner’s words available in the field. The previous submitted answer is cleared when the next question arrives.

These changes did not alter the rubric. They made it more likely that the learner could complete the intended conversation without losing their text or misunderstanding the current step.

Moving the assessment out of the request

On 23 October, pressing Read my level could return the learner to the writing field without a result. The logs showed the model calls reaching their 1,600-token output limit. Producing an overall band, five criterion judgements, reasons and quotations required more output than the request allowed on those conversations.

The failed attempts still took roughly 21 to 29 seconds and counted against the learner’s assessment limit. Several time limits converged around the same point: the application waited 28 seconds, the HTTP API integration allowed at most 30 seconds, the practice function had a 30-second timeout, and post-response work in the Worker had its own limited lifetime.

The API Gateway limit here is specifically the HTTP API integration limit. Other API Gateway products have different limits.

More output room, with the scoring method kept visible

We increased the output allowance to 2,400 tokens. We also tried shortening quotations and taking notes on individual answers during the conversation. Those were different scoring configurations and were evaluated separately. Both failed their evaluation criteria and were set aside.

The final assessment moved to a background job at AWS. The configuration allows up to 120 seconds per model call and an hour for the job as a whole. Those are execution limits, not expected wait times. The page asks for the result every two seconds, and returning to the page retrieves the existing job or stored result.

In one walkthrough, the wait screen appeared 191 milliseconds after the button press and the estimate appeared after 22.2 seconds. Closing and reopening the tab recovered the result without starting another assessment. A separate stress check used eight long answers totalling 1,347 words: 24 individual readings completed without reaching the output limit, with the longest using 1,678 of the 2,400 allowed tokens. Those readings took 17.6 to 22.2 seconds.

These are observations from the tested conversations and flows. They establish the behaviour exercised in those checks, not a service-wide latency distribution.

Seven evaluation configurations that did not pass

Increasing an output limit changes what a scorer can produce, even if the prompt stays the same. The development record tracks seven failed evaluation configurations across 23 and 24 October.

The individual runs used the sets and configurations available at their respective dates. Their aggregate percentages should not be placed on one trend line as though the evaluation population stayed constant.

When a quotation was in the wrong language

The output-limit-only configuration failed on a holdout conversation answered mostly in Ukrainian, with three English words among 63 words. The intended outcome was insufficient for estimating English writing. One of five combined assessments instead returned A1.

The quote check explained an important gap: it verifies that a quote occurs in the answer, not which language it is written in. In a diagnostic sample of 30 readings, the scorer cited sixteen mostly Cyrillic quotations against English criteria, and one raw reading assigned a level. A valid substring was not sufficient evidence of English performance.

We investigated four candidates: a script-share filter on quotations, language identification for each quotation, a target-language prompt rule called P1, and a narrower rule called E.

These probe samples are separate from the earlier diagnostic sample. Their denominators differ and are shown explicitly. They were used to identify tradeoffs, not to estimate population error rates.

The script filter returned insufficient in ten of ten live assessments of the Ukrainian case. It would also introduce a new threshold based on the proportion of letters in a writing system. The language-identification alternative increased the bundled function from approximately 157 kB to 1.8 MB. We did not adopt either code filter and pursued the prompt approach.

P1 and E both addressed the targeted language cases. P1 also increased high readings on the spelling case. The narrower E behaved better there, but each prompt still failed its full evaluation. The targeted repair therefore did not establish that either candidate was ready to replace the scorer.

The A1 and A2 boundary remained difficult in the tests

A new holdout set used the eight-answer format. The boundary-focused prompt K had been designed to distinguish adjacent bands through what the writing accomplished, rather than surface fluency alone. Its exact band agreement on that new set was 76%.

A mostly Norwegian conversation received A2 in all five assessment repeats when it should have received no English level. A constructed A1 case containing phrases intended to tempt a higher rating also received A2 in all five repeats.

The latter weakness was not unique to the new prompt. A separate three-case probe used Kimi K2.5, DeepSeek V3.2 and Sonnet 4.6, with ten assessments of each case per model. None of the three returned A1 for that phrase-heavy A1 case. Kimi handled the basic A1 and Norwegian cases as intended in that probe; the other results varied.

We also asked two fresh AI raters to review two disputed A1 conversations without seeing the scorer’s results. One rated them at A1 or upper A1, while the other rated them at low A2. That disagreement did not resolve the reference labels. It reinforced the need to separate agreement with a synthetic answer key from agreement established by trained human raters.

We retained the test labels and reported the outcomes. The blind AI review was not counted as human calibration.

L1: the complete result for the released configuration

L1 changed the prompt and the B2 written-interaction rubric line. The previous eight-answer holdout became a regression set, and a fourth holdout of sixteen conversations was frozen before its first run. The approval rule now required all five sets to pass.

The full L1 run covered 97 conversations, assessed five times each: 485 assessment outcomes. Twenty assessments stopped at the writing floor, and the record contains 1,395 model-reading attempts. There were no technical errors in the completed run.

The full outcome distribution matters because the exact-rate denominator does not describe the whole run.

The first four rows make up the 425 band results. Accepted alternative bands remain a separate category; they are not silently added to exact agreement or counted as errors.

Seven of the eighteen outcomes classified as overestimates occurred on designated challenge cases and broke the zero-overestimation criterion for that category. They came from two conversations:

• The phrase-heavy A1 case was assessed as A2 in all five repeats.
• An A2 case with many spelling errors was assessed as B1 in two of five repeats.

Other A1 cases also received higher bands. The seven gate-breaking rows are a subset of the eighteen overestimates.

All fifty rows intended to produce insufficient did so. Every other numerical threshold held in every set. The largest observed distance from a primary reference band was one band, and the largest spread across repeats was one band. Ninety-one of the 97 cases had identical combined outcomes across all five repeats. Quote matching was 27,251 of 27,325 returned quotations, approximately 99.7%.

Stable outcomes are useful, but a consistently wrong result remains wrong against its reference. The five identical A2 results on the phrase-heavy A1 case illustrate why repeatability and reference agreement must be reported separately.

The run used 2,440,035 input tokens, 1,678,762 output tokens, 3,964,203 cached input tokens and 17,502 cache-write tokens. At the rates used in that run’s cost calculation, the estimated model usage cost was $33.76. This estimate uses that report’s pricing assumptions, which differ from some of the regional rates used in the 16 October comparison; it is not a directly comparable billing measure across the two experiments.

The run did not produce an approval record for L1. The existing approved record still referred to the September configuration.

The release decision and the production walkthrough

We chose to open the level conversation before the rest of Practice Companion, make it available to TEEI Language learners and allow repeat attempts. This replaced an earlier plan limited to named test accounts.

On 25 October, the background-job infrastructure was installed in staging and production. L1 was released under an explicit exception to the internal evaluation gate, with the known challenge-case failure recorded. The content guardrail remained a separate part of the assessment path.

The released result was labelled as an estimated writing level. The product stated that the method had not yet been validated against teacher ratings, and the learner could choose a different practice level. These choices define the use of the result; they do not turn the failed internal criterion into a pass.

A production walkthrough using a synthetic test learner completed eight answers, returned a B1 estimate after 27.7 seconds and recovered the same job after a reload. The mobile viewport check found no horizontal overflow at 390 pixels. These checks exercised delivery of the estimate, rather than independently validating the level itself.

The production opening occurred while a staging delivery test was still running. That test later found that a callback redirected to a login page could be recorded as delivered after receiving an HTML response with status 200. The production callback had saved its tested result correctly. The delivery check was changed to reject redirects and HTML responses, and the fix reached production the following morning.

That sequence is part of the engineering record. A successful production walkthrough and a completed evaluation answer different questions, and the staging result arrived after the opening decision.

As of launch night, the record contained no completed assessment from a real learner. Every performance figure in this note comes from constructed conversations or synthetic walkthroughs.

What the evidence supports now

The work documents a functioning path from written answers to a stored, quoted estimate, with explicit refusal and insufficient outcomes. It shows how that path behaved on the particular constructed cases, how repeated readings changed the results, and what the model comparison found under its specified conditions.

It also documents several things that remain unestablished.

Agreement with people

No human-rating round has been opened for the level conversation. The code includes storage and calculations for blind ratings, including exact agreement, agreement within one band and linearly weighted Cohen’s kappa. It also contains a lock keeping rating rounds closed. The presence of those components is not a completed validation exercise or a finished operational rating service.

Human validation remains subsequent improvement work. To support claims about agreement, the instrument will need judgements made independently of its own outputs and an account of how disagreements among people were handled. This note makes no numerical claim about agreement with teachers.

Performance with learners

The synthetic cases allowed us to isolate failure modes, repeat inputs and keep a reference fixed while changing code. They do not represent a sampled learner population. Their labels were also generated within the same development process that produced the scorer.

Performance with learners will need its own evidence, including outcomes that are insufficient, refused or technically unsuccessful. A headline exact rate calculated only on returned bands would leave out an important part of that experience.

Comparable tasks and real progress

The system records the language, task plan, variant and scoring versions associated with each estimate. It can reject comparisons between different recorded instruments and avoid treating reuse of the same variant as a comparable repeat.

That traceability is necessary for later measurement, but it does not establish equivalent difficulty. The A and B variants, support and stretch branches, and variations in interviewer wording have not been empirically equated.

Nor have we established when a difference between two estimates represents learning rather than ordinary variation, task familiarity or a different opportunity to show a skill. Expressing the difference as a word instead of a number would not resolve that problem. For now, the product presents versioned estimates and practice choices.

Language and modality coverage

All the evaluation runs described here used English as the target language. Writing in other languages was used in diagnostic cases about whether an English level should be assigned; it was not a validation of assessment in those other languages.

The instrument observes writing. A spoken proficiency claim would require different evidence. Its A1-to-C1 rubric range is an intended scope, and the adequacy of the tasks across that scope remains unvalidated with learners.

Decisions we retained, and approaches we set aside

We retained three separate readings and the lowest eligible result because that combination met the earlier internal criteria and made the tradeoff against high readings explicit. The later L1 failures show its limit: agreement among calls does not resolve a systematic boundary error.

We did not introduce a model panel. The selected panel improved scores on the reused sets, but we had not established that the improvement would generalise. The background job changed the latency constraint, leaving evidence and cost as live questions rather than treating the old timeout as permanent.

We did not replace Sonnet with a cheaper model. Kimi K2.5 was promising under the historical comparison, but a model change creates a different scorer and calls for a new evaluation of that configuration. Its lower cost in one experiment is not enough to establish the product decision by itself.

We did not adopt a median or majority rule. The median failed the September challenge-case criterion. In a later October analysis, switching to a majority would have moved fifteen outcomes: nine towards the reference and six above it. That was a tradeoff to decide, not an automatic correction.

We set aside quotation-language filters and per-answer assessment notes after the experiments described above. The filters introduced new rules and dependencies; the prompt candidates and notes approach did not pass their full evaluations. Their targeted successes remain useful observations in the record.

We used a background job without adding streaming to the assessment result. It removed the need to finish the judgement inside the short request window and supported returning to an existing job. It did not remove all time limits or establish that every assessment will finish quickly.

The product returns an estimated band with evidence and practice suggestions. We did not add a confidence percentage, a certificate or a numerical progress score. The learner’s own practice choice remains separate from the stored estimate.

Building on this work

For TEEI, the useful outcome of this work is a clearer account of what the system observes, how it reaches a result and where the evidence stops. The question to answer next is how those observations compare with independent judgements of learner writing, and what kind of repeated conversation can support a claim about progress.

The Practice Companion engineering note explains the wider learning experience. The first writing-room note follows the feedback on one written sentence. This record supplies the detailed assessment experiments behind the level conversation.

We welcome conversations with language-assessment researchers and applied AI teams working on these questions. If your work concerns reproducible evaluation, writing assessment or useful feedback for learners, talk to us about building with TEEI. The test definitions, scoring rules and version history give us a concrete starting point for that discussion.

Appendix A: the complete model comparison

Exact agreement and missing outcomes, 16 October 2026

The following table reports the lowest-of-three rule for the 21 models whose scheduled runs completed. A completed run can contain unusable or failed assessments; it does not imply that every model returned a usable outcome. Fable’s incomplete run is reported separately above.

D, R and H mean development, regression and the reused September holdout. Exact agreement is shown as a numerator and denominator rather than a rounded percentage. The baseline has 72, 39 and 42 eligible band rows respectively. Missing usable outcomes can reduce those denominators. “No usable outcome” counts such assessment rows across all cases in each set, including cases outside the band metric, so it is not always the difference from the baseline’s band denominator.

“Passed” lists the sets on which the model met every internal criterion. Missing outcomes fail a set even where the remaining exact agreement is high. These are results for the historical prompt, rubric, adaptations and case set described in the main text.

Recorded cost and latency

Estimated cost is recorded token usage multiplied by the listed rates used on 16 October, averaged over the 171 model-backed assessment attempts. It includes the recorded prompt-cache reads and writes. Separate ApplyGuardrail calls, application hosting, job orchestration and other infrastructure are outside this calculation. Failed attempts remain part of the experiment; these are not costs per successful assessment.

Latency is the longest of the three reading durations in an assessment, because the combined path waits for them. The median and share above 28 seconds cover the 171 model-backed rows across all three sets, not only the development set. The 28-second threshold describes the architecture at the time of the experiment.

Fable 5.1’s partial run had an estimated recorded model usage cost of $51.29. The original report also showed $0.300 per scheduled model-backed attempt and a 27.9-second median across its recorded attempt rows. Because that run includes the credit interruption and missing results, we do not place those figures in the completed-run comparison above or present them as a cost and time per successful assessment.

The known-price total for the model experiment was $212.74, including the partial Fable run. GLM-5 was excluded because its price was not available in the pricing record. Historical rates were sourced from Amazon Bedrock pricing and Anthropic API pricing; the live pages may differ from the rates used on 16 October.

Appendix B: scoring rules and configuration

Combining eligible readings

This excerpt combines validated readings after the surrounding path has enforced its usable-reading requirement and handled content-guardrail refusals. It keeps a complete reading, rather than synthesising evidence across readings.

export function combineReadings(readings: ValidatedAssessment[]): ValidatedAssessment | null {
  if (readings.length === 0) return null;
  const scored = readings.filter(
    (r): r is ValidatedAssessment & { level: AssessedBand } =>
      r.status === 'scored' && r.level !== null,
  );
  const insufficientCount = readings.length - scored.length;
  const outcomes = readings.map((r) => (r.status === 'scored' ? r.level : 'insufficient'));
  const disagreed = new Set(outcomes).size > 1;
  const mark = (r: ValidatedAssessment): ValidatedAssessment =>
    disagreed && !r.quality_flags.includes('readings_disagreed')
      ? { ...r, quality_flags: [...r.quality_flags, 'readings_disagreed'] }
      : r;
  if (scored.length === 0 || insufficientCount >= scored.length) {
    const unplaced = readings.find((r) => r.status === 'insufficient') ?? readings[0];
    return unplaced ? mark(unplaced) : null;
  }
  const lowest = scored.reduce((low, r) => (BAND_RANK[r.level] < BAND_RANK[low.level] ? r : low));
  return mark(lowest);
}

Verifying a quotation

The quote must occur in the learner’s answer to the specified turn. This check establishes the text’s provenance, not whether it justifies the assigned band.

function validEvidence(evidence: readonly Evidence[], turns: readonly AssessTurnT[]): Evidence[] {
  return evidence.filter((row) => {
    const turn = turns.find((t) => t.index === row.turn);
    if (!turn || turn.skipped) return false;
    const needle = normaliseForAnchor(row.quote);
    return needle.length > 0 && normaliseForAnchor(turn.answer).includes(needle);
  });
}

The writing floor

export const ASSESS_MIN_SENTENCE_TURNS = 2;
export const ASSESS_MIN_WORDS_PER_SENTENCE_TURN = 5;
export const ASSESS_MIN_TOTAL_WORDS = 15;

Configuration released on 25 October

The seven numerical evaluation criteria

The criteria apply to every required set, using that set’s own rows and the metric definitions described above. A technical error or an inappropriate refusal also fails the evaluation.

These thresholds define TEEI’s internal evaluation of the constructed cases.

Version tracking and comparison

Each stored assessment includes provenance for the language, modality, task plan, variant, rubric, system prompt, tool schema, model, inference configuration and scorer fingerprint. It records the relevant guardrail information as provenance as well.

The comparison key is a SHA-256 hash of the language, modality, task-plan version, rubric version, system-prompt version, tool-schema version, model identifier and scorer fingerprint. A changed value in that key prevents the application from treating two assessments as the same recorded instrument.

The question variant is checked separately. The guardrail version is not part of that key. Neither the key nor a different A/B variant proves that two conversations have equal difficulty. They provide traceability and rule out some inappropriate comparisons; the empirical measurement work remains separate.

Appendix C: the development timeline

Research records and references

The measurements in this note come from TEEI’s versioned development records. The principal records are listed here so that the experiments, dates and result definitions can be distinguished. The tables above reproduce their reported aggregates, with explicit corrections to denominators and outcome categories; they are not newly run model evaluations.

• September evaluation: the 25 September 2026 approval record covering the 60-case run; the instrument matrix and pilot-readiness notes for the level conversation.
• Model comparison: the 16 October 2026 research document, stored-results analysis and Bedrock-only sharing document. The broad and Bedrock-only analyses use different panel pools. Model adaptation, token usage and latency records accompany the comparison.
• Conversation changes: the versioned task plan and the 20 October help-wording experiment; the 23 and 24 October records for mobile interaction, output allowances and background execution.
• L1: the 24 October 2026 estimate report, the completed 485-row result file and its summary, the frozen fourth holdout and the separate blind AI review. The raw result categories are the basis for the complete L1 distribution in this note.
• Release: the 25 and 26 October level-delivery record and dated launch notes, including the production walkthrough and callback correction.
• CEFR: Council of Europe, Common European Framework of Reference for Languages: Companion Volume, 2020; classroom assessment guidance; and relating examinations to the CEFR.
• Infrastructure: AWS, HTTP API quotas in API Gateway. Historical model-cost estimates use the pricing assumptions recorded with each experiment, rather than current prices substituted after the fact.

The underlying TEEI records are internal engineering material. This article does not imply that every raw file or internal test conversation has been published. We can walk prospective research and technology collaborators through the method and the appropriate source material through the technology partnership route.