How We Compare Builds and Verify a Worker Release

On 12 September 2026 we corrected a check that reused one measurement for two build manifests, so it compared every invariant with itself and could never reject a difference in build inputs. Our build-input comparison now uses separate readings and compares those inputs against a declared list of permitted differences. We also read back the version serving traffic on the named Worker. For our Astro website, we remove development-only files before recording the candidate’s digest, and promote only a matching candidate. Before rolling back TEEI Language or TEEI Mentorship, we assess whether the older code can work against the live database schema.

A check can pass without comparing anything. The remedy throughout this note is to read the other side of each comparison on its own: from the build, the Worker or the database. TEEI Language and TEEI Mentorship are two Workers built from one codebase; Evenhall is where partner companies’ managers work. The note belongs to our series on how TEEI runs its platforms on Cloudflare Workers. Its three release chains prove different things, which do not add up to one guarantee.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

What a deploy does and does not tell you

A zero exit code from wrangler deploy tells you that an upload succeeded somewhere, not that the Worker answering your users is the build you tested. In the words of the versions and deployments documentation: “Every time you change your Worker’s code or configuration, Workers creates a version. A deployment determines which version(s) are actively serving traffic.” With wrangler versions upload, we create a version without sending traffic to it; with wrangler versions deploy, we activate the version we name by its ID.

A version is code and configuration. The rollback documentation is plain: “Resources connected to your Worker will not be changed during a rollback.” Our database is outside Cloudflare altogether, so no rollback ever touches it.

Across our three release chains, five records answer five different questions, and none of them stands in for another.

TEEI Language and TEEI Mentorship: two builds of one commit

Nothing uncommitted rides along

On 8 July 2026 a production deploy included another session’s uncommitted feature code. Nothing broke, but we had built from the files on disk, so the commit in our deploy log did not describe what had shipped. On 9 July we added a guard to both deploy commands: before anything is built, a filtered status check over the paths that can change the artefact must come back empty. With the same guard, we take a lock per target, so two sessions cannot deploy the same Worker at once.

The environment is chosen when the code is built

We choose the environment before we build, through CLOUDFLARE_ENV. The Cloudflare Vite plugin writes a flattened Wrangler configuration with no named environments, so wrangler deploy --env staging cannot change the environment already built into it. In our build script, we reject unknown targets and environments. We learned why on 3 August 2026, when a preview was built as production and uploaded carrying the production bindings. Since 25 July each target has had its own output directory and build lock, so building Mentorship cannot overwrite a Language build that is about to ship.

Two builds, compared input by input

Because the environment is part of the build, staging and production are two builds of one commit. We accept that because the ENVIRONMENT variable separates the production flavour, with unrestricted email sending and the production analytics project, from everything else, and our pre-release test journeys must never run against it. We declare which build inputs may differ and compare separate readings of the inputs we measure.

In that comparison, we measure each build separately and read configuration values back from its generated configuration. We reject changed recorded invariants as source divergence, and an undeclared axis key even when both builds record the same value; a declared key that happens to agree is reported, not refused. We reject a changed npm installation record even when the committed lockfile agrees: a stray install can change the dependencies npm installed. We also reject two manifests recorded at the same instant, and any required input missing from either reading, even when both omit it. We compare inputs, not the bytes of the two bundles, because allowing a bundled JavaScript file to differ does not tell us which changes it contains.

The self-comparison from the opening was our first wiring of this check, which also took the permitted differences from flavour labels rather than from the builds. We found it by reading the code that called the comparison, not the comparison itself. In tests, a valid pair is now our positive control, and changed source or lockfile digests, or one required input omitted from both readings, must be refused.

Staging has a separate comparison: the artefact deployed to staging and its configuration must have the same sha256 as the local staging build, which proves staging is the build we made. The table compares the staging and production builds, and we keep the two apart.

Upload, then activate the version we uploaded

Since 15 September 2026 we have also released Language and Mentorship in the two steps Cloudflare separates. We upload an immutable version from a frozen copy of one commit, and after a human approval we activate exactly that version by its ID, never “the latest”.

Adapted for publication.

# Step 1: upload. No traffic moves.
npx wrangler versions upload --config <frozen-copy>/<build-output>/wrangler.json
#   We take the new Version ID from Wrangler's output; that is the ID we activate.

# Step 2: activate exactly that version on the named production Worker.
npx wrangler versions deploy --version-id <version-id> --percentage 100 \
  --name <production-worker> --config <committed-root-config> --yes

We pin the generated build configuration for the upload and the committed root configuration for the activation, because activating a stored version also applies the logging and observability settings of the configuration passed to that command. Migrations run separately, under an approval of their own.

Reading the version back from the named Worker

On 6 August 2026 an audit found that an upload could reach the other programme’s Worker. A build that generates a Wrangler configuration also writes a redirect file, .wrangler/deploy/config.json, and when we omit an explicit configuration, we let Wrangler resolve it for deploy, versions upload and versions deploy. Building both programmes in one checkout, we used whichever build had run last. Our Language probe checks a URL without naming a version, so a misdirected upload would have been probed on the untouched, healthy Language Worker, and passed.

The fix on 7 August has two parts. We pinned the configuration explicitly at five release call sites, which also gets us Wrangler’s confirmation that it did not follow a redirect. In tests, we reject a wrong Worker name inside the right configuration file. Then we read the deployments of the Worker we named and compare the version serving traffic with the one we uploaded, whose ID we take from Wrangler’s own output; a later call would only report whatever is active by then. In a traffic split, a version at zero percent does not pass. The check has three outcomes: the named Worker is serving our version, verification failed, or the check could not be completed. The third is not evidence against the release, and we never roll back on it. Before we activate, we capture the version the release replaces on that same Worker; it is the one a rollback would restore.

A probe that could not reach the network

After a deploy, we confirm the version and probe the live service. When we probe Language, the answer is ok, failed or inconclusive. The third answer comes from 30 July 2026. That morning, during a Language deploy, we recorded probe=FAILED at 10:30 UTC. The operator’s own network was down: requests to Language, Mentorship and cloudflare.com all failed in the same moment, and the automatic rollback could not run either. At 10:36 we recorded the new version as live on all traffic and healthy. Had the network come back a few seconds later, we would have reverted a healthy version. Now, when the Language probe fails, we check whether the network itself is reachable; if it is not, the answer is inconclusive, and we never roll back on it.

A probe is also a contract with production as it is today. The deploy logs we counted on 15 October 2026 hold two automatic rollbacks, one per programme, on 22 and 28 July. Both were triggered by probe problems: they demonstrate the rollback path, not confirmed application failures, and we corrected both probes. On 22 July 2026 the Language probe failed because a heartbeat key the new version had just added had never been seen; the scheduled jobs were healthy. On 28 July, after three failed Mentorship probes since 26 July, we restored the previous version, and the probe failed on that too: it still expected the closed service Mentorship had been before it opened on 24 July, and “could not pass any code that was ever deployed”, in the words of the commit that fixed it. We roll back automatically on a failed probe only in the release script for Language and Mentorship; for the website and Evenhall a person decides.

The website: one build, promoted

For the website, we take the opposite approach. Its build is the same for both environments, so we promote the build that served staging instead of building again. We run the staging gate against the deployed staging site, and we compute the manifest digest over the local promotion candidate after removing the static files that exist only for development, so the gate result and the digest describe different observations in one release. We promote only when the gate passed and the candidate’s digest matches the manifest.

The digest is a sha256 over the candidate’s code and static assets: we sort the paths in a byte order independent of the machine’s locale, hash the files in that order, then hash the ordered checksum records, which include the paths, so moving a file changes the digest. On 1 June 2026 a locale-dependent sort gave one identical build two different hashes.

Two fixes on 22 September 2026 changed what the website records. Until then, we read the commit for the manifest after the build, the deploy and the staging gate, in a working tree where several sessions commit all the time, so it often belonged to someone else’s work. We now capture it once, when the deploy starts. In the words of that commit: “The commit says roughly where the source was; the hash says exactly what was built.”

The second fix was a rollback target that could never have worked. We had recorded it from wrangler deployments list, and the id we read there is a deployment ID, not a version ID; asked for it, the versions API answered 404. We now record the version from wrangler deployments status --json, which describes the deployment serving traffic. As a negative control, we ran the old capture against production and got a 404; the corrected capture recorded the active version, which the versions API could read. The website never rolls back by itself: if the production smoke test fails, we print the rollback command, and a person runs it.

Evenhall: the identity written into the version

For Evenhall, we answer the same question a third way. In its promotion script, we require a person’s staging verification for exactly this candidate, by commit and git tree, and stop if the source being built differs from the commit it claims to be. We pass the commit and tree as variables of the version we deploy, and since 13 September 2026 we ask the running Worker for them afterwards and compare them with what we meant to ship.

That describes the promotion script we inspected. It is not evidence that every recorded activation used that script or passed its staging gate.

On 6 September 2026 we found Evenhall querying a column production did not have. Production was 21 migrations behind the code, and the actions that need an audit write answered 503. Since that day, we stop at the same gate if the actual schema lacks a declared column the code needs without a fallback; the highest migration number cannot prove that column exists.

Evenhall’s operations guide adds one more trap: when we run wrangler secret put, we create and deploy another version, so the rollback target is the version active at 100 percent immediately before the change, not the last code upload.

Rollback moves code, not data

When we rehearsed wrangler rollback on a staging Worker on 2 July 2026, the command took seconds, which says nothing about whether the older code can work against the database as it is now. Since 12 September 2026, when we roll back Language or Mentorship, we first decide compatibility against the live schema, then roll back with the version and the Worker named explicitly, read the deployments back and probe. The target comes from an operator argument or the release journal, never from a line of the deploy log. We have not timed our current production rollback, which does more than the rehearsal.

What can be rolled back onto today’s schema

For an identified candidate, the verdict is SAFE, RESTRICTED or UNSAFE; without an identified candidate and a readable manifest, we report that we could not determine compatibility. If we cannot read the live migration ledger, the verdict is RESTRICTED. If the ledger does not satisfy the candidate’s required set of migrations, it is UNSAFE. An older migration head is expected for a rollback candidate, and we resolve aliases and historical equivalents first, so a renamed ledger entry does not become a fictitious new schema change. Every migration beyond the required set must then pass two layers; if either does not hold, the verdict is RESTRICTED. SAFE holds for one candidate and one observed schema, and only for what the contract examines: the required migrations, the excess migrations’ declarations and the candidate’s non-interpolated writes against the reconstructed schema. It is not evidence about every data or application outcome. If the ledger moves, we decide again.

Layer A is the classification contract: every excess migration is declared additive, and a structural report over them carries none of the contract’s removal flags, namely tokens removed from a redefined trigger, columns removed from a view or table, or a dropped object other than an index. Layer A alone is not enough, and we wrote down why on 11 September 2026: “A structural diff showing only additive changes does NOT authorise moving the ground under an artifact nobody executed against the new schema: a new UNIQUE index, CHECK constraint, foreign key or trigger is structurally additive and can still refuse an old artifact’s writes.”

Layer B is the proof. We read a corpus of SQL write literals from the candidate’s committed application libraries and rebuild the candidate’s schema and the current schema in two scratch databases from the checked-in migrations. With EXPLAIN, we prepare each non-interpolated write against the reconstructed current schema. If preparation fails, we try the candidate’s own schema as a control: a write both schemas reject is excluded, while a write only the current schema rejects is a compatibility failure. We then look for refusal guards that the current schema has and the candidate’s does not: triggers new or changed for the matching write event, and a higher count of CHECK clauses in a table definition, which counts against inserts and updates on that table. We attribute those guards from the schema definitions; we do not observe them firing on requests with real arguments, so we count a relevant new guard against the candidate even when its condition might not fire. For production, we bind the report to the live migration ledger.

Rollback also restores the bindings and secrets a version was uploaded with. On 21 September 2026, wrangler versions view showed two secrets missing from older versions, one of them needed by a callback route that a rollback would have turned into a 503. To run older code with today’s secrets, we upload that code again as a new version. Cloudflare also does not allow a rollback when the target version binds to an R2 bucket, KV namespace or queue that no longer exists.

The rollback decision once had the flaw from our opening. Until 12 September 2026, in the schema check for automatic rollback, we read a stale value from the deploy log, then fell back to a value derived from the constant we were checking against, so it passed by construction. We now decide against the live ledger, and we count a rollback as verified only when both the version check and the probe succeed. An explicit failure takes precedence over an inconclusive result; without a failure, an unavailable check leaves verification inconclusive.

Two Workers, one database, and 14 October

Language and Mentorship share one Turso database, so a migration released with one programme changes the ground under the other. With a cross-programme gate, we decide, for the versions both programmes serve, whether the shared schema can advance independently, needs a coordinated release, or is unknown. We treat a missing or mismatched required migration as unknown. When a serving build predates the plan’s last migration, we require a coordinated release even when the plan is labelled additive; that label describes risk, not permission. The one exception is a plan that creates only tables that did not exist, with their own indexes, triggers and inserts, a class we recompute from the SQL we are about to run and the live schema. A reference to an existing table takes a plan out of it: once new code writes child rows, a foreign key can make the old code’s DELETE fail.

On 14 October 2026 we activated a Mentorship version whose build needed migrations up to 0590 against a production ledger that stood at 0573. The release record names the cause: “operator did not read the production ledger before activation”. Signed-in requests hit a table that did not exist yet, and we recorded seven 5xx responses in 228 runs in the incident log. The rollback verdict for the previous version was RESTRICTED, so we went forward. We first activated the matching Language version so the cross-programme gate could admit the migrations, and read its answer change from “coordinated release required” to “independent”. After a snapshot and a dry run, we applied the 17 missing migrations and read the ledger head back: 0590. The incident log sums up the policy: “Rollback: none by hand. Additive tables, columns and triggers; forward recovery is the default.”

Trade-offs

Two builds trade byte identity for an explicit comparison of measured inputs. We use them because our framework’s Cloudflare integration fixes the environment at build time. A dirty-tree guard is for shared checkouts: if every deploy runs in CI from a fresh checkout, it has little to catch, though you still need to record the build’s inputs and the artefact’s identity. Automatic rollback needs a probe that can tell “broken” from “unreachable” and from “new”. We can split traffic between two versions, but we have not run a canary or a blue-green switch against production and do not claim those modes as proven. And if your migrations remove or tighten anything, a code rollback can be worse than the bug it reverses.

Verify this

In your test project, run npx wrangler versions upload, note the version ID, then run npx wrangler versions deploy <version-id>@100 and check that npx wrangler deployments status names that version and its share of traffic. For runtime version identity, add the version metadata binding and return its id from a test route. If your framework generates .wrangler/deploy/config.json, build a second environment in the same directory and inspect that file before the next command: without --config, you use the configuration named there.

Our own services report where they came from. curl -s https://language.theeducationalequalityinstitute.org/api/lc/health and the same path on mentorship.theeducationalequalityinstitute.org return a commit field: a 12-character commit prefix, with -dirty for uncommitted build-relevant changes, or unknown when we cannot establish where a build came from. The field reports where a build says it came from; it does not show that the bytes serving traffic are identical to an artefact we tested. The build comparisons and schema decisions behind it rest on private release records.