Why our certificate repairs need a new R2 address

Until 24 September 2026, a repair of a credential’s badge artwork could overwrite files we had already served as immutable. Anyone who had cached the broken image could keep seeing it for up to a year. We did not measure how many readers that was. Now we write every repair to a new generation in Cloudflare R2 and point the verification page at the generation that is ready. We draw the badges and PDF certificates with Chromium on AWS Lambda, and in our code we hold every delivery until all four files are stored. We can repair an object in R2 without repairing the copy a reader already holds.

A credential at TEEI recognises verified participation on TEEI Language or TEEI Mentorship: a PNG badge in three sizes, an A4 certificate as a PDF and a page anyone can open from a link. It is not a course and it is not a qualification. This note is part of the series that starts with How TEEI Runs Its Platforms on Cloudflare Workers. In that article we describe the Workers; here we follow the drawing work off Cloudflare and back into R2.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

A repair that the people who had seen the bug could not see

Until that day we picked generation 1 for any drawing job that was not ready. When staff requeue a drawing job for repair, we clear its recorded generation. So the repaired artwork was written over the very generation 1 objects it was meant to replace. Those objects were served at the time as public, max-age=31536000, immutable, and an immutable response is one a browser keeps without asking again. R2 was not the problem: its writes are strongly consistent, so any read from the bucket after the write sees the repaired bytes. A browser that holds a fresh, immutable copy does not ask again, so we give the repair a new address rather than wait for that copy to expire.

We now ask the bucket, not the job row, because the row is exactly what a requeue erases:

Verbatim excerpt.

export function isNotFound(err) {
  const status = err?.$metadata?.httpStatusCode;
  return status === 404 || err?.name === 'NotFound' || err?.name === 'NoSuchKey';
}

export async function firstFreeGeneration({ exists, keyFor, start = 1 }) {
  const from = Number.isInteger(start) && start > 0 ? start : 1;
  for (let generation = from; generation < from + MAX_GENERATION_PROBES; generation++) {
    if (!(await exists(keyFor(generation)))) return generation;
  }
  throw new Error(`credential_artwork_generation_exhausted:${keyFor(from)}`);
}

We probe only the main badge, the first file we upload. Because we upload the four files one after another, a run that fails partway still leaves that generation occupied. The search fails closed: only a definite “not found” frees a generation, while a permissions error or a network error throws, and that credential waits for the next run instead of overwriting a published object on a guess. In the code we read on 15 October 2026, we stop after 50 probes. In five unit tests we check, among other things, that a requeued credential never reuses a published generation and that an unknown error never frees one.

The fix landed five days before the move to Lambda, and the rendering script we run on Lambda uses the same generation search. On the public verification page we show the generation that is ready, which is how a holder sees the repaired version. A new generation does not change the address of anything already shared; the page that points at the current generation has to carry the repair.

Publishing a generation: four files, four ETags, then one row

A new address is only safe if nothing points people at it before its files exist.

The credential starts in the platform that runs TEEI Language and TEEI Mentorship, a React Router 8 application on Workers. In our code as of 15 October 2026, on a cron trigger every quarter of an hour, we issue at most 25 credentials to people who have passed a threshold of completed, verified sessions in one programme; the cap counts credentials issued, not people looked at. We write the issuance claim, credential, drawing job and deliveries in one atomic database batch. For a claim we own, we make its final statement raise a database error unless it finds the credential, one queued drawing job and the expected number of deliveries, so the batch fails as a whole. We insert the drawing job from the claim itself, so this insertion queues artwork only for the claim owner, and is safe to run twice. If you have read our note on a transactional outbox, the shape is familiar: the business fact and the work it causes are written together, or not at all.

We hold credential deliveries until all four primary files have been stored. In the same batch we always queue an in-app notification delivery, and an email delivery when credential emails are switched on; we dispatch neither at issuance. Those deliveries wait until the drawing job is ready, and so do the push we send with the notification and the artwork we show for sharing. Ready has a strict definition in SQL. The fragment below shows the readiness conditions; we leave out the expression that checks the four expected object keys and mark where it goes:

Adapted for publication.

-- URL strings never participate in readiness.
    artwork.status = 'ready'
AND coalesce(length(trim(artwork.badge_800_etag)), 0) > 0
AND coalesce(length(trim(artwork.badge_square_etag)), 0) > 0
AND coalesce(length(trim(artwork.badge_og_etag)), 0) > 0
AND coalesce(length(trim(artwork.certificate_pdf_etag)), 0) > 0
AND artwork.artifact_generation > 0
AND coalesce(length(trim(artwork.renderer_release)), 0) > 0
AND length(artwork.template_sha256) = 64
AND artwork.template_sha256 = lower(artwork.template_sha256)
AND artwork.template_sha256 NOT GLOB '*[^0-9a-f]*'
AND /* omitted: the four object keys are the ones this job expects */

A missing ETag counts as length zero, so a status of 'ready' alone cannot release a delivery. We record an ETag only from R2’s answer to an upload and stop without one, so a row with four non-empty ETags, a generation, a renderer release and a well-formed template hash describes four files that were stored. In our code, we release credential emails only after all four files are stored; we have not measured this guarantee in production, and the unit tests for this module cover its state, URL derivation and enqueueing, not the email itself.

On AWS we write to R2 through its S3-compatible API with the ordinary AWS SDK, so code that runs outside Cloudflare can write to a bucket that a Worker reads through its binding. The order of operations is the contract:

Adapted for publication.

// Never write over a published generation (see the previous section).
const generation = await firstFreeGeneration({
  exists: publishedObjectExists,
  keyFor: (g) => keysFor(credential, g)['badge-800'],
  start: generationHint,
});
const keys = keysFor(credential, generation);
const buffers = await renderCredentialArtefacts(browser, credential, {
  history,
  onFontCheckFail: (kind, face) => {
    throw new Error(`font_check_failed:${kind}:${face}`);
  },
});
const etags = {};
for (const artefact of ARTEFACTS) {
  etags[artefact.kind] = await uploadAndGetEtag(keys[artefact.kind], buffers[artefact.kind], contentTypeOf(artefact));
}
await publishReadyRow({ credential, generation, keys, etags });

async function uploadAndGetEtag(key, body, contentType) {
  // Safe to retry: same key, same bytes, so a repeated PUT is the same object.
  const res = await withRetry('r2 put', () =>
    s3.send(new PutObjectCommand({ Bucket: BUCKET, Key: key, Body: body, ContentType: contentType })),
  );
  const etag = stripQuotes(res.ETag);
  if (!etag) throw new Error('r2_upload_no_etag');
  return etag;
}

We make issuance atomic inside the database. Artwork has a different contract, because it spans two stores: we upload the files first, without holding a database transaction open, and then publish the generation, all four object keys and all four ETags in one database write, the ready-row write. If we stop before it, we leave unused objects in R2 but release no deliveries for a new credential; we never publish a partial set.

When we redraw a credential that is already ready (a new look, or a new release), its old ready row stays in force until the new row replaces it. After publishing the row, we draw WebP contact sheets for the look picker; we exclude them from readiness. We upload the last sheet at the end, so its presence means the set is complete, and as of 15 October 2026, on later runs we redraw a missing set for credentials made ready in the past two days.

As of 15 October 2026, we issue every 15 minutes and render every 5; we have not timed the whole path from a completed session to a finished image.

Drawing on Lambda

We run the renderer as a Lambda function packaged as a container image with Playwright and Chromium inside, deployed on 29 September 2026. Before that, the same script ran on a workstation on a timer. We reserve concurrency of 1 for this function, so its invocations run one at a time; that matters because finding a free generation does not reserve it. We turn retries off on both the asynchronous invocation and the schedule, and we discard a scheduled tick after two minutes of waiting, where Lambda’s default would keep it for up to six hours, because we check for outstanding work again on the next run. Neither setting promises that a run is delivered exactly once, so we make each run safe to repeat.

In our configuration on 15 October 2026, we watch the function with three CloudWatch alarms: one for repeated errors, one for a missed schedule, and one for a run that comes close to its timeout. We bind the mode to the stage in code: staging is a dry run that writes nothing, and in a test we check that the stack offers no property to set the mode from outside.

We start the browser with the usual flags for Chromium on Lambda and leave one out on purpose: --single-process. In a launch-code comment we record that we saw that flag change how type is rasterised on our Story card image, and on a certificate the type is the product.

Each credential becomes four files: badge-800 (800 by 800), badge-square (400 by 400), badge-og (1200 by 630) and certificate.pdf (A4 landscape). We draw three of them in Chromium, one short-lived page each, and keep the results as buffers in memory. Before we keep a drawn file, we check that the display family has loaded:

Verbatim excerpt.

      await page.setContent(html, { waitUntil: 'load' });
      await page.evaluate(() => document.fonts.ready);

      const hasWords = await page.evaluate(() => (document.body?.innerText ?? '').trim().length > 0);
      if (hasWords) {
        const displayFaceLoaded = await page.evaluate(
          (face) => document.fonts.check(`16px '${face}'`),
          expectedFace
        );
        if (!displayFaceLoaded) {
          onFontCheckFail?.(artefact.kind, expectedFace);
        }
      }

document.fonts.ready resolves when the fonts the page has asked for have finished loading, but it resolves just as happily when a face failed and the browser fell back to a system serif. document.fonts.check is the question that catches the fallback: it returns false while a face the page declared has not loaded. For a face the page never declared, it returns true, so it guards declared faces only. We embed the fonts in the HTML and check the declared regular Lora face; we do not verify every weight or script subset. In production, we abort the render when the display font check fails: we upload none of that credential’s new primary files, and we retry on a later run. We draw the look picker’s sheets with the same check after the ready row; a failure there fails the run, but the primary files are already ready.

The square badge is the one file we do not draw as a page. Until 5 October 2026 it was the same card under CSS zoom: 0.5, and at that scale Chromium spaced the letters of the verification line unevenly. Now we scale the 800-pixel badge down with sharp.

A run must also end before its timeout. We added the loop guard on 5 October 2026, after a stop could land between a credential’s ready row and its look picker’s sheets, leaving a row that read complete while its sheets were missing. We time each attempt from choosing a generation through those sheets, failures included. We assume 30 seconds before the first measurement, then use the slowest attempt; we start another credential only while 1.5 times that still fits before a deadline 35 seconds ahead of the Lambda timeout, and the rest wait for the next run. Separately, we ask the rendering process to terminate 20 seconds before the timeout and force it five seconds later. In our boundary tests we start an exact fit and reject one millisecond over.

Two buckets, two contracts

Generations protect a repair. Revocation needs something else. The credential files and the session share cards live in two R2 buckets, and in our code as of 15 October 2026 we give revocable credentials a short cache lifetime and public share cards immutable addresses.

Credentials can be revoked. So for credential files we chose a short cache. Our code for them sets a one-hour max-age and never marks a file immutable:

Adapted for publication.

// One hour and never immutable: a later revocation must not stay pinned in a cache
const cacheControl = 'public, max-age=3600';

For responses that carry this header, one hour is the freshness lifetime. We cannot change copies that a browser or proxy cached earlier: once a copy has left, we cannot control how long it lives.

We chose not to make share cards revocable. A session share card describes a session that is already public. Its address changes whenever its content or layout changes, and we never write over a card that already exists. We send PutObject with IfNoneMatch: '*', so an upload succeeds only if the key is absent; if another request has stored the card first, we keep that card. So an address always names the same file and the card can be served public, max-age=31536000, immutable. The design treats published artwork as a fixed public artefact. A new address preserves the earlier artefact; it does not recall copies that have already been cached or shared. And we never cache a missing card:

Verbatim excerpt.

// A missing card may be written a moment later, so a 404 must never be cached anywhere.
function notFound(): Response {
  return new Response('Not found', {
    status: 404,
    headers: { 'Content-Type': 'text/plain; charset=utf-8', 'Cache-Control': 'no-store' },
  });
}

A card link can be shared before the renderer has finished. If the first request had cached a 404 for a year, the card could stay invisible to that reader until the cached miss expired.

We draw the cards in a separate Lambda service, with Satori and sharp rather than Chromium. We keep the card renderer’s write access separate from credentials, so no bug in it can overwrite a badge someone has put on their LinkedIn profile.

A note on Cloudflare Browser Run

Cloudflare’s headless browser service was called Browser Rendering until 15 April 2026, when it was renamed Browser Run. Our admin configuration retains a Browser Run binding for badge and certificate tools, through Quick Actions. A binding is not evidence of use, and we have not measured any: Cloudflare’s GraphQL dataset for Browser Rendering events returned no rows for 20 September to 15 October 2026, and we do not know whether that dataset counts Quick Actions. In August 2026 we also built a renderer Worker for the programme credentials on Browser Run. In its template we require every listed font to load at least one face and pass document.fonts.check, and every image to finish with a non-zero natural width, which is stricter than the check our Lambda renderer runs. We have not deployed it. We have no recorded comparison of the two renderers and have not measured one against the other.

When this is the wrong pattern

• If nobody has cached your files yet, overwriting is simpler than generations. Generations exist because an overwrite cannot replace the copies that readers have already cached at the same address. The price is that a repair does not change the address of anything already shared, and a failed attempt can leave unused objects behind: once its first upload has occupied a generation, the next attempt chooses another one rather than completing that partial set.
• If each address names fixed content that you will never withdraw, make it immutable. Revocable files cost you a short cache lifetime. We designed our share cards not to need one.
• If people expect the image at once, a schedule is wrong. A batch every five minutes, with the deliveries held until the files exist, trades immediacy for drawing outside anyone’s request. That suits a certificate. It does not suit an interactive preview.

Verify this

Reproduce the font check and the PDF (public reproduction). In a project of your own, install playwright@1.59.1 and its bundled Chromium with npm install --save-exact playwright@1.59.1 and npx playwright install chromium. Save the script below as reproduce.mjs, and place a regular Lora WOFF2 file beside it as lora-latin.woff2; the local font fixture is the face used in this example. We embed that file in the HTML, so rendering makes no font request to another site. The script loads the HTML, waits for fonts, checks the display face and prints A4 landscape with backgrounds.

Adapted for publication. Revised on 22 October 2026 to use a local font fixture and expose the three control variants as arguments.

import { readFile } from 'node:fs/promises';
import { chromium } from 'playwright';

const variant = process.argv[2] ?? 'loaded';
if (!['loaded', 'broken', 'undeclared'].includes(variant)) {
  throw new Error('Use loaded, broken or undeclared');
}
const font = (await readFile(new URL('./lora-latin.woff2', import.meta.url))).toString('base64');
const fontRule = variant === 'undeclared' ? '' : `
  @font-face {
    font-family: 'Lora';
    src: url(data:font/woff2;base64,${variant === 'broken' ? 'AAAA' : font}) format('woff2');
    font-weight: 400;
    font-style: normal;
  }`;
const html = `<!doctype html><html><head>
  <style>${fontRule}
    body { margin: 0; font: 48px 'Lora', serif; background: #f4efe6; }
  </style>
</head><body><p>Certificate of participation</p></body></html>`;

const browser = await chromium.launch();
try {
  const page = await browser.newPage();
  await page.setContent(html, { waitUntil: 'load' });
  await page.evaluate(() => document.fonts.ready);
  const ok = await page.evaluate((face) => document.fonts.check(`16px '${face}'`), 'Lora');
  console.log({ variant, fontLoaded: ok });
  if (!ok) throw new Error('font_check_failed');
  await page.pdf({ path: `certificate-${variant}.pdf`, format: 'A4', landscape: true, printBackground: true,
    margin: { top: '0', bottom: '0', left: '0', right: '0' } });
} finally {
  await browser.close();
}

Run node reproduce.mjs loaded, then node reproduce.mjs broken, then node reproduce.mjs undeclared. The broken variant declares a deliberately invalid font data URL: document.fonts.ready still resolves, the check should return false and the script should stop before writing a PDF. The undeclared variant removes @font-face altogether, so the check should return true even though no Lora face was declared. It guards declared faces only. In our original experiment at 14:04 UTC on 15 October 2026, using Playwright 1.59.1 and its bundled Chromium, the loaded, failed and undeclared cases returned true, false and true. That historical run used an external stylesheet for the loaded case. We repeated the revised local-font example on 22 October 2026 with Playwright 1.59.1 and its bundled Chromium: true, false and true again; the broken case stopped with font_check_failed before producing a PDF.

Further reading

For one-to-one sessions, we require at least five minutes connected together, as we explain event by event in our Amazon Chime SDK note. What a holder sees is described in How You Earn Credentials with TEEI Language and on the public explainers for TEEI Language and TEEI Mentorship. How one application serves both programmes is in one codebase, two programmes.