Keeping Astro Content, Sitemaps and Search in Sync

Our preview reader turns draft: true into draft: false. That lets us preview an article through the same template as a published post, but it changes what the next consumer can infer from the field. If we built a search index from that view, we could freeze draft titles into a public file. We therefore preserve a separate preview marker and filter it out when building the index.

The same Markdown folder also feeds RSS, public JSON, text exports and article pages. The sitemap takes a different path through built routes and HTML. On 15 October 2026 we compared six production surfaces and found the same 64 English article addresses. This note explains the eligibility rules behind those outputs, where they can disagree and how to compare membership rather than counts.

Built through volunteer contributions. TEEI’s software platforms are built through volunteer work, with NOK 0 spent on paid software development. Infrastructure and third-party services are separate from that figure.

Seven surfaces from one folder

We run the website as an Astro 7 site on a Cloudflare Worker, alongside TEEI’s other platforms, and keep the blog as a content collection: a glob() loader over the Markdown folder and a Zod schema every file must pass. We decide where a post may appear with three fields: lang (en, uk, no or fr), draft and unlisted, both false unless we set them in a post.

We make most surfaces at build time, through page templates, the sitemap integration and static file endpoints. We answer the main RSS feed and the search records in the Worker on every request, using the collection loaded at build time. A request-time generatedAt timestamps a response; it does not mean we reread the Markdown then. A content change reaches these surfaces through a new build.

The selectors below describe our source implementation checked on 15 October 2026. The production comparison later in this note is a separate observation from that day.

Authored state and the preview view answer different questions

Astro’s documentation shows an optional filter callback on getCollection() that filters on draft. In our feeds and JSON files we do exactly that, each in its own file, with the language rule its surface needs.

Adapted for publication.

// RSS: published posts in every language
const posts = await getCollection('blog', ({ data }) => !data.draft);

// llms-full.txt: published posts in three languages
const INDEXED_LANGS = new Set(['en', 'no', 'uk']);
const all = await getCollection('blog');
const indexed = all.filter((p) => !p.data.draft).filter((p) => INDEXED_LANGS.has(p.data.lang));

For article pages and both search surfaces we use one shared reader. It is the only place in the site’s source where we read the draft preview flag, a boolean astro:env server variable. With that flag on, we can view drafts in place, each with a Draft mark. The transformation is a pure function:

Adapted for publication.

export function c5Collection<T extends C5Post>(posts: T[], showDrafts: boolean): C5Visible<T>[] {
  if (!showDrafts) return posts;
  return posts.map((post) => {
    if (!post.data.draft || isUnlisted(post)) return post;
    return {
      ...post,
      data: { ...post.data, draft: false },
      c5Draft: true as const,
    };
  });
}

With the flag on, we copy each shown draft and its data, set draft: false on the copy and attach c5Draft outside data. After that rewrite, !data.draft no longer proves that an author published the post. Where we call getCollection() directly, we still read the authored state. Through the shared reader, we ask a second question, isC5Draft(): in the search indexes to exclude shown drafts, in the article template to label them and set noindex.

A prerendered search index needs its own publication filter

That second question matters most for the palette index, because we prerender it through the shared reader, and a prerendered file anyone can download keeps whatever we built it with. So we remove preview drafts from the built palette index even when the flag is on: in its filter we ask whether this is a build, not whether we enable the flag.

Adapted for publication.

/**
 * The posts we may list on a built search surface. On the dev server we keep every input post.
 * In a build we remove shown drafts whether the preview flag is on or not, because we keep in a
 * prerendered index what we built it with.
 */
export function searchablePosts<T extends C5Post>(posts: C5Visible<T>[], dev: boolean): C5Visible<T>[] {
  return dev ? posts : posts.filter((post) => !isC5Draft(post));
}

// The palette index, built with the site
const posts = publishedEnglish(searchablePosts(await loadC5Collection(), import.meta.env.DEV));

The build invariant is independent of the preview flag: no post marked c5Draft may enter the palette index. We enforce language and publication status separately through publishedEnglish. Algolia records always exclude shown drafts; the server-rendered search page excludes them outside the dev server.

In the unit test, with preview on, we exclude the draft from the build selection and keep it in the dev-server one, and check that nothing changes with the flag off. That test covers selector decisions. Inspecting the generated JSON is a separate check.

We take ranking from Algolia and eligibility from our collection

While Algolia is configured, we send each non-empty palette query to Algolia through the Worker once typing pauses, cancelling the previous request; without an answer from Algolia, we search in the Worker ourselves. For an empty query, and when the Worker’s search route is unavailable, we use the prebuilt index in the browser. We fill Algolia from the search records and cover the sync and fallback in How Algolia powers multilingual search at TEEI. For a blog query we ask Algolia for postId only and map the hits back onto the posts we may show.

Verbatim excerpt.

export function postsFromAlgolia<T extends C5Post>(posts: T[], answer: AlgoliaAnswer): T[] {
  const byId = new Map(publishedEnglish(posts).map((post) => [post.id, post]));
  return answer.hits.flatMap((hit) => {
    const post = typeof hit.postId === 'string' ? byId.get(hit.postId) : undefined;
    return post ? [post] : [];
  });
}

We decide from our own collection which ids may appear, and walk Algolia’s hits in the order we received them, so rejected hits disappear and the survivors keep Algolia’s ranking. We take each title and link from the local post, not from the index. If the index still holds a post we have taken down, or we ever fed it a draft or an unlisted article by mistake, we drop the hit before we answer. In a unit test we pass hits for a draft, an unlisted post, a Norwegian post and a published English post, and check that only the English one comes back.

We request no replacements for rejected hits, so we can show fewer articles even when more eligible results exist, and we cannot recover a published post Algolia did not return: we limit which hits we show, not whether the index is complete. Nor do we promise readers identical results on every path: we match different fields with Algolia and our two fallbacks, and in the Worker we bound the query first.

Drafts, the unlisted state and posts held back

Our publication pipeline handles three unpublished cases, and a post in each one leaves the surfaces by a different route.

We leave a draft out of every surface’s filter, and we build no English article page for it. For one English draft in production on 15 October 2026 at 14:18 UTC, we got a 404 for its page and for its Markdown copy.

The unlisted state is for a post meant to be reached through its link without appearing in discovery surfaces. The state takes draft: true plus unlisted: true, so wherever we leave drafts out, such a post stays out too. Only the English article route builds it, with robots set to noindex, nofollow. These are publication and indexing rules, rather than access control. If someone sets unlisted: true and forgets draft: true, we would list that post everywhere while its author believes it is unlisted, so the article route rejects that combination at build time. A single visibility field can avoid invalid combinations when a publication model outgrows two booleans.

Posts held back by a build flag are the bluntest tool: while the flag is off, we exclude them with negated patterns in the glob() loader, so they exist for no surface.

How we build the sitemap from pages and routes

Through @astrojs/sitemap we list the pages our build produced and every route with a fixed path. In its filter(page) we receive only a URL, never the post behind it, so there we cannot tell a draft from a published article. In a production build with the draft flag off, the drafts our article template handles never reach the filter, because we do not build them. Pages in the unlisted state are built, so we remove them later: after astro build finishes, we open the sitemap, find the built file for every <url> and classify it.

According to our code comment, on 10 July 2026 we falsely reported no noindex pages because we matched only quoted attributes. We minify our production HTML with unquoted attributes, so we now match both forms. The pattern still expects name before content, the order in which our SEO component writes the robots tag. In a test with synthetic tags on 22 October 2026, it missed noindex when content came first and read a data-name=robots attribute as the robots name. It checks the HTML we generate, not arbitrary markup.

Adapted for publication.

// Prod HTML is minified (unquoted attrs), so the match does not depend on quotes.
const NOINDEX_META =
  /<meta[^>]*\bname=["']?robots["']?[^>]*\bcontent=["']?[^">]*\bnoindex\b/i;

export function classifyUrl(url, distDir) {
  const pathname = pathnameOf(url);
  // We omit our other path and page-validity checks here.
  const file = distFileFor(url, distDir);
  if (!fs.existsSync(file)) {
    if (isServerRenderedIndexablePath(pathname) && builtWorkerHasRoute(pathname, distDir)) {
      return { indexable: true, reason: 'ok-server-rendered' };
    }
    return { indexable: false, reason: 'no-built-page' };
  }

  const html = fs.readFileSync(file, 'utf8');
  if (NOINDEX_META.test(html)) return { indexable: false, reason: 'noindex' };

  return { indexable: true, reason: 'ok' };
}

If an entry’s <loc> is not indexable, we remove the whole <url> block; in an entry we keep, we remove only the failing hreflang alternates. So a page in the unlisted state leaves the sitemap without anyone listing it. In return, we depend on the built HTML carrying the right robots directive: an otherwise eligible prerendered page without its noindex stays in the sitemap. We use one classifier to prune the sitemap and then to check it, so we apply the same policy twice: consistently, but with no independent proof that it is correct. We test its decisions separately, with fixtures.

Server-rendered pages need the opposite treatment: through the integration we cannot generate entries for dynamic routes in SSR mode, and we find no built HTML for them when pruning. We pass the URLs from one explicit list to customPages and keep each entry in the prune only when we find its exact route string in the generated Worker. So we depend on how that file is serialised, and we still need a rendered response to learn what the page returns.

When we ran the prune before the rename (24 September 2026)

We run the prune in a chain of post-build steps, and in a later step we rename the Worker’s entry file so we can wrap it with a scheduled handler. On 24 September 2026 we found that we had run the prune before that rename while having it look only for the renamed file. We mistook the missing file for missing routes and removed every opted-in server-rendered page from every sitemap we shipped, including the two pages we had added to that list on 23 September.

We fixed it the same day. We had the prune prefer the renamed entry file and fall back to its original name; after the wrapping step, the original name belonged to the wrapper, and only the renamed file still held Astro’s route manifest. We took the list of server-rendered pages from one source instead of a second, hand-kept list in the Astro config, verified each opted-in page after the build, and covered both entry names in regression tests. In the full build that day we found every opted-in route in the sitemap. We did not measure what the gap had cost in search results.

A later change split the server build into chunks, and Astro’s route manifest moved out of both entry files into a file under dist/server/chunks/. In our code as of 22 October 2026, the prune therefore checks the two entry files first and then the files in dist/server/chunks/*.mjs, and it uses the first one that contains route entries. A file that exists but holds no routes no longer counts, because it would drop every server-rendered page again.

Feed links must use the route’s identity

In Astro’s RSS recipe, each link is built as /blog/${post.id}/, which is correct whenever you use the collection id as the route’s slug, as we do. We give our ids a language suffix such as -en. On 9 May 2026 we stripped it from the feed’s links, and according to the note we left in the feed’s code, 50 of the 51 links then pointed at a 404. We now build each item of the main feed and the category feeds, link included, with one shared function and a path helper that keeps the suffix. We have no test dedicated to feed links; we check them from outside with the comparison below.

What we measured

We took all values with curl -s against the public site on 15 October 2026, starting at 13:51 UTC.

We reduced each surface to its set of English article addresses and compared each set with the sitemap’s: every difference was empty. We found no draft’s article address on any of them, checked against our list of drafts that afternoon. In a separate check at 21:50 UTC, none of the 63 draft titles then in our source folder appeared in the saved responses.

This establishes agreement for English article addresses on that day. It does not establish completeness: a post excluded by the loader can disappear from every output at once. Sets also hide duplicates. We read the records supplied to the sync, rather than the live Algolia index, and did not measure deploy lag or the sitemap in staging. We read the tests named in this note; we did not run them for that measurement.

Verify this

On our site, the sitemap, the palette index and the search records are public, so you can repeat the comparison on them. On your own site:

1. Choose one publication scope for every surface; ours was published English articles.
2. Take the article paths from your sitemap, every JSON index and your feed, normalised and limited to that scope.
3. Choose, independently of your collection, the set of addresses you expect.
4. Build with your preview flag off and on. Compare each listing against that set, using the generated files for prerendered outputs and responses from the same candidate for request-time outputs.

For each surface, keep the extracted paths in an array until after checking duplicates. This small comparison takes that array and your independently chosen expected paths:

Illustrative check for your own exports.

function compareSurface(expectedPaths, actualPaths) {
  const expected = new Set(expectedPaths);
  const actual = new Set(actualPaths);
  return {
    missing: [...expected].filter((path) => !actual.has(path)),
    unexpected: [...actual].filter((path) => !expected.has(path)),
    duplicateCount: actualPaths.length - actual.size,
  };
}

The inputs must use the same language scope and URL normalisation. A clean result has empty missing and unexpected arrays and zero duplicates. If both the sitemap and palette omit the same expected post, comparing them only with each other passes; comparing each with the independent expected set fails. This checks membership. Search ranking and the responses behind feed links still need their own checks.

Trade-offs

We copy the !draft filter into each file that reads the collection directly, so even with the preview flag set wrongly we cannot put a draft into the feeds or the JSON list. The cost is that we must add the filter to every new surface, and ask the second question wherever we read through the shared reader. Prefer a shared selector when every consumer needs the same publication policy; where language or preview behaviour differs, keep the differences explicit. We still lack a build in which we compare every surface against one expected set, so the 64-of-64 agreement stays a measurement, not a guarantee.

Configuration

Continue in the Astro series

The homepage case follows a different handoff, from a server-rendered activity line to browser updates. Next, how we compare builds and verify a Worker release explains how to carry a tested output through staging and identify the version serving production.