Artifacts & publishing

An artifact is everything Caffeine needs to answer a query, compiled once and served as static files.

The files

{publishPath}/{indexHandle}/
  current.json                the pointer     mutable, tiny, never cached
  {checksum}.index.json       the query data  immutable, cache forever
  {checksum}.payload.json     the card data   immutable, cache forever
  *.gz, *.br                  precompressed sidecars

The split between the pointer and the payload is what makes Caffeine work behind a full-page cache.

The payload files are immutable and content-addressed. The filename is a hash of the file’s own bytes, so the contents at a given URL can never change. They can be served with a one-year immutable cache header, and a browser that has one never asks again.

The pointer is stable and mutable. Cached HTML embeds the URL of current.json and nothing else — never a versioned URL. The HTML can be months old and still find today’s index, because the name it embeds never moves. It is the one file that must be served uncached, and at a few hundred bytes that costs nothing.

A versioned URL in the HTML would defeat the whole arrangement: the moment the index rebuilt, every cached page would point at a file that no longer existed.

What is inside

Records are addressed by position rather than by element ID, which keeps postings lists dense and lets both engines use array indexing rather than hash lookups on the hot path.

Most of an artifact is lists of integers — postings, permutations, value indexes. JSON spends about five bytes on an ID that a varint spends one or two on, and gzip cannot recover the difference because it cannot see that [1,2,3] is three small numbers rather than seven characters. So integer lists are stored as base64-wrapped delta varints, which is 14–31% smaller than plain JSON depending on index size, on top of gzip.

The payload — the card data — can be split into its own file, so a facet-count request never fetches it.

Publishing

Shards are written first, then the pointer that names them. A publish interrupted halfway leaves unreferenced files, which pruning collects — never a live pointer to a file that was never written. The reverse order would take the index down.

Writes are atomic. The pointer is rewritten on every publish while visitors are reading it, so a torn write is a hard error on a live page. Locally that means a temporary file and a rename; on an object store it comes from the backend, since an S3 PUT to an existing key is atomic.

An identical rebuild does nothing. Because filenames are content hashes, a rebuild that changed nothing lands on filenames that already exist. Caffeine compares the checksum before writing anything, and when it matches it writes nothing, spends no version and leaves the pointer’s timestamp alone — so nothing downstream sees a reason to refetch.

Versions

Superseded versions are kept, because a visitor who loaded the page a moment before a rebuild is still fetching the previous one, and pruning it out from under them turns a rebuild into a 404. The keepVersions setting governs how many; three by default.

Pruning is per file, not per version. Content addressing means versions routinely share shards — an edit that changes one product’s title leaves the entire facet index at its original path — so a retired version’s files are deleted only after checking that nothing retained still points at them.

Where it publishes

Into the local web root by default, served as a static file by nginx without touching PHP. Name a Craft filesystem instead — S3, a CDN-backed volume — and PHP never serves it at all.

.gz sidecars are written alongside every file, and .br where the Brotli extension is available. Where it is missing, nginx simply never finds a .br and falls back to the .gz, so its absence costs nothing.

When a build fails

The previously published artifact stays live and untouched. Nothing is written until the whole thing succeeds, the pointer is swung last, and the stale records stay stale so the next run retries them.

This is a chosen behaviour rather than an accident: a stale index is recoverable, and a missing one takes the page down.

Checking it

caffeine/artifact/verify reads the published artifact back out of the store, decodes it, and compares it against a fresh compile. It exercises the encoder, the decoder and the publisher against your real content, and answers the operational question too: is what visitors are being served actually current?