Search & relevance

The inverted index is built in PHP at index time and ships inside the artifact, so the browser needs no search library — just a binary search over a sorted token array.

Tokenisation

Both engines must produce identical tokens for the same input. Document tokenisation happens only in PHP, at build time; query tokenisation happens in both, and is pinned by fixtures.

1. Strip HTML tags and decode entities.
2. Lowercase, Unicode-aware.
3. Decompose to NFD and remove combining marks, so Café becomes cafe.
4. Split on any run of characters that are neither letters nor numbers.
5. Drop tokens shorter than 2 or longer than 64 characters.

Step 3 is why the engines agree on accented text without shipping a transliteration table, and why scripts with no case or accents — CJK, Arabic — pass through untouched rather than mangled.

Matching

Matching is conjunctive: every token in the query must match. Only the last token is prefix-matched, because that is the one the visitor is still typing. So cordless dri matches a record containing “cordless drill”, but cord drill does not match “cordless drill”.

A token that matches nothing empties the result rather than being quietly dropped. Searching for a word that is genuinely not there should return nothing, not everything.

Weights and score

Each searchable attribute has a weight. A record’s score for a query is the sum, over each query token, of the weight of the matched document token — so a term appearing in both the title and the body outranks one appearing in the title alone.

Where a prefix matches several document tokens, the highest weight among them is used, so a half-typed word is not penalised for being ambiguous.

Weights are quantised to three decimal places when the artifact is compiled. The server renders the first paint from a freshly compiled artifact while the browser queries a decoded one, so a weight that could not survive the round trip exactly would be a place the two engines disagree about relevance order.

Stopwords

An index can declare words to ignore. They are removed from documents and from queries, and that symmetry is the whole requirement: because matching is conjunctive, removing “the” from documents alone would make a search for the saw match nothing at all.

The list ships inside the artifact rather than being read from config, because the browser cannot read project config and a list that differed between the two engines would make the same search return different results on the server and in the browser.

A query consisting entirely of stopwords is treated as an empty query — every record matches — rather than as a query that matched nothing.

Synonyms

Groups of interchangeable words, one group per line: sofa, couch, settee. Every word in a group finds the others.

They are expanded at build time only: a record containing “sofa” is indexed under every word in its group, at the same weight. Neither engine knows what a synonym is, and nothing about them appears in the artifact beyond the extra tokens. Expanding at query time instead would mean shipping the map and implementing the lookup twice, in two languages, for no behavioural difference.

Because they are applied at build time, changing them needs a rebuild.

Sorting

With no text query, every score is zero, so a named sorting is used directly from its precomputed order — filter it and take. This is the common case for a filtered listing, and it avoids sorting entirely.

With a query active, the text score sits between the sort key and the tie-break, so the order has to be recomputed — but only over the records the query already narrowed to.

Nulls sort last in both directions: a record with no value is unrankable rather than smallest, and burying it is the useful behaviour. The final tie-break is always the record’s ID, so the order is total and two builds of the same content produce the same result.

Editions

Full-text search, weights and prefix matching are in Lite. Stopwords and synonyms need Pro.