Positional lexical postings¶

The ordinary term-frequency map and non-positional HMS-TXN-2/HMS-TXN-3 records retain their existing representation. Positions are an opt-in chunk extension, enabled by IngestOptions::positions. They are not reconstructed from term frequencies when reading an older store. Phrase queries require positions on every live chunk; otherwise the lexical API returns an explicit unsupported-index error. Ordinary text/hybrid queries continue to work.

In a binary document record, chunk flag bit 6 indicates a positional payload, following the term frequencies:

byte(version) varint(offset_count) varint(offset){offset_count} bytes(data)

Version 1 represents ordinary positional chunks. Version 2 represents a whole lexical article, selected by IngestOptions::whole_document. Both use the same position encoding. A whole article requires one chunk, lexical encoding, no embedding, byte offset zero and at most 8 MiB. Ordinary chunks retain their 64 KiB limit. Unknown versions are refused as newer than supported. Readers predating this extension reject the previously reserved bit rather than reading the payload as another record.

The offsets number one more than the term count, start at zero and end at the position byte length. Each term's byte range contains unsigned LEB128 token position gaps. The first gap is an absolute zero-based position and may be zero; subsequent gaps are positive. Terms follow the document record's term order. Positions use the same analyzer as frequencies, before title-frequency weighting. Each token position occurs once across all terms, and the positions cover the complete token sequence without holes. Readers check every offset, length, integer overflow, term frequency and allocation bound before accepting a record.

JSON records place the positional record alongside the legacy frequency entries:

{"$hms_positions":{"version":1,"offsets":[0,1,3],"data":[2,0,1]},"fox":1,"red":2}

This encodes red red fox. The reserved key cannot be emitted by the alphanumeric analyzer. It is absent from non-positional records. Chunk text may be discarded; positions persist in both journal forms, through replay and compaction. The golden codec fixtures in tests/golden/lexical-postings-v1/ are regenerated byte-for-byte by unit tests, including old records and every truncated prefix.

The 128-entry bitpacked posting blocks, their skip endpoints, maximum frequencies and conservative minimum lengths are derived memory state. Journal replay rebuilds them from live term tables. No old posting bytes are reinterpreted as blocks and no persisted block cache is added by this change.