Module format 

The persisted format of an HMS store: its version constants, an offline verifier and the migration framework behind hms-format.

The byte-level specification is docs/file-format/; this module is its executable form. Everything that has a version number is defined here once (FormatVersions), and tests/format_golden.rs fails when a constant moves without a matching golden store.

• scan holds the pure byte-slice readers (frames, manifests, headers). They never panic and never allocate from an untrusted length, so the fuzz target can drive them.
• verify validates a store directory without modifying it.
• upgrade is the migration registry and the steps that exist today.

Re-exports

pub use upgrade::plan_upgrade;

pub use upgrade::upgrade_store;

pub use upgrade::UpgradePlan;

pub use upgrade::UpgradeReport;

pub use upgrade::UpgradeStep;

pub use verify::verify_path;

pub use verify::VerifyOptions;

Modules

scan

Pure readers for the persisted artefacts: byte slices in, findings out. None of them panics, none allocates more than a bounded multiple of its input, and none touches the filesystem, so fuzz/fuzz_targets/format_readers.rs can drive every one with arbitrary bytes. The store-level verifier (super::verify) is built from them.

upgrade

The migration framework behind hms-format upgrade.

verify

Offline verification of a store directory. Read-only: it never opens the engine unless deep is set, and then only on a private copy, because opening a store writes to it (end marker, format.json, documents.text, the lock file).

Structs

Finding

One observation about a store.

FormatVersions

Every version number a store carries, as this build reads and writes them.

Report

Result of verifying one store directory.

StoreVersions

Versions read from a store’s files; None where the file is absent or unreadable.

TenantReport

A tenant store inside a server data directory.

Enums

Severity

How bad a finding is. The order is the order of precedence for the exit code.

Verdict

Overall verdict of a verification, and the exit code of hms-format verify.

Constants

AUDIT_ENTRY_LEN

Bytes of one audit.bin entry.

BINARY_TRANSACTION_MAGIC

Prefix of a binary journal frame payload (schema 3 stores).

CURRENT

The versions this build writes.

END_MARK

Version word of the end-of-log marker header.

FRAME_HEADER_LEN

Bytes of a journal frame header.

KDF_ITERATIONS

See KDF_MEMORY_KIB.

KDF_LANES

See KDF_MEMORY_KIB.

KDF_MEMORY_KIB

Argon2id memory cost in KiB, iteration count and lanes used to derive the AES-256 key.

MANIFEST_MAGIC

magic of format.json.

MAX_FRAME_PAYLOAD

Largest decompressed frame payload.

SALT_LEN

Bytes of encryption.salt.

STORAGE_FORMAT_CURRENT

Version of the arena manifest (vectors_data.bin/format.json) this build writes.

STORAGE_FORMAT_OLDEST

Oldest arena manifest version this build opens. Version 1 (no manifest) is read as 2.

STORE_SCHEMA_OLDEST

Oldest store.json schema this build opens.

STORE_SCHEMA_VERSION

version field of the store.json a new store gets. Schema 2 stores write TRANSACTION_MAGIC frames, schema 3 stores BINARY_TRANSACTION_MAGIC frames; a store keeps the schema it was created with.

TEXT_SEGMENT_HEADER_LEN

Length of the documents.text header.

TEXT_SEGMENT_MAGIC

First four bytes of documents.text.

TEXT_SEGMENT_VERSION

Header layout version of documents.text this build writes.

TRANSACTION_MAGIC

Prefix of a JSON journal frame payload (schema 2 stores); the digit is the journal format version.

TRANSACTION_MAGIC_PREFIX

Bytes shared by every journal format version: HMS-TXN-, then ASCII digits, then NUL.

VECTOR_LOG_MAGIC

Vector update log magic.

VECTOR_MANIFEST_MAGIC

Vector checkpoint manifest magic.

VECTOR_ROWS_MAGIC

Full-precision vector row magic.

VECTOR_UPDATE_FORMAT_VERSION

Version of the standalone persisted vector update directory.