Operating hms-server¶

Reference for running the server in production. Settings are in configuration.md; the API is openapi.json.

Deployment artifacts¶

One replica per data directory: a tenant store is owned by a single process through an exclusive file lock. Scale a deployment by giving the pod more CPU and memory, or by running several independent servers for disjoint tenant sets.

TLS and sockets¶

--tls-cert/--tls-key serve HTTPS with rustls (TLS 1.2 and 1.3, ALPN h2 and http/1.1). Replace the files and send SIGHUP or POST /api/v1/admin/tls/reload; a bad pair is rejected and the running certificate stays in use. --unix-socket PATH serves on a socket with mode 0600 instead of TCP (a stale socket file is replaced, a live one or a non-socket file is refused). Without TLS on a non-loopback address the server logs a plaintext_on_public_address warning at startup.

Health, readiness and shutdown¶

• GET /healthz: the process is serving. Use as the liveness probe.
• GET /readyz: 200 once the listener is bound; 503 not_ready from the moment SIGTERM or SIGINT arrives. Use as the readiness probe.
• Shutdown order: readiness fails, --shutdown-delay-seconds pass with the server still answering (so a balancer can notice), the listener closes, open connections get --shutdown-grace-seconds, admitted work finishes, every store is flushed, the process exits 0. Set the orchestrator's termination grace to at least delay + grace + the time a flush takes (the Helm chart defaults to 60 s for 5 + 30).

Metrics¶

GET /metrics (admin scope, or public with --metrics-public) is Prometheus text format 0.0.4:

Per-tenant series are bounded by --max-tenants (only open stores report).

Logs¶

One JSON object per line on stderr (--log-format text for key=value). Every request logs method, route template, status, latency_ms, tenant, key_id, bytes_in and request_id; X-Request-Id from the client is honored when it is 1 to 64 characters of [A-Za-z0-9._-], otherwise a fresh id is generated, and the id appears in the response header and in error bodies. Request bodies, vectors, query text, metadata, document ids and credentials are never written. Lifecycle events (listening, shutdown_started, shutdown_complete, tenant_evicted, snapshot_created, snapshot_restored, compaction_done, keys_reloaded, tls_reloaded) and failures carry identifiers and counts only. A test checks that secrets and content sent through the API never appear in the log.

Backup and restore¶

• POST /api/v1/snapshots flushes the tenant's store and copies it to <data-dir>/snapshots/<tenant>/<id>/. The tenant pauses (requests wait) while the copy runs; other tenants are unaffected. Expect roughly the store's size in extra disk per snapshot; an empty store is about 64 MiB because the engine preallocates its first arena segment.
• GET /api/v1/snapshots/{id}/archive streams the snapshot as a tar. This is the off-box backup: store it anywhere.
• POST /api/v1/snapshots/{id}/restore replaces the tenant's store with the snapshot; PUT /api/v1/restore does the same from an uploaded tar (to the same server or a different one). Both verify the candidate store (checksums, manifests and a full open that authenticates every frame) before touching the live one, refuse archives with links, device nodes or paths outside the store, and, if anything fails after the swap starts, put the previous store back. The replaced store is kept as snapshot pre-restore-<time>; delete it when you no longer need it.
• Snapshots on the same disk protect against mistakes, not against disk loss. Copy the archive off the machine for that.

Compaction¶

--compaction-interval-seconds N compacts every open store that changed since its last compaction, one at a time. GET /api/v1/compaction shows the schedule and the last run (runs, failures, last_trigger, last_started_at, last_duration_ms); POST /api/v1/compaction runs it now. Compaction rewrites the store's logs without the superseded and deleted entries. It runs on a blocking worker outside the request slots; the engine decides how long it holds writers back, so schedule it for quiet periods on large stores.

Capacity planning¶

• Memory is dominated by open stores. --tenant-idle-seconds closes idle ones; hms_tenant_index_bytes and process_resident_memory_bytes show the effect.
• --max-concurrent-work is the number of requests the engine runs at once; ingest and search are CPU-bound, so set it near the number of cores and use --max-queue for bursts.
• Rate limits and quotas are per tenant and off by default; enable them when tenants are mutually untrusted.

Measured behavior¶

All numbers come from benchmarks/results/server_v1.json, server_soak_v1.json and the container checks below. The machine was a shared 10-core M4 at load average 25 to 40, with client and servers on one host, so absolute figures are indicative. Synthetic clustered vectors, 384 dimensions, top 10, 6 s per level, one run.

Query throughput (QPS at 1 / 4 / 16 clients, and p50 at 16)

Recall at 10 was 1.0 for every system and size. hms-server wins at 1,000 documents and loses at 10,000 and 50,000: its vector query is an exact scan inside the engine (about 6.9 of 7.2 ms at 10,000 documents is spent in the engine search), slower than a numpy scan from 10,000 up, and the optional document index (/index/maintain) did not help in these runs. This is an engine cost, not server overhead, and it is the largest open performance item. Ingest was 2.1k to 2.9k documents/s for hms-server against 3.1k to 4.7k for the others (batches of 500, durable commit per batch).

Soak (4 minutes, 16 clients, 137,634 successful mixed requests, tenants rotating through 12 ids with at most 6 open): no errors other than 27 expected tenant_limit refusals, open file descriptors 26 to 35, work slots idle at the end, every tenant's contents matched the client's model both before and after a SIGTERM restart, SIGTERM exit code 0. RSS moved between 55 and 270 MB following tenant open and eviction cycles and ended at 144 MB from 139 MB at the start; the second-half linear fit reads +4.4 MB/min over that oscillation, so a longer run is needed to call it flat.

Container (Lima VM, Docker 29, arm64): image 15,015,731 bytes (15 MB, of which 5.84 MB is the server binary); uid 65532, read-only root, no capabilities; RSS 10.0 MB (cgroup 13.5 MiB) after loading 2,000 documents of 16 dimensions; docker stop took 0.19 s and exited 0; data survived a restart on the volume; 118 MiB at 10,000 documents of 384 dimensions. Compose and the systemd unit were run for real (systemd-analyze security: exposure 1.2 OK). Not measured: x86_64 images, musl against glibc allocator throughput, a live Kubernetes cluster (no kind available; helm lint and kubeconform -strict pass).