Operations reference

Running InvariantDB in production.

Deploy, monitor, back up, respond. The customer-facing shape of self-hosting. Detailed procedures ship with your production contract; this page is the surface you can read on the way to evaluating whether that fits.

If you self-host

You're the operator.

You run the binary on your own hardware or your own cloud. Everything on this page applies. Deployment, monitoring, backups, incident response — you own it, we support it.

Start with Deployment and Monitoring.

If you're on managed cloud

We're the operator.

Point-and-click provisioning, patching, backups, and failover live in the managed dashboard. Read this page to understand what we do on your behalf, or if you're evaluating whether self-host is a better fit for your regulatory posture.

Managed users start at /cloud/me/dashboard.

Contents

  1. Deployment
  2. Monitoring and alerting
  3. Backup and restore
  4. Common operations
  5. Incident playbook
  6. Escalation and support

01 — Deployment

InvariantDB ships as a single statically-linked binary. A production node is the binary plus a systemd unit plus a data directory. There is no external database, no coordinator, no sidecar. The same binary serves reads, writes, backups, and metrics.

Systemd unit template

A working starting point. Adjust the User, WorkingDirectory, and Environment lines to match your host.

# /etc/systemd/system/invariantdb.service [Unit] Description=InvariantDB engine After=network-online.target Wants=network-online.target [Service] Type=notify User=invariantdb Group=invariantdb WorkingDirectory=/var/lib/invariantdb ExecStart=/usr/local/bin/gq-server Restart=on-failure RestartSec=5 # --- required --- Environment="GQ_GRAPH_DIR=/var/lib/invariantdb/graphs" Environment="GQ_HTTP_ADDR=0.0.0.0" Environment="GQ_HTTP_PORT=3000" # --- recommended --- Environment="GQ_REQUIRE_AUTH=1" Environment="GQ_ADMIN_PORT=3199" Environment="GQ_MAX_OPEN_ENGINES=16" [Install] WantedBy=multi-user.target

Environment variables you should know

These are the operator-facing knobs. Everything else has a safe default that we tune on your behalf as part of the release notes. If you need to reach past this list, you're inside the operator handbook territory — talk to support.

VariableDefaultWhat it does
GQ_GRAPH_DIR required Absolute path to the graph data directory. Persistent state lives here. Back this up.
GQ_HTTP_ADDR 127.0.0.1 Address the HTTP listener binds to. Set to 0.0.0.0 behind a load balancer; leave loopback for single-host reverse-proxy setups.
GQ_HTTP_PORT 3000 Public HTTP port. Serves queries, writes, MCP, and /healthz + /metrics.
GQ_ADMIN_PORT 3199 Localhost-only admin listener. Bound to 127.0.0.1 unconditionally. Force-compaction, budget updates, key rotation live here.
GQ_REQUIRE_AUTH 0 When 1, the public HTTP port refuses unauthenticated requests. Recommended for every production deploy. Loopback admin port stays open regardless.
GQ_MAX_OPEN_ENGINES 16 Cap on how many graphs are held resident at once. Graphs above the cap are evicted LRU when idle. Raise this if your working set is larger than 16 graphs; the memory ceiling scales roughly linearly.
GQ_ENGINE_PIN_LIST empty Comma-separated list of graph names that stay resident regardless of eviction pressure. Warmed at startup so the first production request doesn't pay a first-touch latency cost.
GQ_METRICS_CARDINALITY_CAP 10000 Ceiling on unique label combinations emitted at /metrics. Prevents a runaway tenant from blowing up your scrape budget. Raise if you have very many small graphs; a metric-truncation event is surfaced when the cap bites.
Zero-downtime deploys. A high-availability cluster stays online across binary rollovers. On a single-node deploy, writes pause briefly during the restart and resume automatically. In-flight state is durable before the swap; nothing committed is at risk.

02 — Monitoring and alerting

Liveness

GET /healthz is the single source of truth. It returns HTTP 200 with a JSON body reporting engine state, replication topology (if enabled), and the embedder readiness block. Load balancers key on the status code; on-call keys on the body.

$ curl -s http://localhost:3000/healthz { "status": "ok", "service": "graphiquity", "version": "...", "engine": { "graph_dir": "/var/lib/invariantdb/graphs", "next_seq": 18274, "current_snapshot_seq": 18271 }, "raft": { "enabled": false }, "embedder": { "state": "ready", "default_model": "minilm-l6-v2" } }

Metrics

GET /metrics exposes a Prometheus surface. Query rates, mutation rates, per-graph memory budgets, per-graph query budgets. Standard alerting stacks light up without custom integration.

Prometheus scrape config

# prometheus.yml scrape_configs: - job_name: "invariantdb" scrape_interval: 15s scrape_timeout: 10s metrics_path: "/metrics" static_configs: - targets: ["invariantdb.internal:3000"] labels: env: "prod"

Counter families to alert on

gq_writes_total

Committed mutations, labeled by graph. Rate drop to zero on a graph you expect traffic on is a load or auth incident.

gq_cypher_total

Cypher queries served, labeled by graph and outcome. Watch the error-outcome rate for regressions after a deploy.

gq_engine_per_graph_resident_bytes

Working-set bytes per graph. Alert on graphs approaching their configured memory budget.

Alert: rate exceeds 90% of budget for 5m.

gq_id_map_delta_layers_bytes

Bytes held in the id-map delta chain. Steady growth without compaction indicates a stalled promote task — see the incident playbook.

gq_wal_replay_flushes_total

Fires once per startup replay. Rate above zero in steady state is unexpected and worth investigating.

gq_backup_last_success_timestamp

Unix timestamp of the last successful backup. Alert when age exceeds your backup RPO.

Alert: age > 24h.
What we care about most. Any non-zero rate on gq_integrity_refuse_total, gq_loopback_probe_failures_total, or gq_durability_drift_total is a page-immediately signal. These fire only when the engine has detected something it does not want to serve past.

03 — Backup and restore

Backup runs online. The writer stays available; a checkpoint captures a consistent view; a streaming archive is emitted while ordinary reads continue against the checkpointed state. Every archive ships with a cryptographic lineage manifest. Restore is offline and refuses to advance the graph if any file has drifted.

You need a scheduled backup on every production node. Ship archives off-host. Verify integrity on a schedule — the restore path re-hashes every file against the manifest, so a scheduled restore-into-a-scratch-graph is the honest end-to-end check.

Full guarantees, restore semantics, and the shape of the archive: backup + restore.

04 — Common operations

Every routine task exposes a REST endpoint on the admin port. All of the endpoints below are idempotent and safe to retry.

Rotate an API key

Create a new key, deploy it, revoke the old one. Rotation is a two-step so callers can update credentials before the old key stops working.

$ curl -s -X POST http://localhost:3199/admin/api-keys \ -H "content-type: application/json" \ -d '{"name":"acme-2026-q3"}' # → {"id": "...", "secret": "gq_...", ...} # The `secret` field appears ONCE. Save it before you close the terminal. $ curl -s -X DELETE http://localhost:3199/admin/api-keys/<old-id>

Create or drop a graph

Drop is irreversible — on-disk state is deleted. Take a backup first if you might want the data back.

$ curl -s -X POST http://localhost:3000/graphs \ -H "content-type: application/json" \ -d '{"name":"acme-primary"}' $ curl -s -X DELETE http://localhost:3000/graphs/acme-primary

Force a fresh checkpoint

The engine checkpoints automatically on a background sweep. Force one when you need a fresh point-in-time before a backup or immediately after a bulk load. Safe to call on a busy graph; overlapping requests coalesce.

$ curl -s -X POST http://localhost:3199/admin/engine/acme-primary/flush

Reclaim identity-map memory

Bulk-load workloads can grow the identity-map faster than the background task drains it. This endpoint promotes the accumulated deltas immediately, which frees memory and lowers open-time latency on the next restart.

$ curl -s -X POST http://localhost:3199/admin/graphs/acme-primary/idmap-compact # → {"promoted_bytes": N, "chain_depth_after": 0}

Rebuild derived state

Derived indexes and connectivity metadata are rebuilt from canonical sources. Safe to run against a live graph. Use it after suspected on-disk drift, or as part of a support-directed recovery.

$ curl -s -X POST http://localhost:3000/graphs/acme-primary/rebuild-derived # → {"rebuilt": ["id_map", "history_index", ...], "bytes": N}

05 — Incident playbook

Five symptoms cover the overwhelming majority of production incidents. Each maps to a likely cause and a first-response action. The operator handbook shipped with your production contract expands every one of these into a full decision tree.

Symptom

First request after startup is slow.

Likely cause First-touch open on a large graph. The engine is loading a checkpoint, verifying integrity, and rebuilding in-memory indexes.
Mitigation Add the graph to GQ_ENGINE_PIN_LIST. Startup warms every pinned graph before the accept queue opens, so the first request finds the graph already resident. First-touch latency is otherwise proportional to graph size.
Symptom

Memory keeps climbing.

Likely cause Working set exceeds the configured cap. The engine holds up to GQ_MAX_OPEN_ENGINES graphs resident; graphs above the cap are evicted LRU when idle, but a busy fleet may hold them all.
Mitigation Watch gq_engine_per_graph_resident_bytes{graph} for the outliers and either lower GQ_MAX_OPEN_ENGINES, tune per-graph budgets, or size the host up. Set a hard memory budget with POST /admin/graphs/<name>/budget.
Symptom

Startup takes longer than usual.

Likely cause Durability replay. The engine is re-applying committed writes that hadn't reached the last snapshot before the previous shutdown. Time scales with the amount of un-checkpointed work.
Mitigation Nothing to do — startup will complete. Watch gq_wal_replay_flushes_total tick. If startup exceeds your budget, force a compaction after startup and schedule regular compactions to keep replay bounded.
Symptom

Query returns an integrity refuse error.

Likely cause On-disk drift detected. A file's hash didn't match the lineage manifest. The engine refuses rather than serving suspect data.
Mitigation This is a page-immediately signal. Do not force it past the refuse. Contact support with the graph name and the error subcode. Recovery is restore from the last verified backup, and the manifest tells us exactly which file drifted.
Symptom

Vector index query returns a typed error.

Likely cause Model dim mismatch. The vector index was built with a different embedding model than the query is using, or the model catalog has drifted from what the index was stamped with.
Mitigation Drop and rebuild the vector index with the correct metric: CALL db.dropVectorIndex(label, prop) followed by CALL db.buildVectorIndex(label, prop). The dim-lock guarantees this class of mismatch surfaces as a typed error rather than silent corruption. Model selection is governed at the graph level, not per index.

06 — Escalation and support

When to file a ticket

File immediately on any of: integrity refuse, durability drift, loopback probe failures, or any metric family that starts emitting values you don't recognize. File within the business day for: memory or latency regressions, backup verification failures, unexpected metric-truncation events.

The observability surface is designed so the same information you page on is what we need to triage — there is rarely a hidden log we need to ask you to fetch. Structured events at /healthz and counters at /metrics are the escalation package.

What to include

  • Graph name and tenant identifier
  • Timestamp window (UTC)
  • The full /healthz JSON body at the time of the incident
  • Relevant counter deltas from /metrics across the window
  • Any error subcodes returned to callers

Support email

support@invariantdb.com

Response targets

P0: 30 min. P1: 4 hours. P2: 1 business day. Contractual SLA figures are set in the production agreement.

Handbook

Ships with your production contract.

Talk to us before you scale.

Every production customer gets a shared runbook, a paging channel, and a scheduled monthly review of counter trends. Set that up before your first launch, not during your first incident.

Contact sales Quickstart