Ops - what to alert on, how to fail over, how to recover.

Everything below is what we run against managed instances. Most of it is automatic - the engine self-heals on common failure modes - but when something needs hands on it, this is the playbook.

Health checks

Per-tenant usage signal

GET /v1/tenants/:t/usage returns the current snapshot: row count per schema, vector count per table, in-flight Ask queries, subscription status. Poll it from your monitoring system; the dashboard renders the same data live.

A first-class Prometheus exporter is on the roadmap. Until then, hit /usage on a poll interval and graph the deltas.

Write durability

A write is acknowledged only after it has been flushed to durable storage on the instance. The flush has to return before the response is sent, not after - so a 2xx means your data is on disk rather than sitting in memory, and it survives the database process being killed. This holds for row writes, SQL data changes, Cypher writes, transaction commits, vector writes and sequence values.

Secondary index structures are flushed on a short timer instead. Full-text index writes, materialized-view installs and refreshes, and geospatial index updates are written to durable storage but are not individually flushed before the response - they ride a periodic flush. A crash can therefore lose the last few of them even though the response said 2xx. Your underlying rows are unaffected: re-running the index or refresh operation rebuilds these from row data, which is why they are treated as derived state rather than a source of record.

Two recovery caveats are worth knowing, because both are about getting data back after the flush rather than about the flush itself. We publish them because they change what you should do operationally.

• Abrupt power loss - a hard reset can interrupt a write mid-flight and leave a partially-written record behind. Your acknowledged data is still on disk, but the instance does not always reopen unattended: the incomplete record has to be cleared first. In our power-cut testing this occurred in 2 of 6 hard resets, and after trimming every acknowledged write was present - we measured no data loss. Plan for this as an availability event that can need an assisted restart, not as data loss.
• A full storage volume - while the volume is full, writes fail with a 5xx. Nothing is silently dropped and that part is safe. Once space is reclaimed the instance starts accepting writes again and serves them on read, but those writes are not reliably durable until the instance has been restarted and checked - in our testing they did not survive the next restart. Treat a full-volume alert as an incident: reclaim space, then restart and verify before you trust writes made in that window. Alert on free space well before it runs out.

A fix for both is complete in the engine and has not yet reached running instances. This page describes the build your instance is running today, and we will narrow it once the fix has rolled out. Separately, durability on the instance is not the same as durability across a failover - replication to the standby is asynchronous, see failover below.

Backups

Three layers, all archived in the same region as the tenant.

• Daily storage snapshot - always on, and the floor for every configuration. A crash-consistent snapshot of the instance's storage lands in an encrypted vault once a day, so worst-case exposure without the layers below is the time since the last snapshot.
• Off-box archive shipping - opt-in, preview, and not currently something to plan around. Recent changes ship to the archive on a size-and-time trigger rather than per commit. In practice the recoverable point has been advancing far more slowly than that design implies - measured across the fleet it sits around half an hour behind, and on some instances the archive has not advanced at all. We are fixing it. Until we say otherwise, size your recovery expectations from the daily snapshot above, not from this layer.
• Base snapshot shipping - a fresh base is shipped to the archive every 30 minutes. Restore picks the latest base at-or-before the target and replays the archive up to it.
• Restore-to-timestamp - choose a wall-clock target (RFC 3339) from the console or API and OriginChain rebuilds a fresh instance from the chosen snapshot + archive. The result opens cleanly through the same validating loaders that serve live traffic.

Recovery points are retained 30 days; manual purges on data-subject requests are documented in the runbook.

Continuous backup streaming

The default flow gives PITR granularity at the archive cadence - fine for most tenants, too coarse for compliance-heavy ones. The continuous backup stream is designed to tighten it: the tail ships on a timer whenever enough has changed since the last ship. It does not make the archive per-commit - below the change threshold nothing ships until the next base, so a quiet instance has a coarse recovery point by design. It is also not currently delivering that design: on the instances we measured, the tail has stopped advancing and the recoverable point sits roughly half an hour behind, sometimes further. Treat sub-second or per-second recovery as something we have not delivered rather than something you can switch on, and talk to us before you build a compliance commitment on this layer.

On the managed service the continuous backup stream runs as a built-in task alongside restore-side replay (auto-replay of the latest tail at-or-before the target).

Failover

When the primary is wedged or its host goes silent, OriginChain promotes the in-region standby automatically. The automatic path takes about a minute end to end - the old primary's claim has to expire and a grace window has to pass before the standby is eligible. An operator-triggered promotion is closer to 25 seconds, drilled end-to-end as of 2026-04-30. Replication to the standby is asynchronous, so a promotion after an abrupt primary loss can cost the most recent acknowledged writes.

What happens during a failover:

1. Fence the old primary - the previous primary is fenced so it can never accept writes again, even if it comes back later.
2. Promote the standby - the in-region standby takes over as the new primary and begins serving reads and writes.
3. Health-check - the new primary is verified healthy on /health before traffic is cut over.
4. Update DNS - your endpoint is re-pointed at the new primary with a 60s TTL; propagation is typically sub-minute.

Failover is initiated on a confirmed dead primary - a genuine host failure, not a transient network blip. Your application keeps the same endpoint; once DNS propagates, writes resume against the promoted instance.

When failover is held off: the primary is responding but slow (we load-shed rather than fail over); the replica is far behind on replication (failing over would lose data - replication is investigated first); or an online schema migration is mid-backfill.

Schema migrations

Online migrations are first-class - no read-only window, no service bounce. The model:

• Rate-limited backfill - the backfill is capped at a fraction of live throughput so production traffic stays prioritised.
• Dual-read transform - readers see the new (v1) shape during backfill via an on-the-fly transform applied to old (v0) rows.
• Atomic cutover - the version bump is a single commit; reads switch to the v1-native shape on the next read.
• Abort before cutover - once cutover lands, the only path forward is an inverse-rewrite migration. Aborts during backfill are safe and reversible.

Use online migrations any time the manifest version bumps. Troubleshooting a migration stuck mid-backfill is in incident response.

Observability

• EXPLAIN - prefix any SELECT with EXPLAIN to return the plan tree without running the query. Useful for verifying that your indexes are being used. See SQL reference → EXPLAIN.
• Per-tenant /usage - GET /v1/tenants/:t/usage returns row counts, vector counts, in-flight queries, and subscription state. Poll it for monitoring.
• OTLP push tracing - opt-in. Configure the collector endpoint via the dashboard; spans are pushed for every /v1 request with tail-based sampling that retains slow + error traces in full.
• Dashboard - live metrics tiles for every instance: query latency, replication lag, recent writes, error rate. Visit app.originchain.ai.

Incident response

Our internal incident playbook covers the full procedure. Highlights:

• Pager severity: sev 1 (tenant down) → 5 min ack, 1 hr fix-or-mitigate. Sev 2 (one alarm tripped) → 15 min / 4 hr. Sev 3 (drift) → next business day.
• Status page: publishes per-region health and incident timelines. Subscribers get email + webhook on any sev 1.
• Migration stuck mid-backfill: abort the migration via POST /v1/tenants/:t/migrations/:id/abort, then resubmit.
• Order of operations: stop the bleeding, find root cause, write a postmortem, fix the underlying problem. Quiet incidents become loud ones.

Incidents today are handled by core engineering during extended business hours with best-effort overnight coverage; the pager-severity SLAs above are the targets we hold ourselves to. 24/7 named-engineer coverage is available on Enterprise - contact sales.

Compliance posture

• SOC 2 Type 1: underway with Vanta/Drata and an external CPA. Contact for audit timeline; the in-flight gap analysis is available to procurement under NDA.
• HIPAA BAA: available on Enterprise. PHI workloads must run in a region the BAA covers and on a dedicated-capacity instance.
• GDPR DPA: available on Enterprise. EU-region instances support the DPA out of the box; data-subject deletion follows our documented runbook.