Ops - what to alert on, how to fail over, how to recover.

Everything below is what we run against managed instances. Most of it is automatic - the engine self-heals on common failure modes - but when something needs hands on it, this is the playbook.

Health checks

Per-tenant usage signal

GET /v1/tenants/:t/usage returns the current snapshot: row count per schema, vector count per table, in-flight Ask queries, subscription status. Poll it from your monitoring system; the dashboard renders the same data live.

A first-class Prometheus exporter is on the roadmap. Until then, hit /usage on a poll interval and graph the deltas.

Backups

Three layers, all archived in the same region as the tenant.

• Continuous archive shipping - default. Recent changes ship to the archive automatically. Worst-case data loss is the most recent unshipped tail, typically minutes-to-hours on idle tenants.
• Snapshot shipping - hourly auto-compacted snapshots. Restore picks the latest snapshot at-or-before the target and replays the archive up to it.
• Restore-to-timestamp - choose a wall-clock target (RFC 3339) from the console or API and OriginChain rebuilds a fresh instance from the chosen snapshot + archive. The result opens cleanly through the same validating loaders that serve live traffic.

Recovery points are retained 30 days; manual purges on data-subject requests are documented in the runbook.

Continuous backup streaming

The default flow gives PITR granularity at the archive cadence - fine for most tenants, too coarse for compliance-heavy ones. The continuous backup stream ships recent changes every few hundred milliseconds (default 500 ms), driving worst-case data loss to ~0.5–1.5 s.

On the managed service the continuous backup stream runs as a built-in task alongside restore-side replay (auto-replay of the latest tail at-or-before the target).

Failover

When the primary is wedged or its host goes silent, OriginChain promotes the in-region sync replica automatically. RTO is ~25s, drilled end-to-end as of 2026-04-30.

What happens during a failover:

1. Fence the old primary - the previous primary is fenced so it can never accept writes again, even if it comes back later.
2. Promote the replica - the in-region sync replica takes over as the new primary and begins serving reads and writes.
3. Health-check - the new primary is verified healthy on /health before traffic is cut over.
4. Update DNS - your endpoint is re-pointed at the new primary with a 60s TTL; propagation is typically sub-minute.

Failover is initiated on a confirmed dead primary - a genuine host failure, not a transient network blip. Your application keeps the same endpoint; once DNS propagates, writes resume against the promoted instance.

When failover is held off: the primary is responding but slow (we load-shed rather than fail over); the replica is far behind on replication (failing over would lose data - replication is investigated first); or an online schema migration is mid-backfill.

Schema migrations

Online migrations are first-class - no read-only window, no service bounce. The model:

• Rate-limited backfill - the backfill is capped at a fraction of live throughput so production traffic stays prioritised.
• Dual-read transform - readers see the new (v1) shape during backfill via an on-the-fly transform applied to old (v0) rows.
• Atomic cutover - the version bump is a single commit; reads switch to the v1-native shape on the next read.
• Abort before cutover - once cutover lands, the only path forward is an inverse-rewrite migration. Aborts during backfill are safe and reversible.

Use online migrations any time the manifest version bumps. Troubleshooting a migration stuck mid-backfill is in incident response.

Observability

• EXPLAIN - prefix any SELECT with EXPLAIN to return the plan tree without running the query. Useful for verifying that your indexes are being used. See SQL reference → EXPLAIN.
• Per-tenant /usage - GET /v1/tenants/:t/usage returns row counts, vector counts, in-flight queries, and subscription state. Poll it for monitoring.
• OTLP push tracing - opt-in. Configure the collector endpoint via the dashboard; spans are pushed for every /v1 request with tail-based sampling that retains slow + error traces in full.
• Dashboard - live metrics tiles for every instance: query latency, replication lag, recent writes, error rate. Visit app.originchain.ai.

Incident response

Our internal incident playbook covers the full procedure. Highlights:

• Pager severity: sev 1 (tenant down) → 5 min ack, 1 hr fix-or-mitigate. Sev 2 (one alarm tripped) → 15 min / 4 hr. Sev 3 (drift) → next business day.
• Status page: publishes per-region health and incident timelines. Subscribers get email + webhook on any sev 1.
• Migration stuck mid-backfill: abort the migration via POST /v1/tenants/:t/migrations/:id/abort, then resubmit.
• Order of operations: stop the bleeding, find root cause, write a postmortem, fix the underlying problem. Quiet incidents become loud ones.

Incidents today are handled by core engineering during extended business hours with best-effort overnight coverage; the pager-severity SLAs above are the targets we hold ourselves to. 24/7 named-engineer coverage is available on Enterprise - contact sales.

Compliance posture

• SOC 2 Type 1: underway with Vanta/Drata and an external CPA. Contact for audit timeline; the in-flight gap analysis is available to procurement under NDA.
• HIPAA BAA: available on Enterprise. PHI workloads must run in a region the BAA covers and on a dedicated-capacity instance.
• GDPR DPA: available on Enterprise. EU-region instances support the DPA out of the box; data-subject deletion follows our documented runbook.