This runbook covers the production orchestrator surface exposed by core/orchestrator/orchestrator.ts.
GET /api/health: JSON readiness. Local loopback callers receive full diagnostics; public callers receive redacted status.GET /api/metrics: Prometheus text metrics derived from the same health object, with the same redaction. Per-child, hub restart, tracked-storage, xln_disk_free_bytes and external-client metrics are operator-only: scrape them from loopback on the host or with the daemon bearer.Quick checks:
curl -fsS https://xln.finance/api/health | jq '{coreOk, systemOk, degraded, disk, storage, hubMesh, marketMaker, custody}'
curl -fsS https://xln.finance/api/metrics | grep -E 'xln_(core_ok|system_ok|disk_used_pct|process_rss_bytes|hub_online)'
bun run prod:health
For remote operator health, supply the existing node capability through an owner-only file:
bun run prod:health --operator-token-file=/absolute/private/operator-token
XLN_PROD_HEALTH_OPERATOR_TOKEN_FILE supplies the same option for release/preflight scripts.
The client reads the file without creating or rotating credentials. Obtain it through the existing
operator access procedure; never put its contents in command arguments, logs or evidence.
Only API incidents, health and metrics receive the capability. Authenticated requests require HTTPS
(or loopback HTTP) and reject redirects; separate tower URLs and the wallet request receive no token.
A missing/wrong capability remains an authorization failure, not permission to skip incident checks.
Start xln-server with PM2 --no-treekill --kill-timeout 60000, as the canonical
platform deploy does. The orchestrator owns child quiesce and termination; PM2’s
default tree signal races that sequence and its default1.6s grace preempts it.
The current normal shutdown hierarchy requires up to50.15s. Existing parent-liveness
watches still close managed children when their exact parent disappears.
Actual normal and parent-crash drills record supervision before/after, old process identities, current-WAL financial-state comparisons and unchanged unrelated owners. Preserve the current stores on restart.
package.json owns the exact Bun version. Startup and deployment reject a different
version before opening stores, creating keys, killing listeners or deleting fresh
state. Configure each XLN service’s PATH with the verified app-scoped Bun directory
first; server, custody and watchtower execute the selected checked binary. Do not
replace another application’s global runtime to satisfy this requirement.
Record archive/binary hashes and verify the selected executable with bun --version
in the actual service environment. Preserve its current checkpoint/WAL before a
controlled restart. A successful short crypto/worker test does not prove a previously
observed shutdown crash fixed; capture old/new process ownership, exit logs and exact
financial state after the actual restart. See
pinned runtime and recovery evidence.
The actual public pinned-runtime deployment
records executable provenance, current-store backups, bounded restart, exact financial
comparisons and HTTPS verification. Its retained shutdown warnings and release limits
are part of the result.
For an already-authorized testnet update, preserve the previous generated web roots
with scripts/deployment/web-artifacts.sh pack <absolute-archive> and record the
archive digest, source SHA, live process identities and current financial snapshots.
From the clean, checked and published local main, use the canonical
deploy-platform.sh --remote <configured-host> --frontend-only --production path.
It guards the remote fast-forward and installs both generated roots without restarting
the backend. A new checkout SHA alone does not prove the running backend changed.
If the update also requires an existing backend owner to reload, compare actual
storage/financial compatibility first, retain its current stores, and use the exact
parent-owned PM2 procedure above under the stand lock. Do not use --update-env to
propagate the stand runner’s environment into the service. Require exact historical
Account/Runtime state plus current financial comparisons after recovery. Explain any
new committed frames from their actual inputs and hash links; never waive a mismatch.
For this compatible web-only rollback, the same web-artifacts.sh install command
can reinstall the preserved archive. Verify every installed file and the public HTML,
JavaScript, runtime and worker digests; exercise the real wallet and relay before
returning to the current artifact. Preserve process owners and stores throughout.
Actual update and web round-trip evidence
records the exact candidate, first failed comparison and its complete explanation.
This is not a backend rollback procedure or authorization for mainnet activation.
The owner-authorized October17 recovery is recorded in public testnet evidence. Its archived commands identify the exact namespaces and preservation guards; do not reuse a reset list against a different host without resolving its current owners.
scripts/deployment/prepare-remote-main.sh for a guarded fast-forward.
Delete only confirmed obsolete testnet stores; preserve current credentials,
job evidence and the sibling debug-incident journal. Install through
scripts/deployment/web-artifacts.sh and verify every installed file hash.prod:health with three hubs/self-relay and one tower, then
actual public HTTPS wallet/runtime/worker verification. Preserve all failures.These steps regenerate obsolete testnet data. They are not the current-format checkpoint/WAL recovery drill or a rollback proof, and do not authorize a mainnet reset.
xln_core_ok == 0 for 2 minutes.xln_system_ok == 0 for 5 minutes, unless a planned deploy is active.xln_disk_free_bytes < 10GiB or xln_storage_ok == 0.xln_child_online{name=...} == 0 for 2 minutes.xln_hub_mesh_ok == 0 for 5 minutes.xln_process_rss_bytes grows by more than 25% in 30 minutes.xln_storage_scan_truncated == 1; increase scan budget before trusting tracked byte deltas.Snapshot current health and metrics:
curl -fsS https://xln.finance/api/health > /tmp/xln-health.json
curl -fsS https://xln.finance/api/metrics > /tmp/xln-metrics.prom
Check child process state:
jq '.process.children[] | {role, name, online, pid, restartCount, lastErrorLine}' /tmp/xln-health.json
pm2 status
pm2 logs --lines 200
Check storage pressure:
bun run debug:disk
jq '.disk, .storage.tracked[] | {name, kind, currentBytes, bytesPerHour, scanTruncated}' /tmp/xln-health.json
df -h /
du -sh logs .logs playwright-report test-results db 2>/dev/null || true
Test runners clean old generated artifacts by default before new runs.
Use XLN_KEEP_TEST_ARTIFACTS=1 only when preserving traces/videos/logs is
intentional. Generated test artifacts are budgeted at 50 GiB by default via
XLN_TEST_WORKSPACE_MAX_BYTES.
If a hub is down but the orchestrator is healthy, prefer an orchestrator-managed restart over killing child PIDs manually:
pm2 restart xln
sleep 10
curl -fsS https://xln.finance/api/health | jq '{coreOk, systemOk, degraded, hubs}'
*-wal Runtime WAL is authoritative for replay. The *-history-views
database is a rebuildable inspection/UI index, never a commit authority.storage.epochMaxBytes is the byte trigger for a full storage epoch rotation.
The normal runtime default is 16 GiB; small values belong only in explicit
rotation tests and smoke drills. The Runtime writes a full snapshot to the
WAL, prunes replay diffs covered by the retained snapshot, seeds a fresh
*-storage-current DB from live rows plus Merkle rows, and leaves the
replaced DB at *-storage-previous.*-wal as the recovery source of truth. Preserve both *-wal and
*-history-views during incident investigation; the latter can be rebuilt,
but contains useful evidence about materialization progress.For high-load hub drills, use the rotation benchmarks:
bun run bench:radapter:hub10k:rotation
bun run bench:radapter:hub1m:rotation
The 1M drill should show rotationProbe.snapshotDocs == rotationProbe.liveDocs, rotationProbe.nextRetainedHistoryBytes == 0, read p99 under the script cap, durable-write p99 under the script cap, and peak RSS under the configured cap.
If health reports storage verification failure, keep the DB directories intact and collect:
tar -czf /tmp/xln-storage-incident.tgz db data/storage-health-history.json logs .logs 2>/dev/null || true
db/ or .logs/e2e-* during incident capture unless the release owner explicitly approves it.Before declaring a deploy healthy:
bun run gate:ci
bun run soundcheck runtime frontend/src package.json .github/workflows
bun run test:e2e:fast
bun run test:rpc-settlement
bun run security:audit-pack
bun run prod:health
Use bun run deploy:prod for the canonical production rollout. It always resets
the public testnet chain, runtime, mesh, and custody state before bootstrapping.
Hub and market-maker WAL is not a deployment input. bun run deploy:prod:fresh
is an explicit alias for the same clean deployment policy.
Before a public-net release candidate, also run:
bun run gate:release
bun run soak:release
For the public-testnet launch envelope in the current AGENTS.md scope,
with one tower and three hubs, use the existing operator gate below. The owner
removed the global monetary cap on 2026-10-11; these historical command names
do not restrict participation or imply guaranteed unsecured recovery.
export XLN_CAPPED_TESTNET_TOWER_URLS=https://tower.example.com
bun run gate:capped-testnet:preflight
bun run gate:capped-testnet
gate:capped-testnet writes a JSON report under .logs/gates/ and runs the
one-hour capped soak. If any step fails, fix the root cause, commit the fix,
and start the capped gate again from a fresh release-candidate commit.