xln

XLN Ops Runbook

This runbook covers the production orchestrator surface exposed by core/orchestrator/orchestrator.ts.

Health Endpoints

Quick checks:

curl -fsS https://xln.finance/api/health | jq '{coreOk, systemOk, degraded, disk, storage, hubMesh, marketMaker, custody}'
curl -fsS https://xln.finance/api/metrics | grep -E 'xln_(core_ok|system_ok|disk_used_pct|process_rss_bytes|hub_online)'
bun run prod:health

For remote operator health, supply the existing node capability through an owner-only file:

bun run prod:health --operator-token-file=/absolute/private/operator-token

XLN_PROD_HEALTH_OPERATOR_TOKEN_FILE supplies the same option for release/preflight scripts. The client reads the file without creating or rotating credentials. Obtain it through the existing operator access procedure; never put its contents in command arguments, logs or evidence. Only API incidents, health and metrics receive the capability. Authenticated requests require HTTPS (or loopback HTTP) and reject redirects; separate tower URLs and the wallet request receive no token. A missing/wrong capability remains an authorization failure, not permission to skip incident checks.

Orchestrator shutdown ownership

Start xln-server with PM2 --no-treekill --kill-timeout 60000, as the canonical platform deploy does. The orchestrator owns child quiesce and termination; PM2’s default tree signal races that sequence and its default1.6s grace preempts it. The current normal shutdown hierarchy requires up to50.15s. Existing parent-liveness watches still close managed children when their exact parent disappears.

Actual normal and parent-crash drills record supervision before/after, old process identities, current-WAL financial-state comparisons and unchanged unrelated owners. Preserve the current stores on restart.

Pinned runtime executable

package.json owns the exact Bun version. Startup and deployment reject a different version before opening stores, creating keys, killing listeners or deleting fresh state. Configure each XLN service’s PATH with the verified app-scoped Bun directory first; server, custody and watchtower execute the selected checked binary. Do not replace another application’s global runtime to satisfy this requirement.

Record archive/binary hashes and verify the selected executable with bun --version in the actual service environment. Preserve its current checkpoint/WAL before a controlled restart. A successful short crypto/worker test does not prove a previously observed shutdown crash fixed; capture old/new process ownership, exit logs and exact financial state after the actual restart. See pinned runtime and recovery evidence. The actual public pinned-runtime deployment records executable provenance, current-store backups, bounded restart, exact financial comparisons and HTTPS verification. Its retained shutdown warnings and release limits are part of the result.

Updating current testnet web artifacts without a reset

For an already-authorized testnet update, preserve the previous generated web roots with scripts/deployment/web-artifacts.sh pack <absolute-archive> and record the archive digest, source SHA, live process identities and current financial snapshots. From the clean, checked and published local main, use the canonical deploy-platform.sh --remote <configured-host> --frontend-only --production path. It guards the remote fast-forward and installs both generated roots without restarting the backend. A new checkout SHA alone does not prove the running backend changed.

If the update also requires an existing backend owner to reload, compare actual storage/financial compatibility first, retain its current stores, and use the exact parent-owned PM2 procedure above under the stand lock. Do not use --update-env to propagate the stand runner’s environment into the service. Require exact historical Account/Runtime state plus current financial comparisons after recovery. Explain any new committed frames from their actual inputs and hash links; never waive a mismatch.

For this compatible web-only rollback, the same web-artifacts.sh install command can reinstall the preserved archive. Verify every installed file and the public HTML, JavaScript, runtime and worker digests; exercise the real wallet and relay before returning to the current artifact. Preserve process owners and stores throughout. Actual update and web round-trip evidence records the exact candidate, first failed comparison and its complete explanation. This is not a backend rollback procedure or authorization for mainnet activation.

Regenerating obsolete testnet stores

The owner-authorized October17 recovery is recorded in public testnet evidence. Its archived commands identify the exact namespaces and preservation guards; do not reuse a reset list against a different host without resolving its current owners.

  1. Build and hash the current canonical web artifact before stopping services.
  2. Record exact PM2 owners, protected seed/key hashes and modes, unrelated process identities, and current source. Stop only the owning XLN services; verify zero open handles and reject symlinked/unknown reset roots.
  3. Use scripts/deployment/prepare-remote-main.sh for a guarded fast-forward. Delete only confirmed obsolete testnet stores; preserve current credentials, job evidence and the sibling debug-incident journal. Install through scripts/deployment/web-artifacts.sh and verify every installed file hash.
  4. Start existing canonical chain/runtime owners. Observe their bootstrap through bounded checks without restarting a still-progressing managed service. Preserve timeout results separately from later readiness observations.
  5. Snapshot incidents and health before resolving any historical incident. Require its last occurrence to predate the replacement deployment and concrete evidence its condition is gone. Resolve individually through the authenticated operator API; never clear the journal to satisfy release health. Recurrence reopens it.
  6. Run exact-source prod:health with three hubs/self-relay and one tower, then actual public HTTPS wallet/runtime/worker verification. Preserve all failures.

These steps regenerate obsolete testnet data. They are not the current-format checkpoint/WAL recovery drill or a rollback proof, and do not authorize a mainnet reset.

Alert Rules

Triage

  1. Snapshot current health and metrics:

    curl -fsS https://xln.finance/api/health > /tmp/xln-health.json
    curl -fsS https://xln.finance/api/metrics > /tmp/xln-metrics.prom
    
  2. Check child process state:

    jq '.process.children[] | {role, name, online, pid, restartCount, lastErrorLine}' /tmp/xln-health.json
    pm2 status
    pm2 logs --lines 200
    
  3. Check storage pressure:

    bun run debug:disk
    jq '.disk, .storage.tracked[] | {name, kind, currentBytes, bytesPerHour, scanTruncated}' /tmp/xln-health.json
    df -h /
    du -sh logs .logs playwright-report test-results db 2>/dev/null || true
    

    Test runners clean old generated artifacts by default before new runs. Use XLN_KEEP_TEST_ARTIFACTS=1 only when preserving traces/videos/logs is intentional. Generated test artifacts are budgeted at 50 GiB by default via XLN_TEST_WORKSPACE_MAX_BYTES.

  4. If a hub is down but the orchestrator is healthy, prefer an orchestrator-managed restart over killing child PIDs manually:

    pm2 restart xln
    sleep 10
    curl -fsS https://xln.finance/api/health | jq '{coreOk, systemOk, degraded, hubs}'
    

Storage Recovery

Release Check

Before declaring a deploy healthy:

bun run gate:ci
bun run soundcheck runtime frontend/src package.json .github/workflows
bun run test:e2e:fast
bun run test:rpc-settlement
bun run security:audit-pack
bun run prod:health

Use bun run deploy:prod for the canonical production rollout. It always resets the public testnet chain, runtime, mesh, and custody state before bootstrapping. Hub and market-maker WAL is not a deployment input. bun run deploy:prod:fresh is an explicit alias for the same clean deployment policy.

Before a public-net release candidate, also run:

bun run gate:release
bun run soak:release

For the public-testnet launch envelope in the current AGENTS.md scope, with one tower and three hubs, use the existing operator gate below. The owner removed the global monetary cap on 2026-10-11; these historical command names do not restrict participation or imply guaranteed unsecured recovery.

export XLN_CAPPED_TESTNET_TOWER_URLS=https://tower.example.com
bun run gate:capped-testnet:preflight
bun run gate:capped-testnet

gate:capped-testnet writes a JSON report under .logs/gates/ and runs the one-hour capped soak. If any step fails, fix the root cause, commit the fix, and start the capped gate again from a fresh release-candidate commit.