xln

Single Source Network Debugging

This document defines the canonical debugging and event-sourcing flow for XLN networking and HTLC/payment lifecycle.

If you touch runtime, networking, relay, or payment code, follow this file.

Goal

Use one event timeline and one root-cause incident view:

These endpoints must answer, for any incident:

Current Architecture

Single-source pipeline:

  1. Runtime code emits structured events through env.warn, env.error, env.emit.
  2. core/runtime/observability/env-events.ts forwards critical/high-signal events via P2P debug_event.
  3. WS client sends debug_event messages to relay.
  4. Relay stores the high-volume network/debug timeline in an in-memory ring and grouped error incidents in a separate durable journal.
  5. Browser telemetry sends console.error, window.error, unhandled promise rejection and Svelte errors to /api/debug/events/ingest.
  6. Every error updates a separate fingerprinted incident registry. Gossip traffic may rotate the event ring but cannot evict an active root cause.
  7. HTTP API serves the raw timeline and the grouped incident view.

Core files:

Event Model

Relay event shape (from core/api/server/index.ts):

Key relay events:

Buffer:

HTTP Query API

Endpoint:

Supported filters:

Examples:

curl -s "https://xln.finance/api/debug/events?last=200" | jq .
curl -s "https://xln.finance/api/debug/events?event=error&last=200" | jq .
curl -s "https://xln.finance/api/debug/events?runtimeId=0xabc...&since=1770470000000" | jq .
curl -s "https://xln.finance/api/debug/events?msgType=entity_input&status=delivered&last=500" | jq .
curl -s "http://127.0.0.1:8082/api/debug/incidents?state=open" | jq .

Agent/operator shortcut:

bun run debug:incidents
bun run debug:incidents -- --state=unread --ack
bun run debug:incidents -- --resolve=<fingerprint>

The first command exits non-zero while any open incident exists. Run it before reading scattered log files and again after every fix. Resolve an incident only after its L1 and L2 regressions are green; a recurrence reopens it.

Root Cause, Not Log Volume

Mandatory Instrumentation Contract

All critical paths MUST emit structured events through env/p2p pipeline, not ad-hoc console logs.

MUST rules

  1. Use env.error(...) for failures that affect payment correctness, liveness, or consensus.
  2. Use env.warn(...) for degraded behavior that can become failure.
  3. Use env.emit(...) for high-signal lifecycle milestones.
  4. Include stable identifiers in data when available:
    • runtimeId, entityId, fromEntityId, toEntityId
    • accountId, hashlock, lockId, height, route
  5. Keep message deterministic and grep-friendly (all caps with _ for error codes).
  6. Never throw away failure context. Always include reason or normalized error details.
  7. Never rely on browser-only console output for critical diagnostics.
  8. Entity/profile metadata changes MUST flow through REA (entityTx) and be event-sourced; no direct P2P-only metadata mutation.

REA metadata path:

MUST instrument these cases

Coding Pattern

Do:

env.error('network', 'ENVELOPE_DECRYPT_FAIL', {
  lockId,
  fromEntityId,
  toEntityId,
  reason: err.message,
}, state.entityId);

Do:

env.emit('HtlcFinalized', {
  hashlock,
  secret,
  inboundEntity,
  outboundEntity,
  entityId: state.entityId,
});

Avoid:

console.log('failed'); // no structured source-of-truth signal

Debug Playbook

For a failed payment:

  1. Query relay timeline by recent window and runtimeId.
  2. Filter event=debug_event and event=error.
  3. Follow sequence:
    • message -> delivery -> runtime debug_event -> finalize/fail.
  4. If missing finalize, check:
    • decrypt errors
    • missing route/account
    • ACK/consensus failures
    • WS disconnect gap

Radapter Oversized Responses

Symptom:

Source of truth:

Useful prod commands:

ssh root@xln.finance 'grep -R "RADAPTER] RESPONSE_TOO_LARGE" -n /root/.pm2/logs /root/xln/db 2>/dev/null | tail -n 20'
curl -fsS https://xln.finance/api/runtime-import | jq '.manifest.entries[] | {label,access,wsUrl,hasToken:(.token|length>0)}'

Expected event fields:

Root-cause rule:

Minimal incident query:

curl -s "https://xln.finance/api/debug/events?last=1000&event=error" | jq .

Current pipeline is strong but not perfect.

Highest-value improvements:

  1. Add traceId/paymentId propagated end-to-end (route build -> HTLC lock -> resolve).
  2. Enforce schema validation for Runtime debug_event payloads (browser intake is already bounded and validated).
  3. Persist a redacted rolling copy of the high-volume event timeline; grouped incident state is already durable.
  4. Add retention tiers:
    • in-memory hot ring
    • compressed rolling files
  5. Add auth/rate limits for /api/debug/events in production.
  6. Add redaction policy for sensitive fields in debug payloads.
  7. Add alert rules (error spikes, delivery failures, reconnect storms).
  8. Add server-side derived views:
    • payment timeline by hashlock
    • stuck locks
    • unresolved deferred routes

Definition Of Done For New Code

Any PR touching network/payment/consensus is not done unless:

  1. Critical failures are visible in /api/debug/events.
  2. Success milestones are visible for happy path.
  3. E2E test can assert debug endpoint has relevant events.
  4. No critical incident requires multi-console reconstruction.

Main Testing Mantra (Mandatory Gate)

Always validate these 3 scenarios, in this order, before merge/redeploy:

  1. lock-ahb works
  2. e2e ahb works
  3. ah1-3b works

Required sequence

# 0) clean debug state (preserve hub profiles)
curl -s -X POST "https://xln.finance/api/reset" | jq .

# 1) lock-ahb
bun run scenario:lock-ahb

# 2) e2e ahb
bunx playwright test tests/e2e/payments/e2e-ahb-payment.spec.ts

# 3) ah1-3b (multi-hop via H1/H2/H3)
bun run scenario:ah1-3b

Pass criteria

All three must pass in the same run window:

  1. No error events in /api/debug/events attributable to the scenario window.
  2. HTLC lifecycle includes HtlcInitiated and HtlcFinalized.
  3. No stuck AWAITING CONSENSUS accounts after scenario completion.
  4. No lingering unresolved HTLC locks for the scenario entities.

Enforcement rule

If any one of the 3 fails, treat the build as failed. Do not deploy partial fixes.

JAdapter Boundary Guard

Prevent new direct RPC/BrowserVM calls outside adapter internals:

bun run check:jadapter-boundary

Parallel Scenario Runs (Isolated Anvil Per Worker)

For faster system validation, run scenarios in parallel with one Anvil process per worker. This avoids nonce/deploy collisions from sharing a single RPC node.

Command:

bun run test:system:parallel

Useful flags:

# Run only selected scenarios
bun run test:system:parallel -- --scenarios=processbatch,rebalance,settle-rebalance

# Control concurrency and port range (worker i => base-port+i)
bun run test:system:parallel -- --workers=6 --base-port=18545

# Stream prefixed logs live
bun run test:system:parallel -- --stream

Notes:

  1. Each worker enforces chainId=31337.
  2. Logs are written under .logs/system-tests/<timestamp>/.
  3. A scenario timeout is enforced (--timeout-ms, default 15 minutes).
  4. Runner exits non-zero if any scenario fails, and prints failing log tails.

Last updated: 2026-02-18