This document defines the canonical debugging and event-sourcing flow for XLN networking and HTLC/payment lifecycle.
If you touch runtime, networking, relay, or payment code, follow this file.
Use one event timeline and one root-cause incident view:
GET /api/debug/eventsGET /api/debug/incidents?state=openThese endpoints must answer, for any incident:
Single-source pipeline:
env.warn, env.error, env.emit.core/runtime/observability/env-events.ts forwards critical/high-signal events via P2P debug_event.debug_event messages to relay.console.error, window.error, unhandled promise
rejection and Svelte errors to /api/debug/events/ingest.Core files:
core/runtime/observability/env-events.tscore/network/p2p/ws-client.tscore/network/p2p/p2p.tscore/network/p2p/ws-protocol.tscore/api/server/index.tscore/network/relay/debug-http.tsfrontend/src/lib/debug/browser-telemetry.tsRelay event shape (from core/api/server/index.ts):
id, ts, eventruntimeId, from, to, msgType, status, reason, encrypted, size, queueSize, detailsKey relay events:
ws_open, ws_closehellomessagedeliverygossip_store, gossip_requestdebug_eventerrorBuffer:
MAX_RELAY_DEBUG_EVENTS = 5000unread -> acknowledged -> resolved; a new occurrence
reopens the incident as unread${XLN_DEBUG_INCIDENT_JOURNAL_PATH:-<dbRoot>.debug-incidents.jsonl}. Every
mutation is redacted, appended and fsynced; reset keeps monotonic cursors,
restart restores state, a torn final append is discarded, and complete
corruption fails startup.Endpoint:
GET /api/debug/eventsSupported filters:
last (1..5000, default 200)eventruntimeIdfromtomsgTypestatussince (unix ms)Examples:
curl -s "https://xln.finance/api/debug/events?last=200" | jq .
curl -s "https://xln.finance/api/debug/events?event=error&last=200" | jq .
curl -s "https://xln.finance/api/debug/events?runtimeId=0xabc...&since=1770470000000" | jq .
curl -s "https://xln.finance/api/debug/events?msgType=entity_input&status=delivered&last=500" | jq .
curl -s "http://127.0.0.1:8082/api/debug/incidents?state=open" | jq .
Agent/operator shortcut:
bun run debug:incidents
bun run debug:incidents -- --state=unread --ack
bun run debug:incidents -- --resolve=<fingerprint>
The first command exits non-zero while any open incident exists. Run it before reading scattered log files and again after every fix. Resolve an incident only after its L1 and L2 regressions are green; a recurrence reopens it.
All critical paths MUST emit structured events through env/p2p pipeline, not ad-hoc console logs.
env.error(...) for failures that affect payment correctness, liveness, or consensus.env.warn(...) for degraded behavior that can become failure.env.emit(...) for high-signal lifecycle milestones.data when available:
runtimeId, entityId, fromEntityId, toEntityIdaccountId, hashlock, lockId, height, routemessage deterministic and grep-friendly (all caps with _ for error codes).reason or normalized error details.entityTx) and be event-sourced; no direct P2P-only metadata mutation.REA metadata path:
type: 'profile-update'core/entity/tx/apply.ts)debug_event, message, gossip_store)HtlcInitiatedHtlcFinalizedHtlcFailedDo:
env.error('network', 'ENVELOPE_DECRYPT_FAIL', {
lockId,
fromEntityId,
toEntityId,
reason: err.message,
}, state.entityId);
Do:
env.emit('HtlcFinalized', {
hashlock,
secret,
inboundEntity,
outboundEntity,
entityId: state.entityId,
});
Avoid:
console.log('failed'); // no structured source-of-truth signal
For a failed payment:
runtimeId.event=debug_event and event=error.message -> delivery -> runtime debug_event -> finalize/fail.Symptom:
runtime adapter response too large/app never materializes window.isolatedEnvSource of truth:
[RADAPTER] RESPONSE_TOO_LARGE {...}RuntimeAdapterResponseTooLargeUseful prod commands:
ssh root@xln.finance 'grep -R "RADAPTER] RESPONSE_TOO_LARGE" -n /root/.pm2/logs /root/xln/db 2>/dev/null | tail -n 20'
curl -fsS https://xln.finance/api/runtime-import | jq '.manifest.entries[] | {label,access,wsUrl,hasToken:(.token|length>0)}'
Expected event fields:
op, path, querybytes, maxBytesruntimeId, heightpayloadKeysRoot-cause rule:
XLN_RADAPTER_MAX_MESSAGE_BYTES./view-frame, accounts/books and heavy core maps must stay bounded.Minimal incident query:
curl -s "https://xln.finance/api/debug/events?last=1000&event=error" | jq .
Current pipeline is strong but not perfect.
Highest-value improvements:
traceId/paymentId propagated end-to-end (route build -> HTLC lock -> resolve).debug_event payloads (browser intake is already bounded and validated)./api/debug/events in production.Any PR touching network/payment/consensus is not done unless:
/api/debug/events.Always validate these 3 scenarios, in this order, before merge/redeploy:
lock-ahb workse2e ahb worksah1-3b works# 0) clean debug state (preserve hub profiles)
curl -s -X POST "https://xln.finance/api/reset" | jq .
# 1) lock-ahb
bun run scenario:lock-ahb
# 2) e2e ahb
bunx playwright test tests/e2e/payments/e2e-ahb-payment.spec.ts
# 3) ah1-3b (multi-hop via H1/H2/H3)
bun run scenario:ah1-3b
All three must pass in the same run window:
error events in /api/debug/events attributable to the scenario window.HtlcInitiated and HtlcFinalized.AWAITING CONSENSUS accounts after scenario completion.If any one of the 3 fails, treat the build as failed. Do not deploy partial fixes.
Prevent new direct RPC/BrowserVM calls outside adapter internals:
bun run check:jadapter-boundary
For faster system validation, run scenarios in parallel with one Anvil process per worker. This avoids nonce/deploy collisions from sharing a single RPC node.
Command:
bun run test:system:parallel
Useful flags:
# Run only selected scenarios
bun run test:system:parallel -- --scenarios=processbatch,rebalance,settle-rebalance
# Control concurrency and port range (worker i => base-port+i)
bun run test:system:parallel -- --workers=6 --base-port=18545
# Stream prefixed logs live
bun run test:system:parallel -- --stream
Notes:
chainId=31337..logs/system-tests/<timestamp>/.--timeout-ms, default 15 minutes).Last updated: 2026-02-18