Relay Observability

Relay tracks every message from publish to final delivery. The tracing system records timing information, budget consumption, and error details for each delivery attempt. Aggregate metrics provide a bird's-eye view of your message bus health, and the dead letter queue captures messages that could not be delivered.

This guide covers how to use traces, metrics, and debugging tools to understand what is happening inside Relay.

Message Tracing

Every message published through Relay is assigned a trace ID, and each delivery to an endpoint creates a span. A span records the full lifecycle of a single delivery attempt: when it was sent, when it arrived at the endpoint, when processing completed, and whether it succeeded or failed.

Trace Span Fields

Looking Up a Trace

To inspect the full delivery trace for a message, use the trace endpoint with the message ID:

curl http://localhost:4242/api/relay/messages/01HX.../trace

The response includes every span in the trace chain, ordered by sentAt ascending:

{
  "traceId": "01HXABC123",
  "spans": [
    {
      "id": "01HXDEF456",
      "messageId": "01HXABC123",
      "traceId": "01HXABC123",
      "subject": "relay.agent.backend",
      "status": "delivered",
      "sentAt": "2025-02-26T12:00:00.000Z",
      "deliveredAt": "2025-02-26T12:00:00.050Z",
      "processedAt": "2025-02-26T12:00:00.200Z",
      "errorMessage": null,
      "metadata": null
    }
  ]
}

For messages that fan out to multiple endpoints, the trace contains one span per endpoint.

Trace Statuses

Each span transitions through a lifecycle of statuses:

A healthy message moves from sent to delivered. The time between sentAt and deliveredAt is the delivery latency. The time between deliveredAt and processedAt is the processing latency, which includes the time the adapter (e.g., Claude Code) takes to handle the message.

Using MCP Tools

Agents running inside DorkOS can inspect traces using the built-in MCP tools without making HTTP calls:

relay_get_trace     Get the full delivery trace for a message by ID
relay_get_metrics   Get aggregate delivery metrics for the bus

The relay_get_trace tool accepts a messageId parameter and returns the same trace data as the REST endpoint. This is useful for agents that need to verify whether a message they sent was successfully delivered before proceeding with the next step.

Delivery Metrics Dashboard

Relay computes aggregate delivery metrics from the trace store using live SQL aggregates. These metrics provide a summary of bus health without requiring you to inspect individual traces.

Fetching Metrics

Retrieve the current metrics snapshot from the trace metrics endpoint:

curl http://localhost:4242/api/relay/trace/metrics

{
  "totalMessages": 1542,
  "deliveredCount": 1480,
  "failedCount": 12,
  "deadLetteredCount": 50,
  "avgDeliveryLatencyMs": 45.2,
  "p95DeliveryLatencyMs": null,
  "activeEndpoints": 8,
  "budgetRejections": {
    "hopLimit": 0,
    "ttlExpired": 0,
    "cycleDetected": 0,
    "budgetExhausted": 0
  }
}

Metrics Fields

Interpreting the Numbers

A healthy Relay bus has a high delivered-to-total ratio and low dead letter counts. Here are patterns to watch for:

• High deadLetteredCount relative to total -- Check the budget rejections breakdown to find the root cause. A spike in hopLimit rejections often indicates a message loop between two agents. TTL expirations may mean agents are too slow to process messages within the default 1-hour window.

• Rising failedCount -- Subscriber handlers are throwing errors. Check the individual traces for the error field to identify which adapter or endpoint is failing. The circuit breaker will automatically stop delivering to endpoints with 5 consecutive failures.

• High avgDeliveryLatencyMs -- Deliveries are slow on average. This is often caused by a single slow endpoint pulling up the mean. Inspect traces filtered by endpoint to find the bottleneck.

The DorkOS client UI includes a Delivery Metrics Dashboard in the Relay panel that visualizes these numbers with success/failure breakdowns and latency distribution. Access it from the Relay tab in the sidebar when Relay is enabled.

Debugging Failed Deliveries

When a message does not reach its destination, Relay provides several tools to diagnose the issue. Start with the dead letter queue, then drill into individual traces, and check the reliability subsystems.

curl http://localhost:4242/api/relay/dead-letters

curl http://localhost:4242/api/relay/messages/{messageId}/trace

SSE Stream for Real-Time Monitoring

For real-time visibility, subscribe to the Relay SSE stream. The stream emits events for every message delivery and signal:

curl -N http://localhost:4242/api/relay/stream?subject=%3E

The subject query parameter filters which messages appear in the stream. Use > (URL-encoded as %3E) to see all messages, or provide a specific pattern like relay.agent.* to filter by audience.

The stream emits four event types:

Next Steps