Observability

featherbit exposes Prometheus metrics and health probes on the admin port, and logs through the tracing subscriber configured in system.yaml.

Prometheus metrics

GET /metrics on the admin port renders the shared gateway registry in the Prometheus text exposition format (text/plain; charset=utf-8). Unlike the health probes, /metrics requires Basic auth (see Admin API).

Ten metric families are recorded — per-route metrics by the data plane, per-node metrics by the graph engine, plus per-store error and cache-event counters:

Metric	Type	Labels	Description
gateway_requests_total	counter	route, method, status	Total number of requests
gateway_request_duration_seconds	histogram	route	End-to-end request latency (buckets 1 ms to 5 s)
gateway_request_errors_total	counter	route, error_code	Genuine node failures by error code — one increment per error record left in context.errors
gateway_node_executions_total	counter	policy, node_id, node_type	Graph node executions
gateway_node_duration_seconds	histogram	policy, node_id	Per-node execution latency (buckets 0.1 ms to 500 ms)
gateway_node_errors_total	counter	policy, node_id, error_code	Node failures by error code
gateway_consumer_requests_total	counter	consumer, route	Requests attributed to an authenticated consumer, by route
gateway_counter_store_errors_total	counter	store	Counter-store (stores:) backend errors per named store
gateway_session_store_errors_total	counter	store	Session-store backend errors per named store
gateway_cache_events_total	counter	backend, store, event	proxy-cache outcomes. backend is local or redis; store is the named stores: entry for redis, empty for local. event is one of hit, miss, error, too_large, eviction: hit and miss partition every lookup, so the hit rate is hits/(hits+misses); error overlaps the miss or skipped write it caused rather than adding a fifth bucket; too_large counts a response skipped for exceeding max_object_bytes; eviction (backend local only) counts an entry discarded because cache.max_entries was reached; purge counts purge operations, not entries removed — one increment per phase: purge node execution, regardless of how many keys it cleared. The Admin API's DELETE /api/cache/:id and the MCP purge_cache tool purge the same backends directly and are not counted here, since they act outside any policy node

The per-node families let you pinpoint which node inside a routing policy is slow or failing, not just which route.

Errors are failures, not rejections

gateway_request_errors_total and gateway_node_errors_total count only cases where a node could not do its job — an unreachable upstream, a failed IdP callout, a counter store that is down, input the node cannot parse. A deliberate rejection leaves its node through an outcome port (denied, limited, broken, abort, redirect, preflight, routed, hit) and appends no error record, so it does not increment either counter.

Alert on genuine faults with the error counters; measure denials and throttles from gateway_requests_total's status label (status="401", status="429", ...) instead.

curl -u admin:admin http://localhost:9090/metrics

Health and readiness

Both probes are served on the admin port and are exempt from authentication, so orchestrators can hit them without credentials.

Endpoint	Semantics
GET /healthz	Liveness: always 200 OK with {"status": "healthy"} while the process is running
GET /readyz	Readiness: 200 OK with the compiled route count once at least one route is loaded; 503 Service Unavailable with {"status": "not_ready", "reason": "no routes loaded"} while the route table is empty

Use /healthz for liveness probes (restart on failure) and /readyz for readiness probes (remove from load balancing until routes are compiled).

Structured logging

Logging is configured in system.yaml and handled by the tracing subscriber:

logging:
  level: ${LOG_LEVEL:-info}
  format: json        # "json" (default) or any other value for plain text

• level — log level filter (trace through error); defaults to info. The RUST_LOG environment variable, when set, overrides level at startup.
• format — json (the default, recommended for production) or plain text.

Per-request tracing

Metrics and logs aggregate; they tell you that a route is slow or failing, not which node did it. For a step-by-step view of one request — the Context after every plugin, what each one changed, and which edge the engine followed — see Debugging & sandbox.