Skip to content

Metrics and tracing

This guide is an end-to-end lab: enable built-in metrics (Prometheus scrape), optional pipeline tracing, and the control plane so you can fetch traces with conduitctl trace. For OTLP HTTP metrics push, see OTLP metrics push smoke. For event export, see Event export and dnstap.

Prerequisites: Conduit built and on your PATH; an upstream DNS listener on 127.0.0.1:5300 (or adjust the pool backend below). Follow Install and run if you have not started Conduit yet.

What you will verify

  1. Prometheus-format metrics at an HTTP scrape endpoint
  2. Per-query counters increment after DNS traffic
  3. Pipeline trace events for a matching query via conduitctl trace

1. Write the config

Save as conduit-obs-lab.yaml (ports match common lab layouts: DNS 15353, metrics 9090, control 5199):

schema_version: 1
listeners:
  listeners:
    - address: "127.0.0.1:15353"
      protocol: udp
pools:
  - name: default
    backends:
      - address: "127.0.0.1:5300"
control:
  listen_address: "127.0.0.1:5199"
metrics:
  enabled: true
  base: standard
  prometheus:
    listen_address: "127.0.0.1:9090"
    path: /metrics
tracing:
  enabled: true
  activation:
    selectors:
      - type: qtype
        value: A
    sample_percent: 100
  output:
    log_json: false
logging:
  level: info
  output: stderr
Block Role in this lab
control: Required for conduitctl trace
metrics: Hot-path recording + Prometheus scrape on 9090
tracing: Trace every A query (sample_percent: 100)
logging: Default info — quiet under load; bump to debug if you need txn_id on each query

Validate before start:

conduitctl validate --file conduit-obs-lab.yaml

2. Start Conduit

conduit /path/to/conduit-obs-lab.yaml

Confirm startup in the log: dataplane startup summary, listener on 15353, and no bind errors for 9090 or 5199.

Observability changes

Pipeline tracing: activation is established at process start — change it with a restart. metrics: plan knobs (base, categories, collect/emit, granularity) and Prometheus listen address apply live via reload or overlay — Metrics beyond bases, Metrics configurability — Overlay.

3. Smoke-test metrics

Before traffic, scrape should respond (gauges may be zero):

curl -sS "http://127.0.0.1:9090/metrics" | head

Send a query:

dig @127.0.0.1 -p 15353 +time=3 lab.example.com A

Scrape again and look for query counters (names depend on base — standard includes qtype labels):

curl -sS "http://127.0.0.1:9090/metrics" | grep conduit_queries

Expect conduit_queries_total and related series to increment. Series reference: Built-in metrics.

4. Fetch a pipeline trace

The first completed query on a worker often has txn_id=1. If conduitctl trace 1 returns not found, set logging.level: debug, restart, send another query, and read txn_id from a query complete line — see Logging — Per-query summaries.

conduitctl trace 1

Expect top-level phase lines such as lookup and send (with nested route/forward events when upstream runs) and elapsed microseconds plus pool/backend fields on forward attempts. Nested cache provider events include a cache field with the named instance. Alternative: gRPC GetTrace — gRPC and conduitctl — trace.

Traces are stored in memory (5 minute TTL, 1000 entry cap). Activation must match the query — here, A queries only.

5. Optional checks

Check Command / action
Config generation gauge curl -sS http://127.0.0.1:9090/metrics \| grep conduit_config_generation
Phase histograms (base: standard) grep conduit_phase_duration on scrape output after several queries
JSON trace on stderr Set tracing.output.log_json: true, restart, send a matching query; look for conduit::trace at info

What to do next