Pipeline tracing tax

What does enabling full pipeline tracing cost under forward_fast?

Numbers are same-host comparisons on a single reference host and are not service-level objectives. See the performance hub disclaimer.

When this matters

Pipeline tracing records per-query phase detail for diagnosis. Unlike process logging levels, tracing is a dedicated tracing: block with selectors and sampling. This study compares observability off against tracing enabled for qtype A at 100% sample under forward_fast (dnsperf query file is A-heavy).

What we varied

Varied: tracing posture (off via metrics_off vs tracing_on)
Held fixed: metrics disabled, no dnstap, sync, forward_fast

Evidence

Feature tax — tracing off vs on (forward_fast)

Download CSV

Posture	Runtime	Achieved QPS	Avg latency (ms)	Sent	Completed	Lost	Workers
metrics_off	sync	75940.7	26.3	761386	761386	0	ingress=2
tracing_on	sync	48719.5	40.9	489630	489630	0	ingress=2

At a glance

tracing off vs on (forward_fast): tracing_on costs about 36% QPS versus metrics_off (~49k vs ~76k).

Takeaway

Full pipeline tracing is expensive. On this lab it costs about 36% QPS versus observability off (~49k vs ~76k) and raises average latency.

What to do: enable tracing for diagnosis windows; do not leave 100% sampling on in production. Narrow selectors and sample_percent, then remeasure.