Metrics scrape tax

What does enabling richer Prometheus scrape metrics cost under forward_fast?

Numbers are same-host comparisons on a single reference host and are not service-level objectives. See the performance hub disclaimer.

When this matters

Operators choose metrics.base (minimal vs standard) for dashboard richness versus cardinality and hot-path cost. See Operator metrics bases. This study puts an observability-off baseline next to minimal and standard scrape postures under the same forward_fast load. For OTLP push cost under the same recipe, see OTLP tax under load.

What we varied

Varied: observability posture (metrics_off → metrics_minimal_scrape → metrics_standard_scrape)
Held constant: sync runtime, forward_fast stub, ingress workers
Loadgen: published forward_fast dnsperf recipe — -c 16 / -T 8 / -q 2000 (shared with other forward_fast / cache_hit scale and feature-tax cells; see How load is applied) — each pole is also the median of 3 independent rounds

Metrics scrape tax — published forward_fast recipe

These scrape-tax cells use the same published forward_fast dnsperf recipe as other scale / feature-tax fast cells (clients 16, threads 8, max outstanding 2000) and are the median of 3 independent rounds. That is the shared publish recipe so achieved QPS reflects Conduit capacity rather than a thin outstanding window — not a study-only workaround. See How load is applied. Remeasure locally with the same recipe before sizing.

Evidence

Feature tax — metrics scrape tax (forward_fast)

Download CSV

Posture	Runtime	Achieved QPS	Avg latency (ms)	Sent	Completed	Workers
metrics_off	sync	76124.1	26.2	763843	763843	ingress=2
metrics_minimal_scrape	sync	73038.1	27.3	732337	732337	ingress=2
metrics_standard_scrape	sync	70157.8	28.4	703550	703550	ingress=2

At a glance

metrics scrape tax (forward_fast): metrics_minimal_scrape costs about 4% QPS versus metrics_off (~73k vs ~76k); metrics_standard_scrape costs about 8% QPS versus metrics_off (~70k vs ~76k).

Takeaway

Richer Prometheus scrape costs more QPS as you climb the series. Versus observability off on this median (~76k), minimal scrape costs about 4% (~73k), and standard scrape about 8% (~70k). That is a same-host tax signal, not an SLO.

What to do: pick minimal vs standard from the cardinality you need (operator metrics bases), then remeasure on your hardware (--study metrics-scrape-ladder). See also Metrics collect vs emit and, for split_io, Metrics scrape (split_io).