Your server is running, but you're flying blind
Your Rust web server is live. Requests are flowing. Then the dashboard turns red. The database is fine. The network is fine. But response times are spiking. You stare at the logs. They show success messages, but they don't tell you how long things took or how many requests hit a specific endpoint. You need numbers. You need metrics.
Rust doesn't include a metrics library in the standard library. The ecosystem converged on the metrics crate. It provides a clean abstraction layer. You record data in your code using simple macros. You plug in an exporter to send that data to Prometheus, StatsD, or Datadog. You can swap the backend later without touching your business logic.
The metrics abstraction
The metrics crate defines four types of measurements. Each one serves a specific purpose.
A counter only goes up. You use it for total requests, total errors, or total bytes sent. It resets only when the process restarts.
A gauge goes up and down. You use it for active connections, queue depth, or current memory usage. It reflects the state at a specific moment.
A histogram measures distribution. You use it for latency or payload size. You care about the 99th percentile, not just the average. A histogram groups values into buckets. This keeps memory usage low while giving you enough detail to spot slow outliers.
A set tracks unique values. You use it for distinct user IDs or active sessions. The exporter counts the number of unique items.
Think of the metrics crate like a universal power outlet. You plug in your code to record a counter or a histogram. The exporter is the adapter that converts that signal into a format your monitoring system understands. You can swap the adapter later without rewiring your code.
A minimal counter
Here is the smallest working example. It records a counter and exposes it via Prometheus.
use metrics::counter;
use metrics_exporter_prometheus::PrometheusBuilder;
fn main() {
// Install the recorder so metrics actually go somewhere.
// Without this, the macro calls are no-ops.
let _recorder = PrometheusBuilder::new()
.install_recorder()
.expect("Failed to install recorder");
// Record a simple increment.
// The macro expands to a call on the global recorder.
counter!("app_events_total").increment(1);
// Record with labels.
// Labels are key-value pairs that add dimensions to the metric.
counter!("http_requests_total", "method" => "GET").increment(1);
}
The counter! macro takes a name and optional labels. Labels let you slice the data. You can filter by method, path, or status code in your dashboard. The name must be a string literal. Labels must be stringifiable values.
If you pass a non-string label value, the compiler rejects it with E0308 (mismatched types). Labels must resolve to strings at runtime.
Install the recorder first. If you record before installing, you're shouting into the void.
How the recorder works
When you call install_recorder, the metrics crate stores a pointer to your exporter in thread-local storage. Every time you hit a macro, it looks up that pointer and sends the data. This is zero-cost if no recorder is installed. The macros expand to nothing. This keeps the dependency light for libraries that just want to emit metrics but not force a backend on their users.
The metrics-exporter-prometheus crate handles the heavy lifting. It aggregates the updates in memory. It exposes a /metrics endpoint that returns data in the Prometheus text format. Prometheus scrapes that endpoint on a schedule. It stores the time series data and lets you query it.
You need to call run_upkeep periodically on the handle. This cleans up old labels and prevents memory leaks. If you have high churn in your labels, the exporter needs to prune the dead ones. Spawn a background task to run upkeep every few seconds.
Real-world middleware
In a web server, metrics belong in middleware. You want to track every request without cluttering your route handlers. Axum makes this straightforward.
use axum::{
extract::{MatchedPath, Request},
middleware::{self, Next},
response::IntoResponse,
routing::get,
Router,
};
use metrics_exporter_prometheus::{Matcher, PrometheusBuilder, PrometheusHandle};
use std::{future::ready, time::{Duration, Instant}};
/// Sets up the Prometheus recorder and returns the handle for the /metrics endpoint.
fn setup_recorder() -> PrometheusHandle {
// Define buckets for latency.
// Exponential spread captures slow requests well without wasting memory.
const LATENCY_BUCKETS: &[f64] = &[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0];
let handle = PrometheusBuilder::new()
// Map specific buckets to the histogram name.
// This ensures latency data is bucketed correctly.
.set_buckets_for_metric(
Matcher::Full("http_request_duration_seconds".to_string()),
LATENCY_BUCKETS,
)
.unwrap()
.install_recorder()
.unwrap();
// Spawn upkeep to clean up old labels.
// Without this, memory usage grows over time.
let upkeep = handle.clone();
tokio::spawn(async move {
loop {
tokio::time::sleep(Duration::from_secs(5)).await;
upkeep.run_upkeep();
}
});
handle
}
/// Middleware that tracks request duration and status.
async fn track_metrics(req: Request, next: Next) -> impl IntoResponse {
let start = Instant::now();
// Get the route pattern, falling back to the raw path.
// MatchedPath gives the template like /users/:id.
// This prevents high cardinality from dynamic segments.
let path = if let Some(matched) = req.extensions().get::<MatchedPath>() {
matched.as_str().to_owned()
} else {
req.uri().path().to_owned()
};
let method = req.method().clone();
let response = next.run(req).await;
let latency = start.elapsed().as_secs_f64();
let status = response.status().as_u16().to_string();
// Record the histogram with labels.
// The macro handles aggregation and sending to the recorder.
metrics::histogram!(
"http_request_duration_seconds",
"method" => method.to_string(),
"path" => path,
"status" => status,
)
.record(latency);
response
}
fn main() {
let handle = setup_recorder();
let app = Router::new()
.route("/fast", get(|| async { "ok" }))
.route("/slow", get(|| async { tokio::time::sleep(Duration::from_secs(1)).await; "slow" }))
.layer(middleware::from_fn(track_metrics));
// Serve metrics on a separate port.
// This avoids blocking the main app and improves security.
let metrics_app = Router::new()
.route("/metrics", get(move || ready(handle.render())));
// Run both servers concurrently.
let _main_server = tokio::spawn(async {
axum::serve(
tokio::net::TcpListener::bind("127.0.0.1:3000").await.unwrap(),
app,
)
.await
});
let _metrics_server = tokio::spawn(async {
axum::serve(
tokio::net::TcpListener::bind("127.0.0.1:3001").await.unwrap(),
metrics_app,
)
.await
});
}
The middleware captures the start time. It extracts the matched path. It runs the request. It calculates latency. It records the histogram. The MatchedPath extractor is crucial here. It gives you the route template instead of the raw URI. If you use the raw URI, you get /user/123, /user/456, /user/789. That creates a new label value for every user ID. Prometheus will run out of memory. The template /user/:id keeps the label count stable.
Serve the metrics endpoint on a separate port. If your app is under heavy load, you don't want metrics scraping to add latency. A separate port also lets you restrict access with firewall rules. Metrics can leak sensitive information about your internal structure. Keep them behind a firewall.
Watch your label cardinality. One user ID in a label can take down your entire monitoring stack.
Pitfalls and gotchas
The metrics macros compile even if you forget to install a recorder. The code runs silently. You won't see an error. You'll just see empty dashboards. This is by design for library authors. If you're building an app, install the recorder in main before anything else.
Histogram buckets matter. If you pick bad buckets, your latency data becomes useless. Linear buckets waste memory on empty ranges. Exponential buckets adapt to the distribution of real-world traffic. Start with small buckets for fast requests and widen them for slow ones. Adjust based on your p99 latency.
Labels must be low cardinality. Never put user IDs, request IDs, or timestamps in labels. Those values change constantly. They create infinite time series. Use counters or gauges for unique values. Use sets if you must track distinct items, but keep the count reasonable.
The prometheus crate has a different API. It uses explicit metric objects. You create a Counter, clone it, and increment it. The metrics crate uses macros and a global recorder. The macro approach is simpler and less error-prone. You don't have to manage metric lifetimes. The global recorder handles aggregation.
Convention aside: metric names follow the pattern namespace_subsystem_name_unit. Use snake_case. Include the unit in the name. http_request_duration_seconds is correct. http_request_duration is ambiguous. Labels are lowercase. This matches Prometheus conventions and makes queries easier.
Choosing your tool
Use the metrics crate when you want a backend-agnostic API. You can swap Prometheus for StatsD or Datadog later without touching your business logic.
Use the metrics crate when you are writing a library. You can emit metrics without forcing a dependency on a specific exporter for your users.
Reach for the prometheus crate directly when you need advanced features that the abstraction doesn't expose, like custom collectors or complex push-gateway setups.
Use metrics-exporter-prometheus when you are building an application and have decided on Prometheus as your backend. It handles the global recorder and the HTTP endpoint for you.
Trust the borrow checker. It usually has a point.