Why your async app feels slow but htop says it's idle
You've written an async Rust service. Maybe a small HTTP gateway, maybe a Discord bot. It compiles, it runs, and under load it feels sluggish. CPU is sitting at 12%. Memory is fine. top says nothing is wrong. So where is the time going?
With threaded code, you'd attach a sampling profiler and look at hot stack frames. Async Rust ruins that picture: a stack trace usually shows the runtime polling some future, with no clue which future is misbehaving. Tasks are not threads. They don't show up in ps. They don't have stable stacks. You need a tool that understands the runtime.
That's tokio-console. It's a top-like terminal UI for async Rust applications, specifically the ones built on Tokio. It shows you every spawned task, how often each one was polled, how long it took to poll, and whether anything is stuck in a way that should worry you. Once you've seen it, you'll wonder how you debugged async code without it.
What you're actually instrumenting
There are two pieces. The first is console-subscriber, a crate you add to your application. It installs a tracing subscriber that emits structured events about every task spawn, every poll, every wake, every resource (mutexes, semaphores, sleeps). It listens on a local port (default 6669) and serves these events over gRPC.
The second piece is the tokio-console binary, which connects to that port and renders a live dashboard. Tasks list, resource list, drill into a single task to see its history. The dashboard runs in a separate terminal so it doesn't compete with your app's stdout.
Think of it like top plus strace for futures. top shows you what's busy. strace shows you what each one is doing. tokio-console shows you both, and crucially, it shows you which tasks are not making progress when they should be.
Wiring it into your app
First, add console-subscriber to your project. Note: this is the subscriber crate, not tokio-console itself. The CLI is a separate binary.
# Cargo.toml
[dependencies]
tokio = { version = "1", features = ["full", "tracing"] }
console-subscriber = "0.4"
Now initialize the subscriber at the top of main. The subscriber has to run before any tasks are spawned, otherwise it'll miss them.
// main.rs
#[tokio::main]
async fn main() {
// Install the console subscriber. This sets up the tracing subscriber
// and starts the gRPC server that tokio-console connects to.
// Default endpoint is 127.0.0.1:6669.
console_subscriber::init();
// Now spawn whatever your app actually does. Every task from this point
// on will be visible in the console.
let h = tokio::spawn(async {
for i in 0..5 {
// Sleep is itself a tracked resource, so the console will show
// each await as a paused state on this task.
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
println!("tick {i}");
}
});
h.await.unwrap();
}
There's one more knob. Tokio's instrumentation uses unstable APIs that aren't enabled by default. You have to set a cfg flag at compile time via RUSTFLAGS:
# This rebuilds the binary with the unstable Tokio task tracking enabled.
RUSTFLAGS="--cfg tokio_unstable" cargo build
RUSTFLAGS="--cfg tokio_unstable" cargo run
If you forget the flag, your app will still compile and run, but the console will show no tasks because the runtime never reports them. The console-subscriber docs nag you about this and the README has the exact incantation. Many projects put it in a .cargo/config.toml:
# .cargo/config.toml
[build]
rustflags = ["--cfg", "tokio_unstable"]
That way you don't have to remember the env var on every build.
Connecting from another terminal
Install the CLI once:
cargo install --locked tokio-console
Then, with your app running, in a second terminal:
tokio-console
It connects to 127.0.0.1:6669 by default. You'll see a list of tasks. Arrow keys to navigate, Enter to drill in, r to switch to the resources view (mutexes, semaphores, channels, sleeps), t for tasks. Press ? for the full keymap.
What the dashboard tells you
For each task you get:
- Total polls : how many times the runtime woke this task up to make progress.
- Busy : cumulative time spent inside
poll, i.e. actually running. - Idle : time the task was alive but waiting on something (a sleep, a channel, IO).
- Last poll duration : how long the most recent poll took. If a single poll is taking tens of milliseconds, you've probably got a
.await-less blocking call inside it. - Warnings :
tokio-consoleflags tasks that exhibit known bad patterns. Two big ones:- Long-running task with no yields : a task is spending a lot of time inside one poll. Usually a CPU-heavy loop or a blocking syscall.
- Never yielded : the task ran to completion in a single poll. Often fine for short work, suspicious for long work.
The resource view shows things like "this Mutex has been held by task #42 for 3 seconds, with tasks #51 and #67 waiting." Same for semaphores and channels with full buffers.
A more realistic example: tracking a stuck task
Suppose your service has a worker that should churn through a queue. Some days it stops dead. CPU is idle, but the queue grows. Add the subscriber, run with the cfg flag, and start tokio-console in another window.
use tokio::sync::mpsc;
// A worker that pulls jobs off a channel and processes them.
async fn worker(mut rx: mpsc::Receiver<u32>) {
while let Some(job) = rx.recv().await {
// Suppose this calls something that occasionally blocks for a long time.
process(job).await;
}
}
// A toy "process" that occasionally hangs.
async fn process(job: u32) {
if job % 1000 == 0 {
// BUG: this is a synchronous sleep. It blocks the *thread*, not just
// the task. tokio-console will flag the task as never yielding.
std::thread::sleep(std::time::Duration::from_secs(5));
}
tokio::time::sleep(std::time::Duration::from_millis(1)).await;
let _ = job;
}
#[tokio::main]
async fn main() {
console_subscriber::init();
let (tx, rx) = mpsc::channel(100);
tokio::spawn(worker(rx));
for i in 0..10_000 {
tx.send(i).await.unwrap();
}
}
With the console attached, you'll see the worker task occasionally have a single poll with a duration of 5+ seconds, and a warning marker next to it. That's your hint: somewhere in process, you're calling a synchronous blocking function. Replace std::thread::sleep with tokio::time::sleep and the warning goes away.
This is the kind of bug that's almost impossible to find with a regular profiler, because the stack trace points at the runtime, not the offending function. The console points at the task, the time, and the warning.
Common pitfalls
You added console-subscriber and tokio-console shows nothing. The most likely cause: you forgot --cfg tokio_unstable. Also check that tokio has the tracing feature enabled in Cargo.toml.
You see tasks but no resources. The resources view requires the same instrumentation, plus it only tracks Tokio-provided primitives (Tokio mutexes, channels, sleeps). It won't see std::sync::Mutex. If you're using a parking_lot mutex, it won't appear there either.
The console connects but the UI is blank. Double-check your app actually called console_subscriber::init() before spawning anything. If you used console_subscriber::Builder to customize, make sure you called .init() at the end.
You tried tokio-console against a release build and got little useful info. The instrumentation still works in release, but inlining can collapse some of the timing detail. For diagnosing async stalls, debug builds (or release with debug = 1) are usually fine.
Your port is in use. Pass --retain-for 60s and a custom address, e.g. tokio-console http://127.0.0.1:6670, and configure the subscriber on the app side with console_subscriber::Builder::default().server_addr(...).
You shipped console-subscriber to production by mistake. Fine for staging, but in production it adds nontrivial overhead per task. Gate it behind a feature flag or a CLI argument so it's only on when you ask for it.
When to reach for tokio-console
Reach for tokio-console when something in your async app feels off and a regular profiler isn't telling you anything. Stuck tasks. Tasks that keep getting woken but make no progress. A deadlock that looks like idle. CPU at 100% with no obvious hot path.
For pure CPU work, a sampling profiler like perf, samply, or cargo flamegraph is still the right tool. They'll tell you which function is hot. But for "why isn't this future making progress," tokio-console is the answer.
For metrics in production, you probably want something lighter weight: tracing with a JSON or OTLP subscriber, plus a metrics crate like metrics exporting to Prometheus. The console is a development and staging tool, not a long-term observability solution.