The pipe that swallows your data
You write a Rust CLI tool that processes text. You run cat data.txt | my_tool and the terminal hangs. The tool does nothing. You try echo "hello" | my_tool and it prints nothing. The shell pipeline feels like a black hole. This happens because Rust's standard I/O behaves differently when connected to a pipe versus a terminal. The data is flowing, but your tool is waiting for a signal that never comes, or it's holding onto output that the next tool needs right now.
Streams, buffers, and the newline trap
Think of standard input and output as streams of bytes. When you run a program interactively, the input stream is your keyboard and the output stream is your screen. The screen is "blocking" in a specific way: it waits for you to press Enter before sending data to the program. A pipe is different. It's a tube connecting two programs. Data flows through the tube continuously. Rust's stdin() returns a handle to this stream.
The tricky part is buffering. Rust buffers output by default to avoid the overhead of writing one byte at a time. Every write to the operating system costs time. Buffering batches those writes. When you pipe output to another tool, that buffer might never flush, so the next tool sees nothing until your program exits. Input has a similar trap. read_line waits for a newline character. If your piped input has no newlines, read_line waits forever for data that might never arrive.
Pipes are tubes, not keyboards. Treat them as streams of bytes that might arrive in chunks.
Minimal echo tool
Start with a tool that reads one line and echoes it back. This shows the basic mechanics of read_line and write_all.
use std::io::{self, Read, Write};
/// Echoes a single line of input to output.
fn main() -> io::Result<()> {
let mut buffer = String::new();
// read_line reads until a newline or EOF.
// It keeps the newline character in the buffer.
let bytes_read = io::stdin().read_line(&mut buffer)?;
// Check if we actually got data before writing.
if bytes_read > 0 {
// write_all writes the full slice, handling partial writes internally.
io::stdout().write_all(buffer.as_bytes())?;
}
Ok(())
}
read_line returns the number of bytes read. If the input is empty, it returns zero. Checking bytes_read > 0 prevents writing an empty string when the pipe closes immediately. write_all ensures the entire buffer is written. The underlying OS write might only accept part of the data at once. write_all loops internally until every byte is sent.
Trust write_all for completeness, but not for timing. It guarantees the data leaves your program, not that the other side receives it instantly.
How the compiler and runtime handle the flow
When you run this interactively, read_line blocks waiting for input. You type "hello" and press Enter. The newline triggers the read. The function returns. The output goes to the screen. When you pipe echo "hello" | my_tool, the pipe provides "hello\n". read_line sees the newline, returns immediately. The output goes to the pipe's stdout.
If you try to pass a Vec<u8> to read_line, the compiler rejects you with E0308 (mismatched types). read_line expects a &mut String because it assumes the input is valid UTF-8 text. Binary data breaks this assumption. If you are processing binary files, read_line is the wrong tool.
The runtime manages the buffer behind the scenes. io::stdin() returns a Stdin struct. This struct wraps the OS file descriptor for standard input. When you call read_line, it reads bytes from the OS and appends them to your string. The string grows as needed. If you pipe a 1GB file with no newlines, read_line will allocate 1GB of memory. This is a common source of out-of-memory crashes in CLI tools.
Never use read_line for unbounded input. Check the length or use a streaming approach.
Realistic text processor
Real tools process many lines. Allocating a new string for every line is wasteful. BufReader wraps the input stream and provides efficient line-by-line reading. It fetches chunks from the OS and serves them from an internal buffer.
use std::io::{self, BufRead, BufReader, Write};
/// Counts lines and words from stdin, prints summary to stdout.
fn main() -> io::Result<()> {
// BufReader adds a buffer layer.
// Reading byte-by-byte from a pipe is extremely slow.
// BufReader fetches chunks and serves them efficiently.
let reader = BufReader::new(io::stdin());
let mut line_count = 0;
let mut word_count = 0;
// lines() iterates over lines, stripping newlines.
// It handles the buffering internally.
for line_result in reader.lines() {
// Propagate any I/O error immediately.
let line = line_result?;
line_count += 1;
// split_whitespace handles multiple spaces and tabs.
word_count += line.split_whitespace().count();
}
// BufWriter batches output writes.
// Writing one byte at a time to a pipe causes excessive syscalls.
let mut writer = io::BufWriter::new(io::stdout());
// writeln! formats and writes with a newline.
writeln!(writer, "Lines: {}, Words: {}", line_count, word_count)?;
// flush() forces the buffer to the underlying stream.
// Critical when piping to another tool that needs data now.
writer.flush()?;
Ok(())
}
BufReader::new(io::stdin()) creates the wrapper. lines() returns an iterator over Result<String>. Each iteration reads a line, strips the newline, and yields the string. If you try to call lines() directly on io::stdin(), the compiler rejects you with E0277 (trait bound not satisfied). Stdin implements Read, not BufRead. You must wrap it in BufReader.
The community convention is to use lines() for text processing loops. It strips newlines automatically and handles buffering. read_line is reserved for cases where you need to preserve the newline character or process raw input without the iterator overhead.
Use BufReader for text processing. It saves memory and CPU cycles.
The buffering hang
Output buffering causes the most frustrating pipeline bugs. println! writes to stdout. When stdout is connected to a terminal, Rust often flushes on newline. When stdout is a pipe, Rust buffers the output. println! does not guarantee a flush. Your tool might finish, but the next tool sees nothing until your process exits and the buffer is dropped.
This creates a hang. The next tool waits for data. Your tool has already written the data to the buffer and exited. The buffer is dropped, the data is sent, but the next tool has already timed out or moved on.
The fix is explicit flushing. io::stdout().flush()? forces the buffer to the pipe. Call this after critical output. If you use BufWriter, call writer.flush()?.
Rust's is_terminal() function is unstable. The community often uses the atty crate to check if stdout is a terminal, or just flushes explicitly to be safe. Flushing on every line is slow. Flush only when the pipeline demands immediate data.
Buffering is your friend for performance, but your enemy for latency. Flush when the pipeline demands it.
Binary data and the string trap
CLI tools often handle binary data. Images, compressed files, or raw bytes. read_line and lines() assume UTF-8 text. If the input contains invalid UTF-8, these methods return an error. You cannot use them for binary data.
Use read with a fixed-size buffer. read fills a byte slice and returns the number of bytes read. It might read fewer bytes than requested. You must loop until you get the amount you need or hit EOF.
use std::io::{self, Read, Write};
/// Copies binary data from stdin to stdout in chunks.
fn main() -> io::Result<()> {
// 8KB buffer balances memory usage and syscall frequency.
let mut buffer = [0u8; 8192];
let mut stdin = io::stdin();
let mut stdout = io::stdout();
loop {
// read fills the buffer and returns bytes read.
// Returns 0 on EOF.
let bytes_read = stdin.read(&mut buffer)?;
if bytes_read == 0 {
break;
}
// write_all ensures the chunk is fully written.
stdout.write_all(&buffer[..bytes_read])?;
}
Ok(())
}
This pattern is the standard way to copy binary streams. The buffer size of 8KB is a common convention. It matches typical OS page sizes and minimizes syscalls without using too much memory.
Text gets String. Binary gets Vec<u8> or a fixed buffer. Pick the right type for your data.
Decision: when to use this vs alternatives
Use io::stdin().read_line(&mut buf) when you need to preserve newlines and process one line at a time in a tight loop where allocation overhead matters less than simplicity.
Use BufReader::new(io::stdin()).lines() when you are processing text line-by-line and want the iterator ergonomics and automatic newline stripping.
Use io::BufWriter::new(io::stdout()) when you are writing many small chunks and need to reduce syscall overhead in a pipeline.
Use io::stdout().flush() when you pipe output to a tool that requires immediate data, such as a progress bar or an interactive filter.
Use std::process::Command with Stdio::piped() when you need to spawn a child process and capture its output or feed it input programmatically.
Pick the abstraction that matches your data flow. Text processing gets BufReader. Binary streaming gets raw read loops.