Reading and writing binary files
You're building a roguelike dungeon generator. The map data is a flat array of bytes representing tiles, enemies, and loot positions. You don't want to serialize this to JSON and pay the overhead of parsing strings. You want to dump the raw memory layout to disk and load it back instantly. You try File::open, grab some bytes, and suddenly the compiler screams about traits, mutability, and partial reads. Binary I/O in Rust feels like wrestling a bear wrapped in duct tape until you learn the moves.
Rust's approach to binary files is consistent, explicit, and safe. Once you understand the stream model and the difference between read and read_to_end, the code becomes straightforward.
The stream model
Rust treats I/O as streams of bytes. The core abstraction is the Read and Write traits. A File implements these traits, but so does a TcpStream, a Cursor<Vec<u8>>, and stdin. This means you can write a function that processes binary data and pass it a file, a network socket, or an in-memory buffer without changing the logic.
Think of a file as a tape deck. You can't just grab the whole tape at once if it's huge. You have a read head that moves along the tape. Read is the mechanism to pull bytes off the tape into a buffer you provide. Write is the mechanism to push bytes from a buffer onto the tape. The file keeps track of the current position. Every read advances the position. Every write advances the position.
The Read trait provides methods like read, read_exact, and read_to_end. The Write trait provides write, write_all, and flush. The names tell you exactly what they do, but the distinction between the base method and the _all/_exact variants is critical.
Minimal example
For small files that fit in memory, the simplest pattern is to read everything into a Vec and write it out.
use std::fs::File;
use std::io::{Read, Write};
fn main() -> std::io::Result<()> {
// Open file for reading. `?` propagates errors like "file not found".
let mut file = File::open("data.bin")?;
// Buffer to hold the bytes. Grows automatically as needed.
let mut buffer = Vec::new();
// Read all remaining bytes into the buffer until EOF.
file.read_to_end(&mut buffer)?;
// Create a new file, truncating any existing content.
let mut out = File::create("copy.bin")?;
// Write the entire buffer to the new file.
out.write_all(&buffer)?;
Ok(())
}
Walkthrough
File::open returns a File handle. The ? operator checks for errors. If the file doesn't exist, the function returns early with an std::io::Error. The file must be mutable because reading changes the internal file position. If you forget mut, the compiler rejects the code with E0596 (cannot borrow as mutable). read_to_end requires a mutable reference to the buffer because it pushes bytes into it.
read_to_end loops internally until it hits the end of the file. It handles partial reads for you. You don't need to worry about the OS returning fewer bytes than requested. write_all does the same for writing. It loops until every byte is written.
Never use write directly unless you have a specific reason to handle partial writes. write might return early, leaving some data unwritten. write_all guarantees the entire slice is written or returns an error.
Trust write_all. It saves you from subtle data corruption bugs where the last few bytes of a file vanish because a partial write was ignored.
Streaming large files
Loading a 4GB video file into a Vec crashes your program. For large data, you need to stream it in chunks. This is where BufReader and BufWriter become essential.
use std::fs::File;
use std::io::{BufReader, Read, Write};
fn copy_large_file(src: &str, dst: &str) -> std::io::Result<()> {
// Wrap in BufReader to batch system calls.
// Direct reads on File trigger a syscall per call, which is slow.
let mut reader = BufReader::new(File::open(src)?);
let mut writer = File::create(dst)?;
// Reusable buffer. 8KB is a common chunk size.
let mut buffer = vec![0u8; 8192];
loop {
// Read up to buffer.len() bytes. Returns 0 on EOF.
let bytes_read = reader.read(&mut buffer)?;
// 0 bytes means we reached the end of the file.
if bytes_read == 0 {
break;
}
// Write only the bytes we actually read.
// Slicing avoids writing garbage from previous iterations.
writer.write_all(&buffer[..bytes_read])?;
}
Ok(())
}
Walkthrough
BufReader sits between your code and the file. It allocates an internal buffer and reads large blocks from the OS at once. When you call read on the BufReader, it serves bytes from its internal buffer instead of hitting the OS every time. This reduces system call overhead dramatically. The convention is to always wrap File in BufReader for reading and BufWriter for writing, unless you're doing a single massive operation.
The read method returns a usize indicating how many bytes were placed in the buffer. It can return fewer bytes than the buffer size, even if more data is available. It returns 0 only when the stream is exhausted. You must slice the buffer with &buffer[..bytes_read] to write exactly what was read. Writing the whole buffer would include stale data from the previous iteration.
Buffer your I/O. Calling read or write directly on a file triggers a system call every time. Wrap your file in BufReader or BufWriter and your code will fly.
Common pitfalls
Binary I/O has a few traps that catch beginners.
Partial reads. The read method is allowed to return fewer bytes than requested. This is by design. The OS might have only that much data ready. If you're parsing a binary format with a fixed header, read is dangerous. Use read_exact instead. read_exact blocks until the buffer is full or returns an error. If the file is too short, read_exact returns an error with ErrorKind::UnexpectedEof.
Endianness. If you read raw bytes and cast them to a u32, the value depends on the machine's endianness. A file written on a little-endian machine will be interpreted incorrectly on a big-endian machine. Use conversion methods like u32::from_le_bytes or u32::from_be_bytes to enforce a specific byte order. Never rely on native endianness for file formats.
Truncation. File::create truncates the file. If the file exists, its content is wiped immediately upon opening. If you want to append to a file, use OpenOptions.
use std::fs::OpenOptions;
use std::io::Write;
fn append_data(path: &str) -> std::io::Result<()> {
// OpenOptions uses a builder pattern for flexible configuration.
let mut file = OpenOptions::new()
.create(true)
.append(true)
.open(path)?;
file.write_all(b"New data\n")?;
Ok(())
}
Type mismatches. write_all expects a slice of bytes (&[u8]). If you try to write a String, the compiler rejects you with E0308 (mismatched types). Convert the string using string.as_bytes(). This is a safety feature. It forces you to think about encoding. A String is UTF-8 text. A binary file might contain arbitrary bytes. Mixing them up leads to corruption.
Check your endianness. A binary file written on a little-endian machine is garbage on a big-endian machine without explicit conversion.
Decision matrix
Use File::open when you need to read an existing file. Use File::create when you want to write a new file and discard any previous content. Use OpenOptions when you need fine-grained control like appending to a file, creating it only if it doesn't exist, or truncating explicitly. Use BufReader and BufWriter whenever you perform multiple reads or writes; the buffering layer reduces system calls and boosts throughput significantly. Use read_to_end when the file fits comfortably in memory and you want the simplest code. Use read_exact when you're parsing a binary format with fixed-size headers and need to guarantee you got all the bytes. Use a manual read loop when processing a stream larger than RAM, chunk by chunk.