The wall of the standard iterator
You are building a text parser. The input is a massive string loaded from a file. You want to iterate over lines or tokens without allocating a new String for every piece of data. The iterator should hold the buffer and hand out slices that borrow from it. You try to implement the standard Iterator trait. You set type Item = &'a str. The compiler rejects you. The lifetime of the item cannot borrow from self. You have hit the fundamental limitation of Rust's Iterator trait.
This is the lending iterator pattern. It solves the problem where an iterator owns data and needs to yield references into that data. The standard library cannot express this constraint, so you need a custom trait or a specialized crate.
Why Iterator cannot lend
The Iterator trait defines next with this signature:
fn next(&mut self) -> Option<Self::Item>;
Self::Item is an associated type. It is a concrete type chosen by the implementor. The compiler requires Item to be independent of the specific mutable borrow of self in the function call. If Item borrowed from self, the lifetime of the returned reference would be tied to the duration of the next call. The compiler cannot express "Item borrows from self" using a plain associated type.
When you try to return &str from an iterator that owns a String, the compiler sees a conflict. The reference needs a lifetime. That lifetime must come from somewhere. If it comes from self, the reference would outlive the mutable borrow of self, which violates aliasing rules. If it comes from an external source, the iterator isn't lending; it's just traversing external data.
The standard Iterator trait assumes items are either owned by the iterator or borrowed from a source independent of the iterator's internal state. Lending iterators break that assumption.
The GAT solution
Rust solves this with Generic Associated Types (GATs). A GAT allows an associated type to take lifetime parameters. You can define a trait where Item is a type constructor that accepts a lifetime, tying the item's lifetime to the borrow of self.
The trait looks like this:
trait LendingIterator {
// Item is a type constructor that takes a lifetime 'a.
// The item can borrow from self for that lifetime.
type Item<'a> where Self: 'a;
// next borrows self mutably and returns an item
// that borrows from that mutable borrow.
fn next(&mut self) -> Option<Self::Item<'_>>;
}
The Item<'a> syntax tells the compiler that the item can hold references with lifetime 'a. The where Self: 'a bound ensures self lives long enough to support that lifetime. The next method returns Self::Item<'_>, where the underscore is an anonymous lifetime tied to the mutable borrow of self.
This signature enforces a strict rule: you cannot call next again while you still hold the item. The item borrows self mutably. Rust prevents overlapping mutable borrows. You must finish using the item before requesting the next one.
Minimal example
Here is a lending iterator that yields slices from a buffer.
/// A trait for iterators that lend references into their internal state.
trait LendingIterator {
/// The item type, parameterized by the lifetime of the borrow.
type Item<'a> where Self: 'a;
/// Returns the next item, borrowing from self.
fn next(&mut self) -> Option<Self::Item<'_>>;
}
/// Holds a string buffer and yields line slices.
struct LineLender {
buffer: String,
pos: usize,
}
impl LendingIterator for LineLender {
// The item is a string slice that borrows from self.
type Item<'a> = &'a str;
fn next(&mut self) -> Option<Self::Item<'_>> {
// Find the end of the current line.
let end = self.buffer[self.pos..]
.find('\n')
.map(|offset| self.pos + offset)
.unwrap_or(self.buffer.len());
// If we are at the end, return None.
if self.pos >= self.buffer.len() {
return None;
}
// Extract the slice. This borrows from self.buffer.
let line = &self.buffer[self.pos..end];
// Advance the position past the newline.
self.pos = end + 1;
Some(line)
}
}
fn main() {
let mut lender = LineLender {
buffer: "first\nsecond\nthird".to_string(),
pos: 0,
};
// Consume the lending iterator.
// You must use a while let loop.
while let Some(line) = lender.next() {
println!("Got: {}", line);
}
}
The while let loop is the only ergonomic way to consume this. Each iteration calls next, gets a slice, uses it, and drops the slice before the next call. The borrow checker ensures you never hold two slices at once.
Lending iterators force sequential consumption. You cannot skip ahead or store items for later.
Realistic example: Zero-copy tokenizer
Parsers are the primary use case for lending iterators. You want to tokenize input without allocating. The iterator holds the raw bytes and yields token slices.
/// A trait for lending iterators, matching the community standard.
trait LendingIterator {
type Item<'a> where Self: 'a;
fn next(&mut self) -> Option<Self::Item<'_>>;
}
/// Tokenizer that yields word slices from a buffer.
struct Tokenizer {
data: String,
cursor: usize,
}
impl LendingIterator for Tokenizer {
// Tokens are slices borrowing from the tokenizer.
type Item<'a> = &'a str;
fn next(&mut self) -> Option<Self::Item<'_>> {
// Skip whitespace.
while self.cursor < self.data.len() {
let byte = self.data.as_bytes()[self.cursor];
if byte.is_ascii_whitespace() {
self.cursor += 1;
} else {
break;
}
}
// If we hit the end, no more tokens.
if self.cursor >= self.data.len() {
return None;
}
// Find the end of the token.
let start = self.cursor;
while self.cursor < self.data.len() {
let byte = self.data.as_bytes()[self.cursor];
if byte.is_ascii_whitespace() {
break;
}
self.cursor += 1;
}
// Return the slice.
Some(&self.data[start..self.cursor])
}
}
fn main() {
let mut tok = Tokenizer {
data: " hello world rust ".to_string(),
cursor: 0,
};
// Process tokens one by one.
while let Some(token) = tok.next() {
// Use the token immediately.
// Cannot store token and call next() again.
println!("Token: '{}'", token);
}
}
This code allocates nothing during iteration. The slices point directly into the String buffer. When the tokenizer drops, the buffer drops, and all slices become invalid. The lifetimes enforce this safety automatically.
Convention aside: The community converged on the lending-iterator crate to standardize this trait. Writing your own GAT trait is educational, but in production, use the crate to get helper methods and compatibility with other libraries.
The cost: losing the adapter chain
Lending iterators come with a heavy trade-off. You lose the entire iterator adapter ecosystem.
Methods like map, filter, collect, and for_each require the iterator to be stored and called multiple times. Adapters wrap the iterator and call next on demand. If next returns a borrow, the adapter cannot store the borrow and call next again. The mutable borrow would conflict.
You cannot write lender.map(|s| s.to_uppercase()). The closure would need to return a value that outlives the borrow, which defeats the purpose. You cannot write lender.collect::<Vec<_>>(). Collecting requires storing items, but you can't store a borrow and continue iterating.
The for loop also fails. for item in lender desugars to a loop that calls Iterator::next. Your type does not implement Iterator. It implements LendingIterator. The compiler rejects the loop with E0277 (the trait bound Tokenizer: Iterator is not satisfied).
You are stuck with while let loops. This is intentional. The pattern enforces a strict processing model: pull an item, process it, drop it, pull the next. If you need adapters or collection, you probably should not be using a lending iterator.
Pitfall: Trying to store an item from a lending iterator while continuing to iterate. The compiler rejects this with E0502 (cannot borrow as mutable because it is also borrowed as immutable) or a lifetime error. The borrow of self in next overlaps with the stored item. You must finish using the item before calling next again.
If you find yourself fighting the borrow checker to store items, step back. Clone the data into owned types and use a standard Iterator. The complexity of lending iterators rarely pays off when you need random access or storage.
Decision matrix
Use Iterator<Item=&T> when the data lives outside the iterator and you just need to traverse it. The iterator borrows the data, and yields references to that external data. This supports adapters, for loops, and collection.
Use Iterator<Item=T> when you want to move values out of the collection. The iterator owns the data or consumes it. This is the standard pattern for Vec::into_iter() and owned streams.
Use a lending iterator when the iterator owns the data and you need to yield references into that data without cloning. This is common in parsers, buffers, and zero-copy deserialization where allocation must be avoided.
Use the lending-iterator crate when you want a standard trait definition and helper methods without writing GATs by hand. The crate provides the trait and ensures compatibility with other libraries that expect the same pattern.
Reach for owned iterators if performance profiling shows that cloning is not the bottleneck. Lending iterators add complexity to your API and restrict usage patterns. Only adopt them when zero-copy is strictly necessary.
Lending iterators give you zero-copy performance but take away the iterator ecosystem. Choose wisely.