The moving house problem
You are building an async TCP parser. You allocate a byte buffer, read incoming data into it, and want to keep a slice pointing to the unread portion. You try to store both the buffer and the slice inside the same struct. The compiler rejects you immediately. You suspect async is the culprit. It is not. The real problem is that you are trying to build a self-referential struct, and Rust refuses to let you move it. Async just happens to run into the exact same wall.
Self-referential structs and async futures are not the same thing. They solve different problems. But they share a single, mandatory constraint: they both struggle with memory movement. Understanding why they clash, and how Rust resolves the tension, is the key to writing safe async code without fighting the borrow checker.
What self-referential structs actually are
A self-referential struct holds a pointer to one of its own fields. The pointer and the data live in the same allocation. When you create the struct, the pointer is valid. The moment you move the struct to a new memory address, the pointer becomes stale. It still points to the old coordinates. Dereferencing it triggers undefined behavior.
Rust's ownership system assumes values can move freely. Variables get copied, passed by value, returned from functions, and shuffled across stack frames. If a type contains internal pointers, moving it breaks those pointers. The compiler blocks you from creating self-referential structs in safe Rust because it cannot guarantee the struct will stay put.
/// Demonstrates why self-referential structs break on move
struct SelfRef {
data: Vec<u8>,
/// Points into the Vec's heap allocation
slice: &'static [u8],
}
fn create_self_ref() -> SelfRef {
let mut data = vec![1, 2, 3, 4, 5];
// We want slice to point into data's buffer
// This requires unsafe because lifetimes cannot express "points to self"
let slice: &'static [u8] = unsafe {
// SAFETY: We are transmuting a local reference to 'static
// This is only valid if the struct never moves after creation
std::mem::transmute::<&[u8], &'static [u8]>(&data)
};
SelfRef { data, slice }
}
The code above compiles, but it is a time bomb. If create_self_ref returns the struct, the Vec might reallocate its buffer during the return. If the executor moves the struct to another thread, the slice points to garbage. The compiler cannot track this relationship, so safe Rust forbids it. You need unsafe, or a crate like ouroboros, to build these types. Even then, you must guarantee the struct never moves after initialization.
Trust the borrow checker here. It is protecting you from dangling pointers that only appear after a move.
How async futures work under the hood
Async functions in Rust do not run like traditional threads. They compile into state machines that implement the Future trait. Each .await expression becomes a suspension point. The compiler splits your function into discrete states, storing captured variables in a struct. When you call .await, the state machine checks if the operation is ready. If it is, it returns Poll::Ready. If not, it returns Poll::Pending and hands control back to the executor.
The executor stores the future somewhere. It might be on a thread pool queue, in a cache, or on a stack frame. When the underlying resource signals readiness, the executor wakes the future and calls poll() again. Between suspension points, the future can move anywhere. The executor does not promise stability. It promises progress.
use std::future::Future;
use std::task::{Context, Poll};
/// A minimal future that demonstrates state machine movement
struct CountFuture {
count: u32,
}
impl Future for CountFuture {
type Output = u32;
/// The executor calls this repeatedly until Ready
fn poll(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<u32> {
// SAFETY: We only access count, which does not contain internal pointers
// Pin projection is omitted here for brevity, but required in real code
let count = unsafe { self.get_unchecked_mut() }.count;
if count < 3 {
// Increment state and yield to allow other tasks to run
unsafe { self.get_unchecked_mut() }.count += 1;
Poll::Pending
} else {
Poll::Ready(count)
}
}
}
The self: Pin<&mut Self> signature is the first hint that movement matters. Pin is a wrapper that guarantees the value it holds will not be moved in memory after it is pinned. The executor relies on this guarantee. If a future contains internal pointers, those pointers must remain valid across suspension points. The future itself might move between polls, but once pinned, its address is fixed.
Convention aside: the community always pins futures with Box::pin rather than Pin::new. Pin::new requires a mutable reference to a stack-allocated value, which is impossible to return from a function. Box::pin allocates once, pins immediately, and returns a Pin<Box<T>> that can cross function boundaries safely.
Where the two collide
Self-referential structs and async futures collide on the concept of memory stability. A self-referential struct needs to stay put. An async future needs to be movable until it is pinned, and then it must stay put. If you try to put a self-referential struct inside a future, you create a paradox. The future moves during construction and suspension. The struct breaks on every move.
The compiler catches this early. If you try to store a reference to a local variable inside an async block, you get E0597 (borrowed value does not live long enough). If you try to mutate a field while holding a reference to another field in the same struct, you get E0502 (cannot borrow as mutable because it is also borrowed as immutable). These errors are not async-specific. They are lifetime errors. Async just makes them more visible because the state machine captures variables across suspension points.
/// Shows why capturing self-references in async blocks fails
async fn broken_parser() {
let mut buffer = Vec::new();
// We want to keep a slice of the buffer for parsing
let slice = &buffer[..];
// This fails because the async block captures both buffer and slice
// The state machine cannot guarantee buffer won't reallocate
let _ = tokio::task::yield_now().await;
// E0502: cannot borrow `buffer` as mutable because `slice` is borrowed
buffer.push(1);
}
The async state machine tries to capture buffer and slice into the same struct. It cannot, because slice points into buffer. Moving the captured state would invalidate slice. The compiler rejects the code before it even compiles the state machine. This is a feature, not a bug. It prevents you from shipping undefined behavior into your async runtime.
Do not fight the compiler here. Restructure the data so ownership flows one way.
Pinning: the shared escape hatch
Pin is the mechanism Rust uses to solve the movement problem for both self-referential structs and async futures. Pin does not make self-referential structs safe. It only prevents them from moving. You still need unsafe to create the internal pointers. Pin guarantees that once the value is in place, the memory address will not change.
Types that contain internal pointers must opt into !Unpin. The !Unpin marker tells the compiler and the type system that this type cannot be safely moved after pinning. Future requires !Unpin for exactly this reason. If a future contains self-referential data, it must be !Unpin. The executor will pin it, and the type system will enforce that no one moves it.
use std::pin::Pin;
use std::future::Future;
use std::task::{Context, Poll};
/// A future that owns its data and is safe to pin
struct SafeBufferFuture {
buffer: Vec<u8>,
/// We store the length instead of a slice to avoid self-references
read_pos: usize,
}
impl Future for SafeBufferFuture {
type Output = Vec<u8>;
/// Polls until the buffer is fully processed
fn poll(self: Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<Vec<u8>> {
// SAFETY: We only mutate buffer and read_pos, which are independent fields
// No internal pointers exist, so moving would not break invariants
let this = self.get_mut();
if this.read_pos < this.buffer.len() {
this.read_pos += 1;
Poll::Pending
} else {
Poll::Ready(this.buffer.clone())
}
}
}
Notice the design choice. Instead of storing a slice pointing into the buffer, we store an index. Indices are just numbers. They do not break when the struct moves. This is the standard async pattern. You track offsets, lengths, and owned data. You avoid internal pointers entirely. When you need to expose a slice to the outside world, you return it temporarily, not as a stored field.
Convention aside: the community calls this the "offset over pointer" rule for async state machines. If you can replace a reference with a usize index, do it. It makes the future Unpin by default, which means it can move freely before pinning and simplifies executor integration.
Realistic async patterns that avoid the trap
Real async code rarely needs self-referential structs. Streams, parsers, and network buffers use owned allocations combined with pinning. When you need to hold a reference to data that outlives the current scope, you use Arc or Rc. When you need mutable access across suspension points, you use RefCell or Mutex. When you need to guarantee memory stability, you use Pin.
Consider an async line reader. It reads bytes from a socket, stores them in a Vec, and yields complete lines. A naive implementation tries to store a &str slice pointing into the Vec. It fails. The correct implementation stores the Vec and tracks line boundaries with indices. When a line is ready, it extracts the bytes, returns them, and compacts the buffer. No internal pointers. No unsafe. No Pin projection headaches.
/// A realistic async line reader that avoids self-references
struct LineReader {
buffer: Vec<u8>,
/// Tracks where the next line starts
line_start: usize,
}
impl LineReader {
/// Returns the next complete line, or None if not ready
fn extract_line(&mut self) -> Option<String> {
// Find newline starting from line_start
let rest = &self.buffer[self.line_start..];
if let Some(pos) = rest.iter().position(|&b| b == b'\n') {
let end = self.line_start + pos;
let line = String::from_utf8_lossy(&self.buffer[self.line_start..end]).to_string();
// Advance past the newline and the extracted line
self.line_start = end + 1;
Some(line)
} else {
None
}
}
}
This pattern scales. You can wrap it in a Stream implementation, pin it with Box::pin, and feed it to an executor. The state machine moves freely. The indices stay valid. The buffer reallocates safely because no external pointers depend on its address. This is how production async code handles data that looks self-referential but does not need to be.
Treat indices as your first tool. Reach for pointers only when you have measured the cost and accepted the complexity.
Pitfalls and compiler errors
The borrow checker will stop you from creating self-referential async state machines. You will see these errors repeatedly:
E0502 appears when you try to mutate a buffer while holding a slice of it. The compiler refuses to let you capture both a mutable and immutable reference in the same async block.
E0507 appears when you try to move a field out of a struct that is borrowed. Async blocks capture variables by reference or move. If you try to extract owned data from a borrowed struct inside a future, the compiler blocks it.
E0595 appears when you assign to a variable that is currently borrowed. This happens when you try to update a buffer after creating a slice that points into it.
E0277 appears when you try to pin a type that does not implement Unpin correctly, or when you pass a non-Send future to a multi-threaded executor.
These errors are not arbitrary. They map directly to memory safety rules. The compiler is telling you that your data layout cannot survive suspension and movement. The fix is almost always the same: restructure the data. Use indices instead of slices. Use owned types instead of references. Use Arc instead of shared pointers. Use Pin only when you must guarantee stability, and never assume it fixes dangling pointers.
Counter-intuitive but true: the more you lean on Pin and unsafe, the harder your async code becomes to debug. Keep state machines simple.
Decision matrix
Use owned buffers with index tracking when you need to parse or stream data across suspension points. Use Box::pin when you must return a future from a function or store it in a collection. Use Arc<T> when multiple async tasks need to share read-only state without cloning. Use Arc<Mutex<T>> or Arc<RwLock<T>> when multiple tasks need to mutate shared state safely across threads. Use Pin and !Unpin when you are implementing a custom future or stream that contains internal pointers. Use crates like ouroboros or self_cell when you absolutely must build self-referential structs in safe Rust. Reach for plain references when lifetimes are simple and the data outlives the async block.