How to Write an Operating System Kernel in Rust

Write a Rust OS kernel by disabling the standard library, defining a custom entry point, and configuring a linker script for bare-metal targets.

When main doesn't exist

You've written a main function. You compile it. You try to boot it in QEMU. The emulator crashes or hangs immediately. The error isn't a runtime panic. It's a linker error complaining about missing symbols, or the CPU jumps to garbage memory. You're not writing a web server. You're writing the thing that runs before main exists. You're building a kernel.

Standard Rust assumes an operating system. It assumes libc exists to allocate memory, print to stdout, and manage threads. A kernel has no operating system underneath it. You are the operating system. You have to tell the compiler to stop expecting amenities and start working with bare metal.

The no_std foundation

The attribute #![no_std] disables the standard library. It removes std from the build. The compiler still provides core, which contains the fundamental types, traits, and macros that don't depend on an OS. #![no_main] disables the default entry point shim. Rust normally wraps main in a runtime that initializes the stack, registers panic handlers, and calls exit. That runtime doesn't exist here. You define the entry point yourself.

Think of std like a fully furnished apartment. You walk in, turn on the lights, and start living. core is the concrete shell. You have walls and a roof, but you bring your own furniture, wiring, and plumbing. #![no_std] tells the compiler: "Don't try to install the apartment amenities. I'm working in the shell."

Trust the attributes. They tell the compiler exactly what rules to drop.

Minimal kernel entry point

The entry point is the first instruction the CPU executes after the bootloader hands control to your code. On x86_64, the bootloader looks for a symbol named _start. You must provide this symbol, prevent the compiler from renaming it, and ensure it never returns.

// Disable the standard library. We don't have libc or heap yet.
#![no_std]
// Disable the default main entry point. We define our own.
#![no_main]

// Core is always available. It contains traits and types for no_std.
use core::panic::PanicInfo;

// Prevent the linker from mangling the name. The bootloader looks for _start.
#[no_mangle]
// Use C calling convention. The bootloader expects this ABI.
pub extern "C" fn _start() -> ! {
    // Infinite loop. The kernel never returns.
    loop {}
}

// Define what happens on panic. No printing, just halt.
#[panic_handler]
fn panic(_info: &PanicInfo) -> ! {
    // Panic freezes the machine. The kernel is in an unrecoverable state.
    loop {}
}

The #[no_mangle] attribute prevents the compiler from renaming the function. Rust mangles names to encode type information. The bootloader, however, looks for a specific symbol name. If the compiler mangles _start to _ZN5core5_start17h...E, the bootloader finds nothing and the CPU halts. extern "C" sets the calling convention. Bootloaders follow the C ABI for register usage and stack alignment. -> ! marks the function as diverging. It never returns. The kernel runs forever or reboots. It never goes back to the bootloader.

The panic handler is mandatory. Without std, there is no default panic behavior. The compiler requires you to implement #[panic_handler]. In a kernel, panicking usually means the system is in an unrecoverable state. The handler loops forever, effectively freezing the machine. You can add code here to print a message to a serial port or halt the CPU, but the loop is the safe baseline.

Configuring the build

The Cargo.toml needs specific settings to produce a kernel binary. The default settings assume an executable linked against std. You need to switch to a static library and configure panic behavior.

[package]
name = "my-kernel"
version = "0.1.0"
edition = "2021"

[dependencies]
# Usually you'll add volatile, x86_64, etc. later.
# For now, empty is fine.

[profile.dev]
# Panic by aborting. Unwinding requires stack inspection we don't have.
panic = "abort"
# Produce a static library, not an executable.
crate-type = ["staticlib"]

[profile.release]
panic = "abort"
crate-type = ["staticlib"]

Convention: Kernels almost always use panic = "abort". Stack unwinding requires complex runtime support that is too heavy for a minimal kernel. abort just stops. It's simpler and safer for early development. The crate-type = "staticlib" setting prevents Cargo from wrapping the output in an executable format. The linker produces a raw object file that you control.

The linker script

The linker script defines the memory layout. It tells the linker where to place code, data, and the stack. Without a linker script, the linker uses defaults that assume an OS loader will fix up addresses. In bare metal, you control the addresses.

ENTRY(_start)

SECTIONS {
    /* Kernel loads at 1MB. The first 1MB is reserved for Multiboot/BIOS. */
    . = 1M;

    /* Code section. Contains the _start function and all executable code. */
    .text : {
        *(.text._start)
        *(.text*)
    }

    /* Read-only data. Constants and string literals. */
    .rodata : {
        *(.rodata*)
    }

    /* Initialized data. Variables with explicit initial values. */
    .data : {
        *(.data*)
    }

    /* Zero-initialized data. Takes no space in the binary. */
    .bss : {
        *(.bss*)
    }
}

Convention: x86_64 kernels load at 1MB. The first 1MB is reserved for the Multiboot header and BIOS data structures. Loading at 1MB avoids conflicts. The .bss section is special. It holds variables that are zero-initialized. The linker allocates memory for .bss but writes zero bytes to the binary file. The bootloader or kernel zeroes this memory at runtime. This saves space on disk.

The linker script is the map. If the map is wrong, the kernel lands in the ocean.

Building and converting

Build the kernel with the x86_64-unknown-none target. The none in the target triple tells the compiler there is no operating system.

cargo build --target x86_64-unknown-none

The output is an ELF file in target/x86_64-unknown-none/debug/my-kernel. Bootloaders often need a raw binary image without ELF headers. Use objcopy to convert the ELF to a flat binary.

objcopy -O binary target/x86_64-unknown-none/debug/my-kernel kernel.bin

Convention: Use cargo objcopy from the cargo-binutils crate. It integrates with the build system and handles target detection automatically. You can run cargo objcopy --release -- -O binary to produce the binary in one step. The binary is the payload. The ELF is just the shipping container.

Pitfalls and compiler errors

If you forget #![no_std], the compiler tries to link std and fails with E0463 (can't find crate for std) because the target doesn't provide it. If you forget #![no_main], you get E0601 (main function not found in crate my-kernel) because the compiler expects a main but you provided _start.

Forgetting #[no_mangle] causes a linker error. The linker looks for _start, finds _ZN6my_kernel5_start17h...E, and fails to resolve the symbol. The error message mentions an undefined reference to _start.

Using extern "Rust" for the entry point breaks the ABI. The bootloader sets up registers according to the C calling convention. If the function expects Rust's ABI, the arguments and return values land in the wrong registers. The behavior is unpredictable.

The bootloader might not set up a stack pointer. If you call any Rust function that uses the stack before setting up a valid stack, the CPU writes to invalid memory and faults. Convention: Define a static buffer for the stack and set rsp to the end of it before calling any Rust code. The x86_64 crate provides helpers for this.

The linker doesn't care about Rust's type system. It only cares about symbol names and memory addresses. If the names don't match, the build dies.

When to use what

Use #![no_std] when you are writing a kernel, embedded firmware, or any code that runs without an operating system to provide system calls.

Use #![no_main] when you need to define a custom entry point like _start or _reset because the default main shim relies on runtime initialization that doesn't exist in bare metal.

Use #[no_mangle] on your entry point function so the linker can find the symbol by its exact name, matching what the bootloader or hardware expects.

Use extern "C" for the entry point ABI because bootloaders and hardware interrupts follow the C calling convention, not Rust's internal ABI.

Use panic = "abort" in your Cargo.toml profiles because stack unwinding requires complex runtime support that is too heavy for a minimal kernel.

Use objcopy to convert the ELF output to a raw binary image when the bootloader expects a flat binary format without headers or metadata.

Reach for core::panic::PanicInfo when implementing a panic handler, as std::panic is unavailable and you must provide the trait implementation yourself.

Reach for the x86_64 crate when you need to access hardware registers, set up the stack, or handle interrupts, as it provides safe abstractions over assembly instructions.

Where to go next