Teaching the cluster your language
You deploy a database to Kubernetes. It works. Then the disk fills up. You need a backup. You write a script. Then you need to restore from that backup. You write another script. Then you want to scale the read replicas based on load. Now you have a tangle of cron jobs, sidecars, and manual kubectl commands that no one wants to touch. Kubernetes manages Pods and Services, but it does not know what a "Database" is. It does not know that a database needs backups, or that a "HighAvailability" mode means three replicas with a load balancer. You need to teach the cluster your domain language. That is what an operator does.
An operator is a controller that understands a Custom Resource Definition. You define a new type of resource, like MyDatabase, with fields for replicas, backup schedule, and storage size. You write Rust code that watches for MyDatabase objects. When someone creates one, your code reads the desired state and manipulates standard Kubernetes resources to match it. If a pod dies, the operator notices and replaces it. If you change the backup schedule, the operator updates the cron job. The operator runs a reconciliation loop: observe the world, compare it to the desired state, act to close the gap.
Write the operator. Teach the cluster your domain.
The reconciliation loop
Think of an operator as a specialized manager. Kubernetes is the general foreman who knows how to spin up containers and wire networks. Your operator is the specialist who knows exactly how to configure your application. You tell the operator, "I want a database with five replicas and daily backups." The operator watches that request, checks the current state, and makes the changes needed to match your wish.
The core mechanism is the reconciliation loop. The loop runs continuously. It fetches the current state of your custom resource. It calculates what the world should look like. It applies the changes. It updates the status to reflect reality. Then it waits for the next event. The loop must be idempotent. Running it twice with the same input must produce the same result. This property makes the operator robust against failures and retries.
The loop is the heartbeat. Keep it idempotent.
Defining the Custom Resource
You start by defining the shape of your resource. kube-rs provides a derive macro that generates the Rust types from attributes. You write the spec and status structs, and the macro builds the full Kubernetes type, including metadata and the API group structure.
Add kube with the runtime and derive features to your Cargo.toml. You also need tokio for the async runtime.
[dependencies]
kube = { version = "0.90", features = ["runtime", "derive"] }
k8s-openapi = { version = "0.22", features = ["v1_28"] }
tokio = { version = "1", features = ["full"] }
serde = { version = "1", features = ["derive"] }
anyhow = "1"
Define the spec and status. The spec holds the desired state. The status holds the observed state. The CustomResource derive macro ties them together.
use kube::CustomResource;
use serde::{Deserialize, Serialize};
/// The desired state of the MyApp resource.
#[derive(CustomResource, Deserialize, Serialize, Clone, Debug)]
#[kube(group = "example.com", version = "v1", kind = "MyApp", plural = "myapps")]
#[kube(status = "MyAppStatus")]
pub struct MyAppSpec {
/// Number of replicas to run.
pub replicas: i32,
}
/// The observed state of the MyApp resource.
#[derive(Deserialize, Serialize, Clone, Debug)]
pub struct MyAppStatus {
/// Number of replicas currently ready.
pub ready_replicas: i32,
}
The #[kube(...)] attributes map to the CRD metadata. The group, version, and kind must match the YAML you apply to the cluster. The plural attribute defines the URL path for the API endpoint. The #[kube(status = "...")] attribute links the status struct.
Convention aside: The community prefers #[kube(status = "...")] over embedding the status in the spec. Separating spec and status prevents accidental updates to observed state and keeps the API clean.
Trust the derive macro. It generates the boilerplate so you can focus on the reconciliation logic.
Wiring the controller
The controller sets up the watch and calls your reconciliation function. kube-runtime::Controller handles the watch loop, event filtering, and concurrency control. You provide the API object and the reconciliation closure.
use kube::{Api, Client, ResourceExt};
use kube_runtime::controller::{Action, Controller};
use k8s_openapi::api::apps::v1::Deployment;
/// Reconcile a MyApp resource.
async fn reconcile(app: MyApp, ctx: Context) -> Result<Action, Error> {
// Logic goes here
Ok(Action::await_change())
}
#[tokio::main]
async fn main() -> anyhow::Result<()> {
// Client::try_default reads KUBECONFIG or in-cluster config.
let client = Client::try_default().await?;
let apps: Api<MyApp> = Api::default_namespaced(client.clone());
let deployments: Api<Deployment> = Api::default_namespaced(client.clone());
// Controller watches MyApp and calls reconcile on events.
let controller = Controller::new(apps, kube_runtime::config::Config::default())
.shutdown_on_signal()
.run(
reconcile,
kube_runtime::config::Config::default(),
)
.await;
controller.await?;
Ok(())
}
The Controller::new call creates a watcher for MyApp. When an event occurs, the controller calls reconcile. The shutdown_on_signal method ensures the controller stops gracefully on Ctrl+C. The run method starts the loop.
The Context struct carries shared data to the reconciliation function. You can store the Deployment API client in the context to avoid recreating it on every call.
Convention aside: Store long-lived objects in Context. The reconciliation function runs frequently. Creating API clients inside the loop adds overhead. Build the context once and pass it through.
Configure the shutdown. Operators run forever until told to stop.
Realistic reconciliation with status and finalizers
A real operator creates resources, updates status, and handles cleanup. You use Patch::Apply for server-side apply, which is the standard for operators. You update the status via the status subresource. You use finalizers to ensure cleanup happens before deletion.
use kube::api::{Patch, PatchParams};
use kube_runtime::controller::Context;
use std::sync::Arc;
/// Shared context for the controller.
#[derive(Clone)]
struct MyContext {
client: Client,
}
/// Reconcile a MyApp resource.
async fn reconcile(app: MyApp, ctx: Context) -> Result<Action, Error> {
let client = ctx.client;
let name = app.name_any();
let ns = app.namespace().unwrap();
let dep_name = format!("app-{}", name);
let dep_api: Api<Deployment> = Api::namespaced(client.clone(), &ns);
// Check for deletion. If the resource is being deleted, run cleanup.
if let Some(ts) = app.metadata.deletion_timestamp {
// Remove finalizer after cleanup.
// This signals Kubernetes to proceed with deletion.
let mut app = app;
app.metadata.finalizers = Some(vec![]);
let status_api: Api<MyApp> = Api::namespaced(client.clone(), &ns);
status_api.replace_status(&name, &Default::default(), &app).await?;
return Ok(Action::await_change());
}
// Ensure finalizer is present.
let mut app = app;
app.metadata.finalizers = Some(vec!["myapp.example.com".to_string()]);
let status_api: Api<MyApp> = Api::namespaced(client.clone(), &ns);
status_api.replace_status(&name, &Default::default(), &app).await?;
// Create or update the Deployment.
let dep = Deployment::new(&dep_name, &app.spec.replicas);
let patch = Patch::Apply(&dep);
let pps = PatchParams::apply("my-operator");
dep_api.patch(&dep_name, &pps, &patch).await?;
// Update status with observed state.
let mut app = app;
app.status = Some(MyAppStatus { ready_replicas: app.spec.replicas });
status_api.replace_status(&name, &Default::default(), &app).await?;
Ok(Action::await_change())
}
/// Helper to build a Deployment from spec.
fn Deployment::new(name: &str, replicas: &i32) -> Deployment {
// Construct k8s_openapi::api::apps::v1::Deployment
// This is verbose in real code. Use a builder or helper.
unimplemented!()
}
The reconciliation function checks the deletion timestamp. If present, the resource is being deleted. The operator runs cleanup logic and removes the finalizer. Kubernetes deletes the object only after the finalizer is gone. This prevents resource leaks.
The function uses Patch::Apply to create or update the Deployment. Server-side apply handles field ownership and conflict resolution. It is safer than Patch::Merge for operators.
The function updates the status separately. The status subresource allows updating observed state without triggering a new reconciliation loop. Mixing spec and status updates can cause feedback loops.
Convention aside: Use Patch::Apply for operator logic. It is the recommended approach in the Kubernetes API machinery. It avoids race conditions and makes field ownership explicit.
Update status separately. Spec is the wish; status is the reality.
Pitfalls and compiler traps
Operators introduce runtime complexity. The compiler catches type errors, but it cannot catch logic errors in the reconciliation loop.
If you forget Deserialize on your status struct, the compiler rejects you with E0277 (trait bound not satisfied). The derive macro needs serde to turn JSON into Rust types. Add Deserialize and Serialize to all structs that cross the API boundary.
If you mismatch types, the compiler rejects you with E0308 (mismatched types). The k8s-openapi types are strict. A String in Rust must match a string in the CRD schema. Check the schema carefully.
The status update trap is common. If you update the status and the controller watches status changes, you trigger a new reconciliation. This creates a feedback loop. Configure the controller to ignore status events, or use kube-runtime's status watcher configuration. The loop will spin until the API server rate limits you.
RBAC errors are runtime failures. The operator runs as a ServiceAccount. If you do not grant permissions, the API server returns 403. The error will not be a Rust compile error. Check the logs. The operator will fail to list or watch resources. Define a ClusterRole and ClusterRoleBinding that grants access to your CRD and the resources you manage.
Infinite loops happen when the reconciliation function returns an error or a requeue action without fixing the underlying issue. The controller retries. If the error persists, the operator consumes CPU and API server quota. Return Action::await_change() only when the state is consistent. Return Action::requeue(Duration) only when you are waiting for a transient condition.
Watch the logs. If the operator restarts, you have a loop. Break the cycle.
When to use what
Use kube-rs when you want type safety and idiomatic Rust. The derive macros generate types that match the API, and the compiler catches mismatches before you deploy. Use k8s-openapi directly when you need to manipulate low-level fields that the high-level wrappers hide, though this is rare. Use Patch::Apply for operator logic when you want server-side field management and conflict resolution. Use Patch::Merge only when you are patching resources that do not support server-side apply, which is uncommon in modern clusters. Use Action::await_change() when your reconciliation is complete and you want to wait for the next external event. Use Action::requeue(Duration) when you need to poll for a condition that has not met yet, like waiting for a pod to start, but keep the duration short to avoid spamming the API server. Use finalizers when you must clean up external resources before the custom resource is deleted. Use the status subresource when you need to report observed state without triggering a new reconciliation.
Pick the tool that matches the complexity. Simple resources need simple patches.