Kubernetes¶

Control plane, kubelet/CRI, controllers, networking, day-2.

Kubernetes Platform Engineering — A 24-Week Mastery Roadmap¶

Authoring lens: Principal Platform Engineer / Kubernetes Maintainer. Target outcome: A graduate of this curriculum is capable of (a) building and operating a hardened Kubernetes cluster from scratch on bare metal or any cloud, (b) extending the control plane via custom controllers/operators built with controller-runtime or client-go, and (c) contributing patches to kubernetes/kubernetes or one of its core ecosystems (Cilium, Istio, ArgoCD, Crossplane).

This is not "kubectl in 24 weeks." It assumes the reader has used Kubernetes (deployed an app, run kubectl), understands containers (see the CONTAINER_INTERNALS_PLAN curriculum if not), and is ready to read kubernetes/kubernetes source as primary literature.

What you'll learn¶

A 24-week curriculum organized into six monthly modules, plus three reference appendices and a capstone catalog at the end.

How Each Week Is Structured¶

1. Conceptual Core-the why, with a mental model.
2. Mechanical Detail-the how, with kubernetes/kubernetes source pointers.
3. Lab-a hands-on exercise using a real cluster (kind, k3s, or hard-way).
4. Hardening Drill-a security/compliance micro-task.
5. Operations Slice-a Day-2-ops micro-task: monitoring, scaling, recovery.

Each week is sized for ~12–16 focused hours. Almost every lab requires a working cluster-invest early in a smooth local cluster setup (kind or k3d for dev; a 3-node kubeadm cluster on cloud VMs for realistic ops).

Progression Strategy¶

Control Plane ──► Kubelet & CRI ──► Controllers & Operators
      │                │                    │
      └────────┬───────┴────────────────────┘
               ▼
   Networking & Storage
               │
               ▼
   Platform & Day-2 Ops
               │
               ▼
   Hard Way & Capstone

Prerequisites¶

• Container fluency (the CONTAINER_INTERNALS_PLAN weeks 1–3 minimum).
• Linux fluency (the LINUX curriculum weeks 9–10 minimum: namespaces & cgroups).
• Comfortable with at least one of Go, Python, or Rust at a "I can build a small CLI" level.
• A budget for cloud VMs OR hardware to run a multi-node cluster (3 small VMs is sufficient).

Capstone Tracks (pick one in Month 6)¶

1. Hard Way Track-provision a multi-node Kubernetes cluster from scratch on bare metal or cloud, with mTLS, fine-grained RBAC, multi-tenancy, and a documented runbook.
2. Platform Track-build a GitOps-driven platform-as-a-service: ArgoCD/Flux + Crossplane + OPA Gatekeeper + multi-tenancy + self-service. Demonstrate onboarding a new team in <30 minutes.
3. Operator Track-build a non-trivial operator from scratch (e.g., a stateful database operator with backup/restore, or an operator that manages an external SaaS resource via Crossplane composition). Production quality.

Detailed track briefs are in the capstone catalog.

Get started¶

Ready to begin? Start with the Prelude — philosophy, the mental model, and the reading list — then work Month 1 forward. The labs are the unit of mastery: do them.

Prelude-What Kubernetes Actually Is¶

Sit with this document for an evening before week 1.

1. Kubernetes Is a Distributed Reconciliation Loop¶

The most clarifying way to understand Kubernetes:

Kubernetes is a distributed key-value store (etcd) wrapped in an HTTP API server, surrounded by a swarm of independent controllers, each of which watches some types of objects in the store and writes other types of objects in response-until the cluster's actual state matches the desired state.

That's it. No central brain. No orchestration engine in the traditional sense. Every component is a client of the API server. The "control plane" is an emergent property of independent controllers cooperating through a shared, transactional store.

If you internalize this, the rest of the curriculum is bookkeeping.

2. The Control Loop Is the Atom¶

Every interesting behavior in Kubernetes is some controller running this loop:

for {
  desired := apiServer.List(myWatchedKind)
  actual := observe(realWorld)
  diff := compute(desired, actual)
  for _, action := range diff {
    actOn(action)              // create / update / delete real resources
    apiServer.UpdateStatus(...)
  }
  watchOrSleep()
}

The Deployment controller watches Deployment objects, creates/updates ReplicaSet objects. The ReplicaSet controller watches ReplicaSet objects, creates/updates Pod objects. The kubelet watches Pod objects bound to its node, talks to the CRI to actually start containers. Each controller's only knowledge of the others is the objects they share.

This is also why Kubernetes is eventually consistent-there is no central scheduler enforcing global state, just many controllers converging.

3. The Five-Axis Cost Model¶

A working platform engineer reasons along five axes:

Beginner courses teach axis 2 only.

4. The Reading List¶

Primary - Kubernetes in Action (Lukša, 2e). The single best book. - Programming Kubernetes (Hausenblas & Schimanski). Required for Months 3–4. - Production Kubernetes (Vyas, et al.). The Day-2 bible. - Cloud Native Patterns (Davis). Architectural patterns.

Source - kubernetes/kubernetes - the monorepo. Particularly: -cmd/kube-apiserver,pkg/apiserver/,staging/src/k8s.io/apiserver/ - API server. - cmd/kube-scheduler, pkg/scheduler/ - scheduler. -cmd/kube-controller-manager,pkg/controller/ - built-in controllers. - pkg/kubelet/ - node agent. -pkg/proxy/ - service implementations. - kubernetes/community/sig-* - design docs. - KEPs (Kubernetes Enhancement Proposals) atkubernetes/enhancements`. The canonical record of why features exist.

Adjacent canon - Designing Data-Intensive Applications (Kleppmann). Especially chapters on consensus and replication. - The Raft paper. Read in week 1. - Site Reliability Engineering (Google). The "what does Day-2 mean?" book.

5. Curriculum Philosophy¶

1. Source first, blog second. When the curriculum says "study informer mechanics," open staging/src/k8s.io/client-go/tools/cache/. Blogs go stale; commits are dated.
2. Run a real cluster. Many labs assume a multi-node setup. kind is fine for development; weeks 17+ assume something closer to production.
3. Defaults are wrong. Kubernetes ships with permissive defaults to ease onboarding (no NetworkPolicy, no PodSecurity, broad RBAC). Production requires inverting them.

6. What Kubernetes Is Not For¶

A graduate of this curriculum should be able to argue these points:

• Single-server simple deploys. A Postgres + an app on one VM with systemd is operationally simpler than a one-node Kubernetes cluster. Don't add a control plane to host one app.
• Hard real-time / latency-critical hot paths. kube-proxy adds latency. CNI plugins add latency. The scheduler is not designed for sub-millisecond placement decisions. Use bare-metal or VM-based deployments for ultra-low-latency workloads.
• Stateful databases at scale, naively. Kubernetes can run stateful workloads with operators (Postgres operator, MongoDB operator, etc.), but doing it correctly requires a mature operator ecosystem and skilled operators. "Just put your DB in K8s" is not free.
• Teams without ops capacity. Kubernetes is not a Heroku replacement. The complexity is real. If you don't have a platform team, use Cloud Run, Fly, or a managed container service before reaching for K8s.

7. AI-Assisted Workflows¶

• Always read generated YAML. Models hallucinate field names; Kubernetes silently ignores unknown fields by default-your "successful apply" may be doing nothing.
• Verify CRD generation. Tools like controller-gen are deterministic; let them generate, never hand-edit.
• Treat generated RBAC with extreme suspicion. Models tend toward over-broad permissions ("just give it cluster-admin"). Tighten by hand.

You are now ready for Week 1. Open the Control Plane month.

Month 1-The Control Plane: etcd, kube-apiserver, scheduler, controllers¶

Goal: by the end of week 4 you can (a) describe etcd's Raft model and replay a leader election, (b) trace an apply through the API-server's request pipeline (auth → admission → validation → storage), (c) explain how the scheduler picks a node, and (d) read the source of one built-in controller (Deployment) end-to-end.

Weeks¶

• Week 1 - etcd and the Raft Consensus Foundation
• Week 2 - The kube-apiserver
• Week 3 - The Scheduler
• Week 4 - Built-in Controllers and client-go Foundations

Week 1 - etcd and the Raft Consensus Foundation¶

1.1 Conceptual Core¶

• etcd is the persistent, consistent state store for everything in Kubernetes. The API server is its only client; every other component reads via the API server, never etcd directly.
• Raft (the consensus protocol) gives etcd: linearizable writes, fault-tolerant majority reads, and bounded recovery time after node failure.
• A Kubernetes cluster's reliability is bounded by its etcd cluster's reliability. Run 3 or 5 etcd nodes (never even numbers); back them up; monitor them.

1.2 Mechanical Detail¶

• Read the Raft paper. Then read etcd-io/raft/raft.go end to end (~3000 lines). The paper takes 90 minutes; the source another 4 hours; together they're worth a year of intuition.
• etcd's data model: a flat keyspace, mvcc revisions, watch streams. Kubernetes uses keys like /registry/pods/<namespace>/<name> and stores protobuf-encoded objects.
• Watch streams are the foundation of every Kubernetes controller. The API server multiplexes per-resource watches over a single etcd watch stream.
• Performance characteristics:
• Write latency = network RTT × 2 (leader → quorum) + fsync.
• Read latency = local read on leader (or any member with serializable=true).
• Throughput is bound by the leader's fsync rate. SSDs help dramatically.
• Compaction and defragmentation are required ops; without them etcd grows unbounded.

1.3 Lab-"etcd, Up Close"¶

1. Bring up a 3-node etcd cluster locally (etcd binaries, no Kubernetes yet). Configure peer/client URLs.
2. Use etcdctl to put/get keys; observe consistent reads.
3. Kill the leader. Use etcdctl endpoint status --cluster to identify the new leader within seconds.
4. Use etcdctl watch /foo from one terminal; put values from another. Internalize the watch model.
5. Use etcdctl --command-timeout=60s defrag to compact + defragment. Observe disk-usage drop.

1.4 Hardening Drill¶

• Configure mTLS between etcd peers and between client and etcd. Configure auth (role, user).
• Take a snapshot via etcdctl snapshot save. Restore to a new cluster. Verify integrity.

1.5 Operations Slice¶

• Wire etcd metrics to Prometheus: etcd_server_has_leader, etcd_disk_wal_fsync_duration_seconds, etcd_mvcc_db_total_size_in_bytes. Alert on absent leader, fsync p99 > 100ms, db-size approaching quota-backend-bytes.

Week 2 - The kube-apiserver¶

2.1 Conceptual Core¶

• The API server is the only stateful component (well-the only stateless component that talks to etcd). It exposes the REST/JSON+YAML+protobuf API, performs authn/authz/admission, and writes to etcd.
• Every Kubernetes operation-kubectl apply, controller reconciliation, kubelet status update-is an HTTP request to this server.
• Three middleware stages every request traverses: Authentication (who are you?), Authorization (RBAC: what can you do?), Admission (mutating + validating webhooks).

2.2 Mechanical Detail¶

• Authentication mechanisms: x509 client certs, bearer tokens (ServiceAccount tokens, OIDC), webhook tokens. Each is a request handler chain entry.
• Authorization: RBAC is the dominant mode. ABAC and webhook authz exist but are rare. RBAC binds subjects (User, Group, ServiceAccount) to roles (verb + resource + namespace combinations) via RoleBinding/ClusterRoleBinding.
• Admission:
• Mutating webhooks: can modify the object before validation (e.g., inject sidecars).
• Validating webhooks: can only accept or reject.
• Built-in admission controllers: LimitRanger, ResourceQuota, ServiceAccount, PodSecurity, NodeRestriction, etc. Read plugin/pkg/admission/ in the k8s tree.
• Aggregated API server: third parties can register their own API surface (e.g., metrics-server, Knative). The main apiserver proxies requests to them.
• Storage: every object has a "storage version" in etcd. The server converts between API versions on read/write. This is what allows v1beta1 → v1 migrations.

2.3 Lab-"Read the Pipeline"¶

1. Use kubectl --v=8 to dump the wire-level request/response of a kubectl apply. Read it carefully.
2. Use kubectl get --raw to hit /apis/, /api/v1, /apis/apps/v1 and see the discovery surface.
3. Configure the apiserver to log all requests with - -audit-policy-file=audit.yaml`. Apply a few changes; read the audit log.
4. Write a tiny mutating webhook (in Go, using controller-runtime's webhook facilities) that adds a label to every Pod. Deploy and verify.

2.4 Hardening Drill¶

• Audit policy template: log Metadata for every request, Request for secrets/configmaps, RequestResponse for roles/rolebindings/clusterroles/clusterrolebindings. Ship logs off-cluster.

2.5 Operations Slice¶

• Wire apiserver metrics: apiserver_request_total, apiserver_request_duration_seconds, apiserver_storage_objects. Alert on per-resource latency p99 spikes.

Week 3 - The Scheduler¶

3.1 Conceptual Core¶

• The default scheduler is a single-replica controller (with leader election) that watches unscheduled Pods and binds them to Nodes. The "binding" is just a write to the Pod's spec.nodeName field.
• The scheduler's algorithm is filter then score: filter Nodes that can't host the Pod (resources, affinities, taints), score the remaining ones, pick the highest-scoring.
• The framework is plugin-based: filter plugins, score plugins, reserve plugins, pre-bind plugins, bind plugins. You can add custom plugins without forking.

3.2 Mechanical Detail¶

• Scheduler framework extension points (read pkg/scheduler/framework/types.go):
• PreFilter-short-circuit conditions.
• Filter-must return Success for the Node to be eligible.
• PostFilter-invoked when no Node passes filter (e.g., to trigger preemption).
• PreScore, Score, NormalizeScore.
• Reserve, Permit, PreBind, Bind, PostBind.
• Built-in plugins: NodeResourcesFit, NodeAffinity, PodTopologySpread, InterPodAffinity, TaintToleration, NodePorts, VolumeBinding, ImageLocality, NodeResourcesBalancedAllocation.
• Scheduling profiles: multiple "scheduler personalities" can run in one binary, each with a different plugin config. Used for batch workloads with different priorities.
• Preemption: when a high-priority Pod can't fit, the scheduler may preempt (delete) lower-priority Pods. priorityClass is the knob.

3.3 Lab-"Scheduler in Action"¶

1. Use kubectl describe on a pending Pod to see filter/score reasons.
2. Set a Node taint (kubectl taint nodes node1 key=value:NoSchedule); observe new Pods avoid it.
3. Define PriorityClasses (high, default, batch); deploy mixed-priority Pods; trigger preemption by oversaturating.
4. Write a custom scheduler plugin (a tiny score plugin) using the scheduler framework. Configure your scheduler binary; run it. Verify selection difference vs default.

3.4 Hardening Drill¶

• Set priorityClassName on system-critical Pods (CSI driver, ingress controller). Use system-cluster-critical and system-node-critical for cluster-internal pods.

3.5 Operations Slice¶

• Wire scheduler metrics: scheduler_pending_pods, scheduler_pod_scheduling_duration_seconds, scheduler_pod_scheduling_attempts. Alert on persistent pending Pods.

Week 4 - Built-in Controllers and client-go Foundations¶

4.1 Conceptual Core¶

• The kube-controller-manager is a single binary running ~30 built-in controllers. Each is a goroutine running the reconciliation loop pattern.
• The Deployment controller is the best worked example: watches Deployment objects, creates/updates ReplicaSet objects, drives rolling-update progression.
• The patterns the built-in controllers establish-informers, work queues, structured logging, leader election-are the templates you'll use when building custom controllers.

4.2 Mechanical Detail¶

• Informers (staging/src/k8s.io/client-go/tools/cache/):
• A shared in-memory cache populated by a single watch stream per resource type.
• Event handlers: OnAdd, OnUpdate, OnDelete.
• Cache provides O(1) lookup by namespace/name.
• Multiple controllers in one process share informers via SharedInformerFactory.
• Work queues (client-go/util/workqueue): rate-limited, deduplicated, item-keyed queues. Reconcile functions pull a key, list-from-cache, act, requeue on error.
• The Deployment controller flow (pkg/controller/deployment/):
• Informer detects a Deployment change.
• Reconciler computes the desired ReplicaSet count and per-RS replica counts based on strategy (rolling vs recreate).
• Creates new RS / scales old RS / scales new RS.
• Updates Deployment status with progress.

4.3 Lab-"Read the Deployment Controller"¶

1. Read pkg/controller/deployment/deployment_controller.go end-to-end (~1500 lines).
2. Trace a kubectl rollout through the source: which conditions are checked, which fields updated, what triggers the next loop iteration.
3. Reproduce a stuck-rollout scenario (deploy a bad image); observe Progressing=False after the deadline; inspect status conditions.
4. Manually scale a Deployment to 0 with kubectl scale; trace what the controller does in response.

4.4 Hardening Drill¶

• Set sensible Deployment defaults in your platform: progressDeadlineSeconds, revisionHistoryLimit, rolling-update maxSurge/maxUnavailable for production workloads.

4.5 Operations Slice¶

• Wire workqueue metrics: workqueue_adds_total, workqueue_depth, workqueue_queue_duration_seconds. Alert on persistent depth or processing latency.

Month 1 Capstone Deliverable¶

A control-plane/ workspace: 1. etcd-cluster/ - week 1's 3-node cluster + backup/restore script. 2.audit-pipeline/ - week 2's audit-log shipping + sample queries. 3. custom-scheduler-plugin/ - week 3's scheduler plugin + deployment. 4.controller-walkthrough.md - week 4's annotated tour of the Deployment controller.

Month 2-The Node: kubelet, CRI, kube-proxy, CSI, Devices¶

Goal: by the end of week 8 you can (a) describe how the kubelet maintains the desired Pod state on a node, (b) trace a service request from client to backing Pod through kube-proxy or eBPF dataplane, (c) explain CSI volume lifecycle (provision → attach → mount), and (d) write a basic device plugin or CSI driver shim.

Weeks¶

• Week 5 - Kubelet Internals
• Week 6 - CRI: kubelet ↔ Runtime
• Week 7 - kube-proxy, Services, and the Networking Dataplane
• Week 8 - CSI, Storage, and Device Plugins

Week 5 - Kubelet Internals¶

5.1 Conceptual Core¶

• The kubelet is the per-node agent. Its job: watch Pods bound to this node, drive the CRI to make the actual containers match. Plus: report node status, manage volumes, run health checks, evict on resource pressure.
• The kubelet is also a PLEG (Pod Lifecycle Event Generator) that polls the runtime to detect actual container state changes-necessary because container exits aren't always pushed events.
• Kubelet is the component most often blamed for "weird" Kubernetes behavior; understanding it is non-optional.

5.2 Mechanical Detail¶

• Read pkg/kubelet/kubelet.go. Major loops:
• `syncLoop - the main reconciliation loop.
• `PLEG - pod-lifecycle event generation.
• `volumeManager - volume mount/unmount.
• `statusManager - Pod status updates back to apiserver.
• `evictionManager - resource-pressure eviction.
• Static pods-Pods defined as YAML files on disk (/etc/kubernetes/manifests/). Kubelet runs them directly without an apiserver. How control-plane pods bootstrap themselves.
• Pod lifecycle phases: Pending → Running → Succeeded/Failed. With container states Waiting / Running / Terminated.
• Pod resource enforcement: kubelet sets cgroups based on requests/limits. With cpu-manager-policy=static, the kubelet pins exclusive CPUs to Guaranteed-class Pods. Same idea for memory-manager-policy and topology-manager-policy.
• Eviction: when a node runs low on memory, disk, PID space, the kubelet evicts Pods in priority order. Soft vs hard thresholds.

5.3 Lab-"Kubelet Forensics"¶

1. SSH to a node. journalctl -u kubelet -f and trigger a Pod creation. Watch the log.
2. crictl ps, crictl pods, `crictl inspect - operate at the CRI layer directly.
3. Place a static pod manifest; observe kubelet picking it up.
4. Trigger a memory eviction by setting low evictionHard and oversubscribing. Read the eviction event and the kubelet's decision.

5.4 Hardening Drill¶

• Set kubelet args: - -read-only-port=0, - -anonymous-auth=false, - -authorization-mode=Webhook, - -protect-kernel-defaults=true, - -make-iptables-util-chains=true, - -tls-min-version=VersionTLS12.

5.5 Operations Slice¶

• Wire kubelet metrics: kubelet_pod_start_duration_seconds, kubelet_running_pods, kubelet_volume_stats_used_bytes. Alert on slow Pod starts.

Week 6 - CRI: kubelet ↔ Runtime¶

6.1 Conceptual Core¶

• The kubelet does not run containers itself; it talks gRPC to a CRI implementation (containerd, CRI-O). The CRI provides RuntimeService (containers + sandboxes) and ImageService (pull/list/remove).
• Every Pod is a sandbox (a network namespace + the "pause" container) plus N containers sharing it.

6.2 Mechanical Detail¶

• The CRI proto: cri-api/pkg/apis/runtime/v1/api.proto. The most relevant calls: RunPodSandbox, CreateContainer, StartContainer, StopContainer, RemovePodSandbox, Exec, Attach, PortForward, PullImage.
• The pause container is a tiny binary (it just calls pause(2)); it holds open the network namespace so the application containers can come and go.
• crictl is the CLI for talking to a CRI directly. crictl ps, crictl inspect, crictl exec. Different from `kubectl - talks straight to kubelet's runtime, bypassing the API server.
• Runtime classes: RuntimeClass objects bind a name (gvisor, kata) to a runtime handler. Pods reference via spec.runtimeClassName.

6.3 Lab-"CRI Direct"¶

1. From a node, crictl pull alpine; crictl runp pod-config.json; crictl create <pod-id> ctr-config.json img-config.json; crictl start <ctr-id>. You've launched a pod-equivalent without the apiserver.
2. Compare with kubectl deploying the same: trace each CRI call in the kubelet log.
3. Configure containerd with multiple runtimes (runc + runsc); register both as RuntimeClasses; deploy Pods against each.

6.4 Hardening Drill¶

• Configure containerd to default to a non-root user, drop default capabilities, apply default seccomp. The same hardening from the Container curriculum, applied at the daemon level.

6.5 Operations Slice¶

• Monitor container_runtime_* metrics from cAdvisor (built into kubelet). Alert on container-restart rate spikes.

Week 7 - kube-proxy, Services, and the Networking Dataplane¶

7.1 Conceptual Core¶

• A Service is a stable virtual IP and port that load-balances across a set of Pods. It is implemented at L4 by kube-proxy-or, in modern eBPF-based clusters, by the CNI directly (Cilium replaces kube-proxy entirely).
• Modes:
• iptables (default): kube-proxy programs iptables DNAT rules. O(N) match per packet; degrades with many Services.
• IPVS: kube-proxy programs the kernel IPVS load balancer. O(1) lookup; better for >1000 services.
• eBPF (Cilium): bypasses iptables entirely; programs are attached at the socket layer (bpf_sock_ops) and at the egress point. Lowest overhead.

7.2 Mechanical Detail¶

• EndpointSlices replaced Endpoints in 1.21+: split per-Service endpoint lists into multiple objects to scale beyond ~1000 endpoints per Service.
• Service types: ClusterIP (default, internal), NodePort (open a port on every node), LoadBalancer (cloud LB integration), ExternalName (DNS CNAME).
• Headless Services (clusterIP: None): no virtual IP; DNS returns Pod IPs directly. Used by StatefulSets.
• Topology-aware routing: prefer endpoints in the same zone (since 1.27 stable). Saves cross-zone egress costs.
• Service IPs are virtual: no NIC has them; they live only in iptables/IPVS/eBPF rules.

7.3 Lab-"Service Path"¶

1. Create a Service + Deployment. From a Pod, curl <service>.<ns>.svc.cluster.local. Trace the DNS lookup (CoreDNS) and the iptables/IPVS rules that DNAT.
2. Switch kube-proxy to IPVS mode (mode: ipvs in kube-proxy config). Verify with ipvsadm -L -n.
3. Install Cilium with kubeProxyReplacement=true. Observe kube-proxy not running. Verify Service connectivity still works.
4. Compare per-packet latency under each mode with a small benchmark.

7.4 Hardening Drill¶

• Enable topology-aware routing to keep traffic in zone. Apply NetworkPolicies (next month) that allow only intended traffic.

7.5 Operations Slice¶

• Wire kubeproxy_sync_proxy_rules_duration_seconds. With many Services and iptables mode, this can take seconds-a known scale ceiling.

Week 8 - CSI, Storage, and Device Plugins¶

8.1 Conceptual Core¶

• CSI (Container Storage Interface) is the standard plugin interface for storage. Every cloud and many on-prem systems ship a CSI driver. Kubernetes calls the driver via gRPC.
• A CSI driver runs in two modes (or both):
• Controller plugin-provision, delete, attach, detach, snapshot. Cluster-wide.
• Node plugin-stage and publish (mount) the volume on the kubelet node.
• PVC → PV → CSI flow: user creates a PVC; the external-provisioner sidecar sees it, calls the CSI controller's CreateVolume, which creates a PV bound to the PVC. Kubelet then asks the CSI node plugin to mount.

8.2 Mechanical Detail¶

• StorageClass parameters: provisioner (CSI driver name), parameters (driver-specific), reclaimPolicy (Delete vs Retain), volumeBindingMode (Immediate vs WaitForFirstConsumer), allowVolumeExpansion.
• WaitForFirstConsumer is critical for zone-aware provisioning-wait until the Pod is scheduled to know which zone to provision in.
• Snapshots: VolumeSnapshot API; the external-snapshotter sidecar drives the CSI driver's snapshot calls.
• Device plugins: a separate gRPC API (pluginapi.proto) for exposing custom resources (GPUs, FPGAs, RDMA NICs) to Pods. NVIDIA's k8s-device-plugin is the canonical example.

8.3 Lab-"Storage Hands-On"¶

1. Install a local-path CSI driver (rancher/local-path-provisioner works for kind). Create a PVC; observe binding.
2. Take a snapshot; restore to a new PVC.
3. Author a mock device plugin that exposes 4 instances of a fake resource. Deploy a Pod requesting it; verify scheduling and resource accounting.
4. Read the CSI proto. Diagram the provision + attach + mount flow on paper.

8.4 Hardening Drill¶

• Use volumeBindingMode: WaitForFirstConsumer for all multi-zone clusters. Without it, you'll provision a volume in zone A and try to schedule its Pod in zone B.

8.5 Operations Slice¶

• Monitor csi_* metrics emitted by sidecars. Alert on provision/attach errors and slow Mount operations.

Month 2 Capstone Deliverable¶

A node-and-cri/ workspace: 1. kubelet-tour/ - week 5's annotated journal-log walkthrough. 2.cri-direct/ - week 6's crictl - based pod-launch demo. 3.dataplane-bench/ - week 7's iptables vs IPVS vs Cilium-eBPF comparison. 4. `mock-device-plugin/ - week 8's working device plugin.

Month 3-Controllers and Operators: client-go, controller-runtime, CRDs¶

Goal: by the end of week 12 you can (a) build a controller from scratch with client-go and informers, (b) build a more sophisticated controller with controller-runtime (including webhooks, finalizers, status conditions), (c) define and version CRDs idiomatically, and (d) ship a non-trivial operator that manages an external system.

Weeks¶

• Week 9 - client-go Internals and a Bare Controller
• Week 10 - controller-runtime and Kubebuilder
• Week 11 - CRDs: Schema, Versioning, Validation
• Week 12 - Operator Patterns: Finalizers, External Resources, Multi-Cluster

Week 9 - client-go Internals and a Bare Controller¶

9.1 Conceptual Core¶

• client-go is the Kubernetes Go client library-typed clients, informers, work queues, leader election, the lot.
• Building a controller "from scratch" in client-go is verbose but instructive-every other framework hides these primitives.
• The pattern (the informer + workqueue pattern):
• Create a SharedInformerFactory for the resources you watch.
• For each kind, register OnAdd/OnUpdate/OnDelete handlers that compute a key (namespace/name) and Add it to a RateLimitingQueue.
• Start N workers that pull keys from the queue and run reconcile(key).
• reconcile: list-from-cache (never call apiserver in the hot path), compute diff, apply changes, requeue on error with backoff.

9.2 Mechanical Detail¶

• The informer's resync period: re-deliver every cached object every N (default 10 minutes). Belt-and-suspenders against missed events.
• Indexers (cache.Indexer): O(1) lookup by namespace, by label, by custom key. Free with the informer.
• Listers (<group>/<version>/<resource>/lister.go in generated client code): typed accessors over the indexer.
• Leader election (tools/leaderelection): only one replica of the controller acts; others stand by. Uses a Lease resource as the lock.
• Generated clients: for built-in types, client-go ships them. For your own CRDs, generate with controller-gen or kubebuilder (week 10).

9.3 Lab-"Controller From Scratch"¶

Build a controller that watches ConfigMaps with the label mirror=true and copies them into every namespace whose name matches a configurable prefix. - Use client-go informers + workqueue directly. - Add leader election. - Idempotent: same input twice produces same result. - Handle deletions: when the source is deleted, delete all mirrors. - Run as a Deployment in the cluster.

9.4 Hardening Drill¶

• Define a minimum RBAC: only get/list/watch on configmaps and namespaces, plus create/update/delete on configmaps (constrained by namespace prefix? Use admission webhooks or namespace selectors).

9.5 Operations Slice¶

• Expose controller_runtime_* - style metrics: queue depth, work duration, error rate. Add a/healthzand/readyz. Run with/livez` probe.

Week 10 - controller-runtime and Kubebuilder¶

10.1 Conceptual Core¶

• controller-runtime is the modern, opinionated framework for controllers. Built atop client-go, it provides:
• Manager (informer factory + leader election + metrics + healthz wired together).
• Reconciler (typed reconcile method).
• Client (cached read, direct write).
• Webhook scaffolding (mutating + validating + conversion).
• Finalizers helpers.
• Kubebuilder is a CLI on top of controller-runtime that scaffolds projects from CRD definitions. The de facto starting point for new operators.

10.2 Mechanical Detail¶

• Project structure (kubebuilder init && kubebuilder create api):

  api/v1/         # CRD types (Go structs annotated for codegen)
  config/         # YAML scaffolds (CRDs, RBAC, kustomize bases)
  internal/controller/
                  # Reconciler implementations
  cmd/main.go     # Manager bootstrap
  

• The Reconcile method is the hot path; it should be idempotent and make no assumption about why it was called. Re-derive everything each call.
• controllerutil.CreateOrUpdate-the reliable upsert helper.
• Owner references-when a controller creates a child object, it sets the parent as the owner. Garbage collection handles cascading deletion.
• Finalizers-string keys on metadata.finalizers. Block deletion until the controller removes the finalizer (after performing cleanup). The pattern for cleaning up external resources before the K8s object disappears.
• Status subresource-separates spec writes from status writes; allows least-privilege RBAC.

10.3 Lab-"Rebuild Week 9 in controller-runtime"¶

Take week 9's mirror controller; rebuild with kubebuilder + controller-runtime. Compare LOC and verbosity. The framework should save substantial code.

10.4 Hardening Drill¶

• Use controller-runtime's metric and health endpoints. Configure leader election with a non-default lease duration appropriate to your environment.

10.5 Operations Slice¶

• Wire controller_runtime_reconcile_* metrics. Establish dashboards: reconcile rate, error rate, average reconcile duration per controller.

Week 11 - CRDs: Schema, Versioning, Validation¶

11.1 Conceptual Core¶

• A CRD (CustomResourceDefinition) registers a new resource kind with the apiserver. Once registered, you can kubectl get/apply it like any built-in.
• The CRD includes an OpenAPI v3 schema that the apiserver uses for validation. Get this right or you'll ship buggy custom resources.
• Multiple versions can coexist; conversion webhooks translate between them. The pattern that allows v1alpha1 → v1beta1 → v1 evolution.

11.2 Mechanical Detail¶

• Marker comments (+kubebuilder:...) on Go types generate the CRD YAML via controller-gen. Examples:
• +kubebuilder:validation:Required
• +kubebuilder:validation:MinLength=3
• +kubebuilder:validation:Enum=foo;bar;baz
• +kubebuilder:subresource:status
• +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=.status.phase``
• Status conditions: array of {type, status, reason, message, lastTransitionTime}. The standard pattern for surfacing controller state. Use the Kubernetes types directly (metav1.Condition).
• Server-side apply: with SSA, multiple controllers can own different fields of the same object via fieldManager. Replaces hand-rolled patches for many use cases.
• Conversion webhooks: invoked when apiserver needs to translate between stored and requested versions. Implement carefully-round-trip stability is essential.

11.3 Lab-"A Well-Versioned CRD"¶

1. Define a CRD with v1alpha1.
2. Add validation, defaults, status conditions, printer columns.
3. Add a v1beta1 with renamed fields and a conversion webhook between them.
4. Verify round-trip: kubectl get -o v1alpha1 then - o v1beta1` returns identical content.

11.4 Hardening Drill¶

• Validation only at the schema level is not enough. Add admission webhooks for cross-field validation (e.g., "if mode=X then field Y is required").

11.5 Operations Slice¶

• Track apiserver_storage_objects per CRD. CRDs that grow unbounded are a frequent platform failure mode.

Week 12 - Operator Patterns: Finalizers, External Resources, Multi-Cluster¶

12.1 Conceptual Core¶

• The "operator" pattern: a controller that encapsulates operational knowledge for a specific application. Examples: Postgres operator (provisions DBs, handles backups, failover), Cert-Manager (ACME-driven cert lifecycle), Prometheus operator (manages Prometheus + Alertmanager + ServiceMonitor stack).
• An operator is a controller plus one or more CRDs representing the application's domain concepts.
• Production operators handle: leader election, finalizers, status conditions, observability, RBAC, upgrades, multi-tenant isolation, external-system reconciliation, retries with backoff.

12.2 Mechanical Detail¶

• External resources (cloud APIs, SaaS): the controller's reconcile loop calls outward. Idempotency is essential-assume your reconcile may run multiple times before the external API confirms.
• Crossplane (week 19) generalizes this: every external resource is itself a Kubernetes object backed by a controller that talks to the cloud. You compose them.
• Cluster-scoped vs namespace-scoped operators: namespace-scoped is safer (lower blast radius) but limits multi-tenant operator deployment.
• Operator SDK vs Kubebuilder: largely converged today; pick whichever your team prefers. The patterns are identical.

12.3 Lab-"An Operator That Manages an External Resource"¶

Build an operator with a GitHubRepo CRD: spec includes a repo name and visibility; the controller calls the GitHub API to create/update/delete the repo to match. Includes: - Authentication via a Secret referenced by the CR. - Finalizers for cleanup. - Status conditions: Ready, Synced, Error with reasons. - Rate-limited reconciles with exponential backoff. - E2E test using a fake GitHub API server.

12.4 Hardening Drill¶

• Define an OPA/Kyverno policy: every GitHubRepo must reference a Secret in the same namespace; cross-namespace references denied. Tests for the policy.

12.5 Operations Slice¶

• Add tracing to the reconcile path; export traces via OTel. The operator's hop into GitHub appears as an external span-useful for diagnosing outages.

Month 3 Capstone Deliverable¶

A controllers-and-operators/ workspace: 1. mirror-controller-clientgo/ (week 9). 2. mirror-controller-cr/ (week 10). 3. versioned-crd/ (week 11). 4. github-repo-operator/ (week 12).

Month 4-Networking and Storage at Scale¶

Goal: by the end of week 16 you can (a) explain the CNI spec and trace a Pod-to-Pod packet through a working CNI, (b) install and operate Cilium with eBPF-based dataplane, kube-proxy replacement, and L7 visibility, (c) reason about service-mesh tradeoffs (Istio vs Linkerd vs Cilium Service Mesh), and (d) operate dynamic CSI provisioning at scale with backups and snapshots.

Weeks¶

• Week 13 - The CNI Spec and Pod Networking
• Week 14 - Cilium and eBPF Networking
• Week 15 - Service Meshes: Istio, Linkerd, Cilium Service Mesh
• Week 16 - CSI at Scale: Snapshots, Backup, Cloning

Week 13 - The CNI Spec and Pod Networking¶

13.1 Conceptual Core¶

• The CNI (Container Network Interface) spec is small (~30 pages). A CNI plugin is a binary that the kubelet (via the runtime) invokes when a Pod sandbox is created. Inputs: container ID, network namespace path, JSON config. Outputs: assigned IP, routes.
• Kubelet does not know about networking beyond "ask the CNI." This is what makes the dataplane pluggable.
• Modern CNIs ship as DaemonSets that program kernel rules (iptables, OVS, eBPF) and run a small "agent" plus a thin "delegator" CNI binary.

13.2 Mechanical Detail¶

• Read containernetworking/cni/SPEC.md. Operations: ADD, DEL, CHECK, VERSION.
• The CNI binary must be at /opt/cni/bin/<name>; config at /etc/cni/net.d/*.conf.
• The kubelet → CRI → CNI flow:
• Kubelet asks runtime to create a sandbox.
• Runtime creates a netns; calls CNI ADD.
• CNI assigns IP, sets up the netns.
• Sandbox containers join via CLONE_NEWNS=false plus the existing netns.
• CNI chains: multiple plugins composed, each running in order (e.g., a primary CNI + a metering plugin + a port-mapping plugin).

13.3 Lab-"Read a CNI's Source"¶

1. Pick a simple CNI (flannel or the reference bridge plugin from containernetworking/plugins). Read its cmdAdd end to end.
2. Deploy a small kind cluster; trace a Pod creation in the kubelet log; correlate with the CNI binary invocation.
3. Use nsenter -t <pause-pid> -n ip a to inspect the container's network namespace from the host.

13.4 Hardening Drill¶

• Default-deny NetworkPolicy per namespace. Allow only intended Pod-to-Pod and Pod-to-Service traffic.

13.5 Operations Slice¶

• Monitor CNI errors in kubelet logs. A node with consistent CNI ADD failures will have stuck pending Pods-alert on this.

Week 14 - Cilium and eBPF Networking¶

14.1 Conceptual Core¶

Cilium is the dominant eBPF-based CNI. It replaces iptables-based packet processing with eBPF programs attached at three layers:

• Socket layer (bpf_sock_ops) - connection-level decisions before packets exist.
• Cgroup egress - per-pod outbound policy enforcement.
• NIC-level XDP - ingress filtering at line rate, before the kernel network stack.

The shift from iptables matters at scale: an iptables-based kube-proxy walks a linear chain of rules per packet - O(services). eBPF programs do hash-table lookups: O(1) per packet, regardless of service count.

Beyond replacing the CNI, Cilium provides: - Kube-proxy replacement (eBPF-based service load balancing - no iptables churn on every endpoint change). - L7 NetworkPolicy (HTTP, gRPC, Kafka filtering at the dataplane, not in a sidecar). - ClusterMesh (multi-cluster service discovery and cross-cluster policy). - Hubble (eBPF-based flow observability - every pod-to-pod connection visible without sampling). - Service Mesh (sidecar-less mTLS via eBPF + SPIFFE).

This is the bridge to Linux Month 3 - eBPF in production. See also: eBPF in the observability cross-topic page.

14.2 Mechanical Detail¶

• Dataplane as eBPF graph: Cilium's eBPF programs live under bpf/ in cilium/cilium. The agent compiles them at startup with the cluster's specific configuration baked in (BTF-driven CO-RE for portability across kernels).
• Identity-based policy: pods are assigned a numeric identity derived from their labels (app=foo,env=prod → identity 1234). eBPF programs match on these identities, not on IPs. This is what allows policy to scale to thousands of pods without per-pod iptables rules - identities are stable across pod restarts and IP changes.
• Service load balancing: instead of iptables DNAT chains, Cilium uses an eBPF map indexed by (service IP, port) returning a backend. Connection state lives in a separate eBPF map; updates are atomic, no kernel reload, no race during endpoint churn.
• Encryption: WireGuard (recommended; in-kernel since 5.6) or IPsec tunnels between nodes. Per-NetworkPolicy opt-in or cluster-wide.
• Hubble captures every packet's metadata via eBPF - source/dest identity, verdict (allowed/denied), L7 protocol info - and exposes it via gRPC + a CLI + a UI. Per-packet overhead is single-digit-percent CPU.

14.3 Lab - "Install and Drive Cilium"¶

1. Install via Helm with: kubeProxyReplacement=true, hubble.enabled=true, hubble.relay.enabled=true, hubble.ui.enabled=true, encryption.enabled=true, encryption.type=wireguard.
2. Use the Hubble UI (cilium hubble ui) to visualize pod-to-pod traffic in real time.
3. Author L4 NetworkPolicy (standard k8s API); test enforcement with a denied + allowed flow.
4. Author an L7 CiliumNetworkPolicy (e.g., allow only HTTP GET /api/* from frontend → backend); test enforcement.
5. Enable Cilium Service Mesh; observe sidecar-free mTLS between two test services.

14.4 Hardening Drill¶

Enable transparent encryption (WireGuard) between nodes. Combined with default-deny NetworkPolicy (start: deny everything, allow explicitly), this gives defense-in-depth: even if a node is compromised, the attacker sees only encrypted traffic for flows they haven't been explicitly authorized to observe.

14.5 Operations Slice¶

Monitor cilium_* Prometheus metrics. Alert on: - policy-drop rate spikes - legitimate workloads being denied (usually a NetworkPolicy author mistake, or a new service that didn't get its allow rule). - identity-table pressure - Cilium has a max identity count per cluster; approaching it means too many distinct label combinations, often from a bad operator emitting unique labels per request. - endpoint regeneration time - if it climbs past 5-10s, your label churn is overwhelming the agent.

Week 15 - Service Meshes: Istio, Linkerd, Cilium Service Mesh¶

15.1 Conceptual Core¶

• A service mesh adds: mTLS between Services, retries/timeouts/circuit-breaking, traffic shifting (canary, blue/green), observability (RED metrics + traces), policy enforcement.
• Two architectural patterns:
• Sidecar (Istio classic, Linkerd)-Envoy/linkerd-proxy runs in every Pod. ~50 MB memory per Pod, ~1 ms latency overhead.
• Sidecar-less (Istio ambient, Cilium SM)-eBPF + per-node proxy. Much lower per-Pod overhead.
• Decision matrix:
• Mature, full-featured, complex → Istio.
• Minimalist, Rust-based, fast to install → Linkerd.
• Already running Cilium, want sidecar-less → Cilium Service Mesh.

15.2 Mechanical Detail¶

• Envoy (under Istio + others) is the dataplane proxy. xDS APIs (LDS, RDS, CDS, EDS) push config from the control plane.
• mTLS rotation: the mesh control plane issues short-lived certs (typically 24h) signed by an internal CA (or SPIFFE-compatible).
• Traffic management: Istio VirtualService + DestinationRule for routing rules. K8s Gateway API is the standard-track replacement, supported by all major meshes.
• Observability: every mesh emits RED metrics (Rate, Errors, Duration) per-service. With OTel, traces propagate through the mesh.

15.3 Lab-"Three Meshes"¶

1. Install Istio in ambient mode on a test cluster. Apply a VirtualService that does 90/10 canary routing. Verify with Hubble or Kiali.
2. Repeat with Linkerd. Compare install footprint, configuration ergonomics, and observability quality.
3. (If running Cilium) enable Cilium Service Mesh. Compare again.
4. Document tradeoffs: install effort, per-Pod overhead, feature gaps.

15.4 Hardening Drill¶

• Enable mTLS in STRICT mode. Define AuthorizationPolicys denying cross-namespace traffic by default; allow only intended pairs.

15.5 Operations Slice¶

• Wire the mesh's RED metrics into your service-level dashboards. Define SLOs per service: latency p99, error rate, mTLS handshake success rate.

Week 16 - CSI at Scale: Snapshots, Backup, Cloning¶

16.1 Conceptual Core¶

• Production storage in K8s requires:
• Dynamic provisioning (week 8).
• Volume Snapshots (point-in-time captures).
• Backups (off-cluster, often app-consistent via operator hooks).
• Cloning (PVC from snapshot, or PVC-from-PVC).
• Resizing (online expansion).
• Velero is the de-facto cluster backup tool: backs up resource manifests + PV snapshots to object storage; restores selectively.

16.2 Mechanical Detail¶

• VolumeSnapshotClass ↔ VolumeSnapshot ↔ VolumeSnapshotContent. Mirrors the SC/PVC/PV trio.
• The external-snapshotter sidecar runs alongside the CSI controller, watching VolumeSnapshot objects.
• Volume populators (since 1.24+)-populate a new PVC from arbitrary sources (snapshots, other PVCs, S3, etc.). Modular framework.
• Velero: install, configure storage location (S3-compatible bucket), schedule backups via Schedule resource. Plugins for cloud providers and for "BackupStorageLocation" abstraction.

16.3 Lab-"Backup and Restore"¶

1. Install Velero against a MinIO bucket.
2. Schedule a daily backup of one namespace.
3. Delete the namespace; restore from backup; verify Pods come back, PVs reattach, data intact.
4. Create a stateful workload (Postgres via an operator); test snapshot + clone flow for fast dev/test environment provisioning.

16.4 Hardening Drill¶

• Test restore into a different cluster. This is the actual disaster-recovery scenario, and the most commonly broken backup story.

16.5 Operations Slice¶

• Wire Velero metrics: backup success rate, backup duration, restore-test outcomes (run a synthetic restore weekly to validate).

Month 4 Capstone Deliverable¶

A networking-and-storage/ workspace: 1. cni-source-walkthrough.md (week 13). 2. cilium-policies/ - L4 + L7 + identity-based examples (week 14). 3.mesh-comparison/ - three meshes, RED dashboards (week 15). 4. `velero-DR/ - backup, restore, and cross-cluster-restore demos (week 16).

Month 5-Platform Engineering and Day-2 Operations¶

Goal: by the end of week 20 you can (a) operate a GitOps workflow with ArgoCD or Flux at scale, (b) provision cloud infrastructure declaratively via Crossplane (or Terraform from K8s), (c) configure HPA/VPA against custom Prometheus metrics, and (d) author and enforce policies with OPA Gatekeeper or Kyverno.

Weeks¶

• Week 17 - GitOps: ArgoCD and Flux
• Week 18 - IaC From Within K8s: Crossplane and Terraform
• Week 19 - HPA, VPA, KEDA: Autoscaling
• Week 20 - Admission Control: Webhooks, OPA Gatekeeper, Kyverno

Week 17 - GitOps: ArgoCD and Flux¶

17.1 Conceptual Core¶

• GitOps = the cluster's desired state is the contents of a git repo. A controller in the cluster watches the repo and reconciles drift.
• The two dominant tools:
• ArgoCD-UI-rich, opinionated about app structure (Application CRD), wide adoption.
• Flux-CLI/CRD-first, more composable (Kustomization, HelmRelease, GitRepository, OCIRepository), favored by CNCF-style purists.
• Both implement the same control loop: pull manifests from git → render (Kustomize/Helm) → apply → reconcile drift.

17.2 Mechanical Detail¶

• ArgoCD Application: spec.source (git path or Helm chart), spec.destination (cluster + namespace), spec.syncPolicy (manual vs automatic, prune, self-heal).
• ApplicationSet (Argo)-generate many Apps from templates; the foundation for multi-tenant fleet management.
• Flux Kustomization + HelmRelease-separate CRs for source-of-truth, transform, and apply.
• Sync waves / dependencies: both tools support ordering. Critical for "install CRDs before the resources that use them."
• Drift detection: tools auto-revert manual changes by default. Sometimes that is not what you want during incident response-know how to disable temporarily.

17.3 Lab-"Two GitOps Stacks"¶

1. Install ArgoCD. Set up an Application for a small app from a git repo. Verify auto-sync and auto-prune.
2. Install Flux. Set up the equivalent. Compare ergonomics.
3. Use ApplicationSet (Argo) to deploy the same app to three environment overlays (dev, staging, prod). Verify per-environment configuration via Kustomize overlays.

17.4 Hardening Drill¶

• ArgoCD/Flux talk to a git repo with read access. Use SSH deploy keys or fine-scoped GitHub apps; never broad PATs. Encrypt secrets at rest with sealed-secrets or sops.

17.5 Operations Slice¶

• Wire ArgoCD/Flux metrics: per-Application sync rate, drift rate, reconciliation duration. Alert on persistent OutOfSync or Failed states.

Week 18 - IaC From Within K8s: Crossplane and Terraform¶

18.1 Conceptual Core¶

• Crossplane flips IaC inside out: cloud resources are Kubernetes resources (CRDs), reconciled by Crossplane's providers (provider-aws, provider-gcp, provider-azure, provider-helm, provider-kubernetes). You manage cloud infra with kubectl apply.
• Compositions let you bundle low-level primitives into domain-specific abstractions: define a XPostgresInstance that, when applied, creates a VPC subnet, an RDS instance, IAM bindings, and a ServiceMonitor. Platform teams ship Compositions; app teams consume them.
• Terraform alternative: run Terraform Cloud / Atlantis externally; treat the cluster as a deploy target only. Simpler in some shops; doesn't unify the control plane.

18.2 Mechanical Detail¶

• Provider = a controller image that knows how to talk to one external system. Install via Provider CRD.
• ProviderConfig = credentials + connection details for the provider.
• Managed Resource (MR) = the K8s representation of a cloud resource (Bucket, Database, IAMRole).
• Composition = a YAML transform: "given this XPostgresInstance claim, produce these MRs with these field mappings."
• Composite Resource Definition (XRD) = the schema for the abstract type; the platform-team-facing equivalent of CRD.

18.3 Lab-"Self-Service Database"¶

1. Install Crossplane. Install provider-aws (or provider-gcp).
2. Configure provider credentials.
3. Define an XRD XDatabase with parameters: size, engine, version, region.
4. Define a Composition that materializes an RDS instance + a Secret with credentials.
5. As an "app team" persona, create a Database claim. Watch it become a real RDS instance. Delete; watch it be torn down.

18.4 Hardening Drill¶

• Restrict Composition selectors and compositionRef so app teams cannot select unintended Compositions. Use OPA/Gatekeeper to enforce naming, region, size limits.

18.5 Operations Slice¶

• Compositions are platform contracts. Version them. Provide migration paths. Treat as you would a public API: SLAs, deprecation windows, changelogs.

Week 19 - HPA, VPA, KEDA: Autoscaling¶

19.1 Conceptual Core¶

• HPA (Horizontal Pod Autoscaler): scales replica count based on metrics. CPU/memory by default; with metrics.k8s.io + custom.metrics.k8s.io adapters (e.g., prometheus-adapter), any metric is fair game.
• VPA (Vertical Pod Autoscaler): adjusts a Pod's CPU/memory requests based on observed usage. Two modes: Auto (recreate pod with new resources), Off/Initial (only on creation).
• KEDA (Kubernetes Event-Driven Autoscaling): scale to zero, scale on event-source backlog (Kafka lag, SQS depth, custom). Sits in front of HPA.

19.2 Mechanical Detail¶

• HPA reconcile interval: 15s by default. Picking metrics that are too jittery causes flapping; smooth at the source.
• HPA scaling policies: scaleUp.policies and scaleDown.policies with stabilization windows. Tune to workload's elasticity profile.
• Custom metrics adapter (prometheus-adapter): translates Prometheus queries into the custom.metrics.k8s.io API the HPA reads. Define rules in adapter config.
• VPA's recommender computes percentile-based recommendations from historical usage. Often used in Off mode just to suggest resource changes; production safety prefers manual approval.

19.3 Lab-"Autoscale on Custom Metrics"¶

1. Deploy a load-test target with a Prometheus-exposed requests_per_second metric.
2. Install prometheus-adapter mapping that metric to custom.metrics.k8s.io.
3. Author HPA targeting AverageValue=200 of that metric. Drive load; watch scaling.
4. Add KEDA in front for scale-to-zero behavior. Verify cold-start latency.

19.4 Hardening Drill¶

• Set minReplicas to a non-zero value for any tier-1 service (avoid cold-start during incident traffic). Cap maxReplicas to avoid runaway autoscaling on metric anomalies.

19.5 Operations Slice¶

• Wire HPA event metrics. Alert on persistent desiredReplicas == maxReplicas (you've hit the cap) and on flapping (scaleUp and scaleDown events alternating rapidly).

Week 20 - Admission Control: Webhooks, OPA Gatekeeper, Kyverno¶

20.1 Conceptual Core¶

• Admission control is the apiserver's last gate: every create/update is run through configured admission webhooks before persistence.
• Two policy-engine choices in the modern ecosystem:
• OPA Gatekeeper-Rego-language policies; the standard for "policy as code."
• Kyverno-YAML-native policies; lower learning curve, strong template/mutation/generate support.
• Pod Security Admission (replacement for the deprecated PodSecurityPolicy)-built into the apiserver. Three profiles: privileged, baseline, restricted. Apply per-namespace.

20.2 Mechanical Detail¶

• Validating webhooks: receive AdmissionReview, return Allowed=true/false with reasons. Cannot mutate.
• Mutating webhooks: also return JSON Patch / strategic merge for changes. Applied before validating.
• Failure policy (Fail vs Ignore): if the webhook is unreachable, fail closed (safer) or open (operationally simpler). Trade off carefully.
• Gatekeeper's ConstraintTemplate (Rego) + Constraint (instance) model. Audit mode reports without enforcing-start there in any new policy rollout.
• Kyverno's ClusterPolicy / Policy CRDs cover validate, mutate, generate, verifyImages.

20.3 Lab-"Three Policy Layers"¶

1. Apply Pod Security Admission per-namespace: restricted everywhere except a priv namespace.
2. Author 5 Gatekeeper Constraints: require resource limits, forbid latest tags, enforce non-root, label-required, namespace-must-have-team-label.
3. Author equivalents in Kyverno. Compare expressiveness.
4. Run in audit-mode for a week against a pre-existing cluster; triage findings before enforcing.

20.4 Hardening Drill¶

• Mandate signed images via Kyverno's verifyImages with cosign keys. Combined with Sigstore policy from the Container curriculum, this closes the supply-chain gate at the cluster.

20.5 Operations Slice¶

• Track admission-webhook latency. Slow webhooks slow every apply. Pod-creation latency p99 is your warning signal.

Month 5 Capstone Deliverable¶

A platform-and-day2/ workspace: 1. gitops-stack/ (week 17)-ArgoCD + ApplicationSet + multi-env overlays. 2. crossplane-platform/ (week 18)-XDatabase composition + claim demo. 3. hpa-custom-metrics/ (week 19)-Prom-adapter + HPA + KEDA scale-to-zero demo. 4. policy-suite/ (week 20)-Gatekeeper + Kyverno + PSA examples.

Month 6-Kubernetes The Hard Way + Capstone¶

Goal: by the end of week 24 you have built (or substantially built) a multi-node Kubernetes cluster from raw VMs, with mTLS-everywhere, fine-grained RBAC, multi-tenancy isolation, and a documented operational runbook.

Weeks¶

• Week 21 - Bootstrap: VMs, Certificates, etcd
• Week 22 - Control Plane and Worker Nodes
• Week 23 - RBAC, Multi-Tenancy, mTLS Everywhere
• Week 24 - Defense, Documentation, and the Capstone Demo

Week 21 - Bootstrap: VMs, Certificates, etcd¶

21.1 Conceptual Core¶

• "Kubernetes the Hard Way" is Kelsey Hightower's exercise: bring up a Kubernetes cluster step by step, from raw VMs, generating certs by hand, configuring every flag explicitly. The point is not operational efficiency; it is deep understanding of every moving part.
• This curriculum's hard-way variant: bring up 3 control-plane nodes + 3 worker nodes on cloud VMs (or bare metal). Use modern toolchain (containerd, Cilium, latest stable Kubernetes).

21.2 Mechanical Detail¶

• VM provisioning: 6 VMs, ~2 vCPU 4 GB each. Cloud (AWS/GCP/Hetzner) or bare metal.
• PKI: a CA + intermediate CAs for etcd, kube-apiserver, kubelet, front-proxy. Use cfssl or easy-rsa. Every component identifies itself with x509.
• etcd cluster: 3 nodes, mTLS between peers and clients, snapshots scheduled.
• Loopback bootstrap considerations: kubelet needs a kubeconfig before the apiserver is up. Either use static-pod manifests for control-plane components (the kubeadm approach) or run the control plane outside the cluster on the VMs themselves.

21.3 Lab-"Bring Up etcd"¶

1. Provision 3 VMs labeled etcd-{1,2,3}.
2. Generate CA + per-node certs.
3. Install etcd binaries; configure systemd units with mTLS.
4. Bring up; verify etcdctl member list shows healthy quorum.
5. Take a snapshot. Restore on a separate test machine.

21.4 Hardening Drill¶

• etcd encryption-at-rest is separate from the K8s secret encryption (next week). Configure etcd with an encryption-providers config from day one.

21.5 Operations Slice¶

• etcd backup automation: etcdctl snapshot save cron'd to S3 every 6 hours. Verify restore weekly.

Week 22 - Control Plane and Worker Nodes¶

22.1 Conceptual Core¶

• The control plane: kube-apiserver, kube-scheduler, kube-controller-manager. Run all three as systemd-managed binaries on each control-plane node, behind a load balancer (HAProxy or cloud LB) for HA.
• The worker plane: containerd + kubelet + kube-proxy (or Cilium replacement). Joins the cluster via a kubelet kubeconfig signed by the cluster CA.

22.2 Mechanical Detail¶

• kube-apiserver flags:
• • -etcd-servers=https://etcd-{1,2,3}:2379` with mTLS.
• • -encryption-provider-config=...` for secret encryption-at-rest.
• • -audit-policy-file=...and - -audit-log-path=....
• • -authorization-mode=Node,RBAC`.
• • -enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota,...`.
• • -service-account-issuer, - -service-account-signing-key-file for ServiceAccount tokens (projected, OIDC-compatible).
• kubelet bootstrap: TLS bootstrap using a bootstrap token; kubelet auto-rotates its cert via kubelet-csr-approver.
• CNI: install Cilium first (DaemonSet); only after Cilium is healthy do worker-node Pods become ready.
• DNS: install CoreDNS as a Deployment; the kubelet's cluster-DNS arg points at its Service IP.

22.3 Lab-"Cluster Live"¶

1. Bring up 3 control-plane nodes; HAProxy in front.
2. Bring up 3 workers; join via bootstrap tokens.
3. Install Cilium; verify Pod-to-Pod connectivity.
4. Install CoreDNS; verify Service DNS works.
5. Smoke test: deploy a sample app + Service + Ingress; verify end-to-end.

22.4 Hardening Drill¶

• Apply CIS Kubernetes Benchmark v1.8 (or current). Use kube-bench to score. Address all FAILs; document WARNs.

22.5 Operations Slice¶

• Wire control-plane components to Prometheus. Define SLOs: apiserver request p99 < 1s, etcd-leader-changes per hour < 1, scheduler queue depth < 100.

Week 23 - RBAC, Multi-Tenancy, mTLS Everywhere¶

23.1 Conceptual Core¶

• Multi-tenancy is the hardest sustained problem in Kubernetes. The kernel and Kubernetes give you soft isolation by default; converting that to hard isolation requires layered controls.
• The required layers: namespace-per-tenant + RBAC + NetworkPolicy + ResourceQuota + LimitRange + PodSecurity + node-pool isolation + (optionally) sandboxed runtime.
• mTLS everywhere: control-plane (already from week 22), service mesh between Services (week 15), workload identity for Pods talking to cloud APIs (e.g., AWS IRSA, GCP Workload Identity).

23.2 Mechanical Detail¶

• Tenant onboarding as code (Crossplane Composition or Helm chart):
• Namespace.
• ResourceQuota + LimitRange.
• Default-deny NetworkPolicy + an allow-namespace-internal exception.
• PodSecurity admission label (restricted).
• RoleBindings for the tenant's group.
• ServiceAccount with workload identity binding for cloud access.
• GitOps Application(Set) entries to deploy the tenant's app catalog.
• Hard isolation tiers:
• Tier 1: namespace + RBAC. Default. Suitable for trusted internal teams.
• Tier 2: + sandboxed runtime (gVisor) for tenant-owned untrusted workloads.
• Tier 3: + dedicated node pool with taints. Suitable for compliance-bound workloads.
• Tier 4: separate cluster (vCluster, Cluster API). Strongest isolation; highest cost.

23.3 Lab-"Onboard a Tenant"¶

1. Author a tenant Composition (Crossplane) or Helm chart that, given {tenant: "acme"}, materializes everything in §23.2.
2. Onboard acme. Have a "tenant developer" persona deploy an app via GitOps.
3. Verify isolation: from acme's namespace, can you read another tenant's secrets? Pods? Logs? Each should fail.

23.4 Hardening Drill¶

• Run kubescape or polaris against the cluster. Address findings until score is >90%.

23.5 Operations Slice¶

• Per-tenant cost attribution: label every resource with tenant=; export kube-state-metrics with that label to Prometheus; cost-allocate via OpenCost.

Week 24 - Defense, Documentation, and the Capstone Demo¶

24.1 Conceptual Core¶

The final week is integration and defense. Bring the capstone (whichever track) to production-defensible quality.

24.2 Final Hardening Checklist¶

• CIS benchmark green (kube-bench).
• All control-plane components mTLS, with cert auto-rotation tested.
• Encryption-at-rest enabled for secrets in etcd.
• Audit logging enabled; logs shipped off-cluster.
• Default-deny NetworkPolicy in every namespace.
• PodSecurity restricted everywhere except documented exceptions.
• Image admission requires signed images (Sigstore policy).
• Velero backups + tested cross-cluster restore.
• Chaos: drain a node, kill a master, partition the network-cluster recovers.
• Observability: Prometheus + Grafana + Loki + Tempo (or equivalent) integrated.
• Cost attribution per tenant.
• Runbooks: node-not-ready, etcd-degraded, apiserver-OOM, namespace-stuck-terminating, pod-pending-forever.

24.3 Lab-"Defend the Cluster"¶

Schedule a 60-minute mock review. Demo: 1. The architecture diagram. 2. Provisioning (Ansible/Terraform/Crossplane). 3. Tenant onboarding from request to running app. 4. Failure injection: kill a control-plane node; show cluster recovery. 5. Observability: trace a request from ingress through service mesh to backend, with metrics, logs, and trace ID correlation. 6. Backup + restore.

24.4 Operations Slice¶

• Tag the cluster manifest repo v1.0.0. Sign with cosign. Publish a RUNBOOK.md that, in principle, lets a successor team rebuild the cluster from scratch.

Month 6 Deliverable¶

The capstone artifact (per the capstone catalog), plus the aggregated kubernetes-mastery/ repo containing every prior month's deliverable.

Appendix A-Kubernetes Hardening Reference¶

Cumulative hardening checklist. By week 24 the reader's cluster-baseline/ template should encode every section.

A.1 Control Plane¶

• etcd: 3 or 5 nodes, mTLS, encryption-at-rest, snapshot+restore tested.
• kube-apiserver: encryption providers, audit logging, NodeRestriction admission, PodSecurity admission, OIDC (or trustedSA) for users.
• kube-scheduler: leader election; default + custom plugins reviewed.
• kube-controller-manager: leader election; minimum SA permissions.
• kubelet: read-only port disabled, TLS bootstrap with CSR approval, anonymous-auth false, authorization webhook.

A.2 RBAC¶

• No bindings to the cluster-admin ClusterRole except for break-glass.
• Per-tenant Roles, not ClusterRoles.
• Audit system:authenticated and system:unauthenticated group bindings-both should be empty.
• Use kubectl auth can-i --as=... to verify least-privilege per persona.

A.3 Pod Security¶

• PodSecurity admission restricted everywhere by default.
• Exceptions documented in code (namespace labels) with justification.
• Pod-level: runAsNonRoot, readOnlyRootFilesystem, drop all caps, seccomp RuntimeDefault.
• Mutating webhook to inject defaults if Pod spec omits them.

A.4 Network¶

• CNI with NetworkPolicy support (Cilium, Calico).
• Default-deny ingress + egress in every namespace.
• Allowed flows declared per workload as labeled NetworkPolicy.
• L7 policy on ingress (Cilium L7 NetworkPolicy or service mesh).
• mTLS between Services (mesh).
• Egress controls: explicit allowed CIDRs / FQDNs.

A.5 Image Supply Chain¶

• Image admission (Kyverno / Cosign policy-controller) requires signature.
• Allowlisted registries.
• No latest tags; pin by digest in production.
• SBOM and SLSA provenance attestations attached to every image.

A.6 Secrets¶

• etcd encryption-at-rest with rotated keys.
• External Secret Operator (ESO) for cloud-KMS-sourced secrets.
• No secrets in env vars where possible (use volume mounts, watch for restart).
• No secrets committed to git, even in sealed form, without sealed-secrets/sops ratchet.

A.7 Multi-Tenancy¶

• One namespace per tenant; ResourceQuota + LimitRange.
• Hierarchical Namespaces or Capsule for nested tenants.
• PriorityClasses by tier; preemption tuned.
• Per-tenant cost attribution via labels + OpenCost.

A.8 Observability¶

• Audit logs shipped off-cluster (read-only on cluster).
• Container logs (Loki / cloud equivalent).
• Metrics (Prometheus + kube-state-metrics + node-exporter).
• Traces (OTel Collector + Tempo / Jaeger / cloud).
• Continuous profiling (Parca / Pyroscope) optional but recommended.
• SLO tracking per service (Pyrra / Sloth).

A.9 Backup + DR¶

• Velero scheduled backups to off-cluster storage.
• Cross-region or cross-cluster restore tested at least quarterly.
• etcd snapshot tested for catastrophic-recovery scenario.
• DR runbook with RTO + RPO documented.

A.10 The cluster-baseline/ Template¶

cluster-baseline/
  bootstrap/
    pki/                    # CA + per-component certs (cfssl)
    etcd/                   # systemd unit + config
    kube-apiserver/
    kube-scheduler/
    kube-controller-manager/
    kubelet/
  cni/cilium-values.yaml
  service-mesh/             # istio or linkerd values
  observability/
    prometheus/
    grafana/
    loki/
    tempo/
    parca/
  policy/
    pod-security/
    networkpolicy-default-deny.yaml
    gatekeeper-constraints/
    kyverno-policies/
    sigstore-policy.yaml
  tenancy/
    namespace-template/      # Crossplane composition
    rbac-template/
    quotas-template/
  velero/
    schedule.yaml
    locations.yaml
  runbooks/
    node-not-ready.md
    etcd-degraded.md
    apiserver-oom.md
    pod-pending-forever.md
    cluster-rebuild.md
  RUNBOOK.md
  THREAT_MODEL.md

This is the artifact every cluster you bring up after week 24 should be provisioned from.

Appendix B-Troubleshooting Reference Flows¶

Reference flows for the failure modes you will see in production.

B.1 Pod Pending Forever¶

kubectl describe pod <pod>

Common causes (in observed-frequency order): 1. No node satisfies scheduling constraints. Events: shows FailedScheduling. Read the reason: insufficient CPU/memory, no matching nodeSelector, taints unmatched, no PV available, topology spread blocked. 2. PVC stuck pending. kubectl get pvc <pvc> - if Pending, check StorageClass, provisioner pods, cloud-side quota. 3. **Image pull failure**.Events:showsErrImagePull/ImagePullBackOff. Check registry auth, image tag exists, network egress to registry. 4. **Admission webhook rejected**. Often hidden in apiserver logs;kubectl get events -Amay surface it. 5. **Quota exceeded**.ResourceQuota` denied creation.

Drilldown: kubectl get events -A --sort-by=.lastTimestamp | tail -30.

B.2 Pod CrashLoopBackOff¶

kubectl logs <pod> --previous
kubectl describe pod <pod>

Common causes: 1. App-level crash. Read the previous container's logs. 2. Liveness probe failing. The probe is killing the container. Check probe path/port; loosen initialDelaySeconds. 3. OOMKilled. kubectl describe shows Reason: OOMKilled. Increase memory limit or fix leak. 4. ConfigMap / Secret missing. Pod is mounting it; if missing, kubelet fails the start. Watch for events. 5. Init container failure. Pod won't progress; check init container logs first.

B.3 Node NotReady¶

kubectl describe node <node>
ssh <node> sudo journalctl -u kubelet -f

Common causes: 1. kubelet down. systemd unit failure; check journal. 2. CNI agent down. The node has no functional networking; Cilium/Calico DaemonSet pod has crashed. 3. Disk pressure. Events: shows EvictionThresholdMet. Free space (delete old container images, journal logs). 4. PID pressure. Too many processes. 5. Out-of-resources kernel-side. Check dmesg on the node.

B.4 etcd Degraded¶

etcdctl endpoint status --cluster
etcdctl endpoint health --cluster

Common causes: 1. Disk full or slow. fsync latency spikes; everything else feels slow. Check etcd_disk_wal_fsync_duration_seconds. 2. Leader election thrashing. Network instability between etcd nodes; check inter-node latency. 3. Database size growth. Forgot to compact. etcdctl compact <rev>; etcdctl defrag. 4. Quorum lost. Majority of nodes down. Restore from snapshot to a new cluster; recover.

B.5 Apiserver 5xx / Timeouts¶

kubectl get --raw=/livez
kubectl get --raw=/readyz?verbose

Common causes: 1. etcd issues (above). 2. Webhook timeouts. Slow admission webhooks block every apply. Check webhook latency; consider failurePolicy: Ignore with caution. 3. Aggregated API down (e.g., metrics-server). kubectl top fails; downstream features (HPA) degrade. 4. Apiserver overload. Too many list/watch consumers; CPU pegged. Add replicas; review priority-and-fairness flow control.

B.6 Service Has No Endpoints¶

kubectl get endpoints <service>
kubectl get endpointslices -l kubernetes.io/service-name=<service>

Common causes: 1. Selector mismatch. Service spec.selector doesn't match Pod labels. Most common. 2. Pods not ready. ReadinessProbe failing; only ready Pods join Endpoints. 3. Port mismatch. Service port name vs container port name out of sync. 4. Topology-aware routing dropping endpoints. Check service.kubernetes.io/topology-aware-hints.

B.7 Namespace Stuck Terminating¶

kubectl get namespace <ns> -o json | jq .spec.finalizers

Cause: A finalizer can't be removed because its owning controller is gone (or stuck).

Fix path (carefully-you are bypassing a safety):

kubectl get namespace <ns> -o json \
  | jq '.spec.finalizers = []' \
  | kubectl replace --raw "/api/v1/namespaces/<ns>/finalize" -f -

But also: investigate why the finalizer wouldn't clear. Often a dangling external resource the operator was waiting on.

B.8 ImagePullBackOff in a Private Registry¶

1. `kubectl get secret -o yaml - exists and well-formed?
2. Pod's spec.imagePullSecrets references it?
3. Secret type is kubernetes.io/dockerconfigjson?
4. Decoded .dockerconfigjson has the right registry URL and credentials?
5. From the node, can you crictl pull the image manually with the same creds?

B.9 HPA Not Scaling¶

1. `kubectl describe hpa - events show why.
2. Metrics available? kubectl get --raw "/apis/metrics.k8s.io/v1beta1/pods" for resource metrics; kubectl get --raw "/apis/custom.metrics.k8s.io/..." for custom.
3. Pod requests set? HPA uses requests as the denominator; without them, percentage-based metrics are meaningless.
4. behavior policies preventing fast scaling? Check scaleUp.policies and stabilizationWindowSeconds.

B.10 Mesh: 503 from Sidecar¶

(Istio specifics, but general patterns apply) 1. Service has Endpoints? 2. mTLS mode strict, but caller without sidecar? Check PeerAuthentication. 3. AuthorizationPolicy denying the call? 4. DestinationRule with circuit-breaker tripped? kubectl describe destinationrule. 5. Envoy access log: istioctl proxy-config log <pod> --level debug and re-issue.

Appendix C-Contributing to Kubernetes¶

Kubernetes is the largest open-source project in the world by contributor count. The flip side: the bureaucracy is real. This appendix is the on-ramp.

C.1 Mental Model¶

The Kubernetes project is governed by SIGs (Special Interest Groups)-domain-scoped groups (sig-node, sig-network, sig-storage, sig-api-machinery, sig-scheduling, sig-cli, sig-auth, etc.). Each SIG has chairs, technical leads, regular meetings, and a Slack channel. Almost every PR maps to a SIG.

Major changes go through KEPs (Kubernetes Enhancement Proposals)-design docs in the kubernetes/enhancements repo. KEPs progress through provisional → implementable → implemented → deprecated over multiple releases.

Implications for newcomers: 1. Find the right SIG before opening a PR. 2. For non-trivial changes, write or piggyback on a KEP first. 3. The cycle time is slow. Two-week review is normal; six-week is common.

C.2 Setting Up¶

git clone https://github.com/kubernetes/kubernetes
cd kubernetes
make all -j$(nproc)

The build is heavy (it's all of Kubernetes); plan for ~10 minutes the first time.

For tests:

make test                       # unit
make test-integration KUBE_TEST_ARGS="-run <name>"
make test-e2e KUBE_TEST_ARGS=...

For local dev cluster: kind (uses your local Docker / containerd, spins up a multi-node cluster in containers in seconds).

C.3 Where the Easy Wins Are¶

C.3.1 Documentation¶

• kubernetes/website (the website repo). Docs improvements are welcome and reviewed quickly.

C.3.2 e2e flakes¶

• The flakey label on kubernetes/kubernetes issues. Fixing flakes is unglamorous but high-impact.

C.3.3 kubectl¶

• `staging/src/k8s.io/kubectl - small, contained. Bug fixes and small features are tractable.

C.3.4 client-go improvements¶

• `staging/src/k8s.io/client-go - used by every controller in the world. Improvements compound.

C.3.5 SIG-specific work¶

• Pick a SIG matching your interest. Their backlogs have good first issue labels.

C.3.6 Don't start here (yet)¶

• Scheduler core (high stakes; small SIG; deep changes need KEPs).
• API machinery (apiserver internals, conversion, validation).
• kubelet (touches every node; PR latency is high for safety).
• Anything touching scaling / performance critical paths.

C.4 The First-PR Workflow¶

1. Find an issue. Read CONTRIBUTING.md and SIG-specific contribution guides. Comment /assign to claim.
2. Branch from master.
3. Implement. Run make verify, make test, the relevant make test-integration subset.
4. Commit with Signed-off-by (DCO).
5. Open the PR. SIG bots will auto-assign reviewers. Use the PR template; fill in every field.
6. CI cycle. Tests run on Prow. Re-run with /test all. Address comments.
7. Approval flow: a reviewer adds /lgtm; an approver adds /approve. Both required. The bot then merges.
8. Backport (if applicable): for bugfixes, the PR may need cherry-picks to release branches. Use the cherry-pick robot or do manually.

C.5 The KEP Process¶

For changes that: - Add or modify the API. - Have user-visible behavior changes. - Affect multiple SIGs.

Process: 1. Open an issue in the relevant SIG. 2. Get at least informal agreement that the problem is real. 3. Write a KEP using the template in kubernetes/enhancements/keps/NNNN-template/. 4. Submit as a PR. The KEP itself goes through review. 5. Once implementable, you can submit code PRs referencing the KEP number. 6. KEPs target a Kubernetes release (alpha → beta → GA over multiple releases).

Time scale: months to a year for a substantial KEP.

C.6 Adjacent Targets if k/k Is Too Heavy¶

The CNCF ecosystem has many high-impact projects with smaller surface area:

A merged contribution to any of these is a credible Kubernetes-ecosystem signal in interviews.

C.7 Calibration¶

A reasonable goal for a curriculum graduate:

• By end of week 23: a PR open against kubernetes/website, a kubectl plugin, or a small fix to an ecosystem project.
• By end of capstone: that PR merged.
• 6 months post-curriculum: a substantive contribution-a kubectl feature, a new operator, a Cilium policy plugin.

Patient contributors become trusted contributors. Trusted contributors become reviewers. Reviewers become approvers. Approvers become SIG chairs. The path exists; it just takes time.

Capstone Projects-Three Tracks, One Choice¶

Pick one. The work performed here is what you describe in interviews.

Track 1-Hard Way: A Production-Grade Cluster From Scratch¶

Outcome: a multi-node Kubernetes cluster brought up on bare metal or cloud VMs, with mTLS-everywhere, fine-grained RBAC, multi-tenancy, GitOps-managed workloads, and a documented operational runbook.

Functional spec¶

• 3 control-plane nodes + 3 workers (minimum). HAProxy or cloud LB in front of the apiservers.
• etcd with mTLS, encryption-at-rest, scheduled snapshots to off-cluster storage.
• CNI: Cilium with kube-proxy replacement, Hubble enabled.
• Service mesh: Istio (ambient) or Linkerd, mTLS strict between services.
• Storage: a real CSI driver (local-path for dev; OpenEBS / Longhorn / cloud CSI for "real").
• Observability: Prometheus + Grafana + Loki + Tempo + OTel Collector.
• GitOps: ArgoCD or Flux managing platform addons.
• Policy: Pod Security restricted, NetworkPolicy default-deny, Kyverno or Gatekeeper enforcing org rules.
• Backup: Velero scheduled, restore tested.

Non-functional spec¶

• CIS benchmark green (kube-bench ≥90% pass).
• Cluster rebuild from scratch in <2 hours via Ansible/Terraform.
• Zero-downtime kubelet upgrades (drain + replace pattern).
• A demo: kill a control-plane node; cluster recovers without intervention.

Acceptance¶

• Public repo with provisioning playbooks and runbooks.
• A 30-minute screencast walking the assessor through bring-up, an incident drill, and a tenant onboarding.
• A RUNBOOK.md covering: cluster provisioning, node addition/removal, etcd backup/restore, certificate rotation, upgrade procedure, top 5 incident types and remediation.

Skills exercised¶

• All months-but Months 1, 2, 6 most heavily.

Track 2-Platform: GitOps Multi-Tenant PaaS¶

Outcome: a self-service developer platform built on Kubernetes that demonstrates onboarding a new team in <30 minutes, with policy guardrails, infra-from-code, and full observability.

Functional spec¶

• Tenant model: each tenant gets a Namespace, ResourceQuota, LimitRange, RBAC bindings, default NetworkPolicy, monitoring scrape config, GitOps Application (Argo) entry-all materialized from a single tenant claim (Crossplane Composition or Helm chart).
• Self-service: developers commit a manifest.yaml to their repo; ArgoCD/Flux picks it up; their app deploys.
• Policy: Kyverno or OPA Gatekeeper enforces: image signatures, no latest tags, mandatory labels, resource limits, no privileged Pods.
• Observability: each tenant's metrics/logs/traces are isolated (via labels and Loki/Prom multi-tenancy); a per-tenant Grafana folder with default dashboards.
• Cost attribution: OpenCost emits per-tenant cost; surface in a dashboard.
• Crossplane: a Database claim that materializes a real cloud database (or, for demo, a chart-deployed Postgres).

Non-functional spec¶

• Tenant onboarding: from "claim PR opened" to "deployed app reachable" in <30 minutes (target: <5).
• Failure isolation: a tenant exceeding quota does not affect other tenants.
• Compliance: every running Pod can be traced back to a git commit + signature verification.

Acceptance¶

• Public repo with platform manifests + tenant-onboarding template.
• Live demo: onboard a fresh tenant; deploy a sample app; demonstrate observability + policy denial; demonstrate quota enforcement.
• A PLATFORM.md describing the contract between platform team and tenants: versioning, deprecation, support, escalation.

Skills exercised¶

• Months 3 (operators / Crossplane), 5 (GitOps + IaC + autoscaling + admission), 6 (multi-tenancy).

Track 3-Operator: Production-Quality Operator From Scratch¶

Outcome: a non-trivial operator that manages a stateful application or external system, complete with backup/restore, upgrades, observability, and a thoughtful API.

Suggested scopes¶

1. elasticsearch-mini-operator: manage Elasticsearch clusters with auto-scaling, snapshot lifecycle, index lifecycle policies.
2. postgres-mini-operator: with automatic failover (using the Postgres replication primitives), backup/restore via WAL-G to S3, point-in-time recovery.
3. saas-resource-operator: manage external SaaS resources via Crossplane composition (e.g., a GitHubRepo operator complete with branch protection, secret scanning, codeowners).

Acceptance¶

• Public repo, written with controller-runtime + Kubebuilder.
• CRDs versioned (v1alpha1 + v1beta1 + conversion webhook).
• Status conditions, finalizers, owner references-all idiomatic.
• Comprehensive RBAC (least-privilege, generated from kubebuilder markers).
• Mutating + validating admission webhooks.
• E2E tests (Ginkgo + envtest, plus a kind-based suite).
• Helm chart or kustomize manifests for installation.
• Observability: Prometheus metrics, structured logs (logr), OTel traces.
• Helm-test-style upgrade tests across three operator versions.
• Documentation: design rationale, API reference, examples.

Skills exercised¶

• Months 3 (operators), 4 (storage if stateful), 5 (admission), 6 (defense).

Cross-Track Requirements¶

• cluster-baseline/ template (Appendix A) integrated.
• ADRs (≥3).
• THREAT_MODEL.md.
• RUNBOOK.md.
• Defense readiness: 60-minute walkthrough.

The track choice signals career direction: Track 1 for SRE/cluster-operator roles, Track 2 for platform-engineering roles, Track 3 for software-engineering-on-Kubernetes roles. Pick based on where you want the next interview loop.

Workshop - Bootstrap a Kubernetes control plane by hand¶

Companion to Kubernetes -> Month 06 -> Weeks 21-22: Bootstrap, Control Plane, and Worker Nodes ("Kubernetes the Hard Way"). Every other workshop built things on top of Kubernetes. This one goes underneath: you start etcd and the kube-apiserver as plain processes by hand, talk to your hand-built control plane with kubectl, then add the scheduler and a kubelet - and watch a Pod actually run. By the end, the single most important truth about Kubernetes is something you've proven with your own hands: the control plane is just a few Go binaries and a database.

~120 minutes - the capstone of the workshop series. Needs: a Linux host, root, and the Kubernetes binaries (etcd, kube-apiserver, kube-scheduler, kubelet, kubectl - download from the official releases). This deliberately does not use kind/kubeadm - the whole point is no automation.

What you'll build, and the idea it makes concrete¶

You'll assemble a working (single-node, no-TLS-shortcuts-where-safe) Kubernetes control plane from individual binaries: etcd for storage, the apiserver as the front door, then the scheduler and kubelet so Pods actually run. No kubeadm, no kind, no installer - you start each process and wire them together yourself.

The idea this makes concrete:

Kubernetes is not a monolithic system - it's a handful of independent programs coordinating through one shared database (etcd) and one API (the apiserver). "The control plane" is: etcd (the only stateful component - the entire cluster state lives here as key/values), kube-apiserver (the only thing that talks to etcd; everything else talks to it), kube-scheduler and kube-controller-manager (control loops that watch the apiserver and act - the loops you built in every prior workshop), and on each node a kubelet (runs containers) and kube-proxy (networking). Take away the installers and a cluster is these processes plus etcd. There is no magic - there is process supervision and a database.

Every prior workshop built a participant in this system (a controller, a scheduler, an operator). This workshop builds the system itself, revealing that your custom scheduler and the real one are peers - both just clients of the apiserver.

Step 0: the architecture you're about to assemble by hand¶

                          +------------------+
   kubectl --------------> |                  |
   your custom controller->|  kube-apiserver  | <----- the ONLY component that
   kube-scheduler -------->|  (the front door)|        reads/writes etcd
   kube-controller-mgr --->|                  |
   kubelet (each node) --->+--------+---------+
                                    | (the only etcd client)
                                    v
                              +-----------+
                              |   etcd    |   <-- ALL cluster state lives here
                              | (the DB)  |       (every object, as key/value)
                              +-----------+

The non-obvious truths this layout encodes, which you'll verify: - etcd is the only stateful thing. Every Pod, Deployment, Secret, your Website CR from the operator workshop - all of it is rows in etcd. Lose etcd, lose the cluster. Back up etcd, back up everything. - Only the apiserver touches etcd. The scheduler, controllers, kubelet - none of them know etcd exists. They all go through the apiserver's REST API. This is why the apiserver is the central component and why everything is "just an API client." - Everything else is a stateless client running a watch-and-act loop. Restart the scheduler and nothing is lost - it re-reads state from the apiserver. This is why control-plane components are easy to make HA (run several, lease-elect a leader - the controller workshop's leader election).

Step 1: start etcd - the cluster's brain¶

etcd is a distributed key/value store. For a single-node learning cluster, one instance:

$ etcd \
  --data-dir=/tmp/etcd-data \
  --listen-client-urls=http://127.0.0.1:2379 \
  --advertise-client-urls=http://127.0.0.1:2379 &

It's just a process listening on :2379. Prove it's a plain key/value store - put and get a key directly:

$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 put /hello world
OK
$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 get /hello
/hello
world

That's the database the entire cluster will live in. Right now it holds your test key; in a minute it'll hold Kubernetes objects. (In production etcd runs as a 3 or 5 node Raft cluster for HA - Week 1's consensus material - but it's the same program.)

Step 2: start the apiserver - the front door¶

The apiserver is the REST API in front of etcd. Point it at your etcd:

$ kube-apiserver \
  --etcd-servers=http://127.0.0.1:2379 \
  --service-cluster-ip-range=10.0.0.0/24 \
  --bind-address=127.0.0.1 \
  --secure-port=6443 \
  --authorization-mode=AlwaysAllow \   # workshop shortcut; real clusters use RBAC + TLS auth
  --token-auth-file=/tmp/tokens.csv \  # a trivial token for kubectl (workshop)
  ... (cert flags) &

(Real bootstrap requires a PKI - CA, certs for each component, TLS everywhere. "Kubernetes the Hard Way" spends a whole section on certificates; for this workshop you can use --authorization-mode=AlwaysAllow and minimal certs to focus on the architecture. The security hardening is Week 23.)

Now point kubectl at your apiserver and talk to your hand-built control plane:

$ kubectl --server=https://127.0.0.1:6443 --token=<your-token> --insecure-skip-tls-verify get nodes
No resources found.        # the apiserver answers! (no nodes yet - we haven't added a kubelet)
$ kubectl ... get namespaces
NAME              STATUS   AGE
default           Active   10s
kube-system       Active   10s        <- the apiserver created the built-in namespaces in YOUR etcd

You're talking to a Kubernetes API you assembled from two processes. And here's the etcd connection made literal - look at what the apiserver wrote into etcd:

$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 get /registry --prefix --keys-only | head
/registry/namespaces/default
/registry/namespaces/kube-system
/registry/apiregistration.k8s.io/apiservices/v1.
...

Every Kubernetes object lives under /registry/... in etcd. When you kubectl get namespaces, the apiserver reads these keys. When you kubectl create anything, it writes here. You can see the entire cluster as key/value pairs - the abstraction is gone, it's a database.

Step 3: create an object, watch it land in etcd¶

Make the etcd-is-the-state truth undeniable. Create a ConfigMap via the API, then read it straight from etcd:

$ kubectl ... create configmap demo --from-literal=k=v
configmap/demo created
$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 get /registry/configmaps/default/demo
/registry/configmaps/default/demo
{"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"demo",...},"data":{"k":"v"}}

The object you created through the API is sitting in etcd as a value at a predictable key. kubectl -> apiserver -> etcd. That's the entire write path of Kubernetes, and you just watched a single object travel it. This is also why etcd backup = cluster backup: etcdctl snapshot save captures every object; restore it and the whole cluster comes back.

Step 4: add the scheduler - now Pods can be placed¶

Right now if you created a Pod it would sit Pending forever (exactly the scheduler workshop's lesson - no scheduler, no placement). Start the real scheduler as another client of your apiserver:

$ kube-scheduler \
  --kubeconfig=/tmp/scheduler.kubeconfig \   # points at https://127.0.0.1:6443
  --bind-address=127.0.0.1 &

It's just another process pointed at the apiserver - the same way your custom scheduler connected in that workshop. The real scheduler and your mini one are peers: both watch the apiserver for unscheduled Pods and write bindings back. There's nothing privileged about the "official" one; it's a client like any other.

Step 5: add a kubelet - now Pods actually run¶

The control plane decides; the kubelet executes (the scheduler workshop's decoupling). Start a kubelet to turn this host into a worker node:

$ kubelet \
  --kubeconfig=/tmp/kubelet.kubeconfig \     # registers with your apiserver
  --config=/tmp/kubelet-config.yaml \        # cgroup driver, container runtime endpoint, etc.
  --container-runtime-endpoint=unix:///run/containerd/containerd.sock &

The kubelet registers itself as a Node with the apiserver and starts watching for Pods assigned to it. Confirm your cluster now has a node:

$ kubectl ... get nodes
NAME       STATUS   ROLES    AGE   VERSION
myhost     Ready    <none>   20s   v1.29.x      <- your kubelet registered itself

Step 6: the payoff - run a Pod on the cluster you built¶

Everything's assembled: etcd (state) + apiserver (API) + scheduler (placement) + kubelet (execution). Create a Pod and watch it travel the entire pipeline you built by hand:

$ kubectl ... run hello --image=nginx
$ kubectl ... get pod hello -o wide -w
NAME    READY   STATUS              NODE
hello   0/1     Pending             <none>      <- created, stored in etcd, not yet scheduled
hello   0/1     ContainerCreating   myhost      <- scheduler bound it; kubelet is starting it
hello   1/1     Running             myhost      <- kubelet started the container

Trace what just happened across the components you started: 1. kubectl run -> apiserver validated and wrote the Pod to etcd (spec.nodeName=""). 2. The scheduler (watching the apiserver) saw an unscheduled Pod, picked the node, wrote the binding back through the apiserver to etcd. 3. The kubelet (watching the apiserver) saw a Pod assigned to it, called the container runtime, started nginx, and reported status back.

A container is running on a Kubernetes cluster that you assembled from individual processes - no kubeadm, no kind, no cloud. Every step went through the apiserver; all state lives in etcd; the scheduler and kubelet are just clients running watch-and-act loops. You've seen the whole machine with the cover off.

Step 7: prove the resilience model - kill and restart components¶

The "stateless clients + one stateful store" design has a dramatic consequence you can demonstrate. Kill the scheduler and the apiserver - leave etcd alone:

$ kill %scheduler %apiserver       # control plane "down"
$ kubectl ... get pods             # fails - the API is gone
The connection to the server was refused

The API is down, but the running Pod keeps running (the kubelet is independent and the container doesn't care the control plane is down). Now restart the apiserver and scheduler:

$ kube-apiserver ... &             # restart, pointed at the SAME etcd
$ kubectl ... get pods
NAME    READY   STATUS    AGE
hello   1/1     Running   5m         <- still there! state was never lost

Nothing was lost, because all state was in etcd the whole time - the apiserver and scheduler are stateless, so restarting them just re-reads from etcd. This is why control-plane components are trivially HA (run several, they share etcd) and why the only thing you must back up is etcd. Lose a control-plane process: restart it. Lose etcd without a backup: lose the cluster. You just proved both halves.

Now extend it (toward the real "Hard Way")¶

1. Add the PKI. Do it properly: a CA, TLS certs for every component, --authorization-mode=Node,RBAC. This is half of "Kubernetes the Hard Way" and where Week 23's security lives. Painful and illuminating.
2. Add kube-controller-manager. Start it and watch the built-in controllers (Deployment, ReplicaSet, the ones from the controller workshop) come alive - now kubectl create deployment actually creates Pods.
3. Add kube-proxy + a CNI. Wire up Service networking (kube-proxy) and pod networking (your CNI from that workshop) so Pods get IPs and Services route - a fully functional node.
4. Multi-node etcd. Run a 3-member etcd cluster and kill one member; watch Raft keep the cluster available (Week 1 consensus, live).
5. Then appreciate kubeadm/kind. Run kubeadm init or kind create cluster and recognize every step it automates - it's doing exactly what you just did by hand, plus the PKI and CNI.

What you might wonder¶

"Is this really how production clusters are built?" The components are identical - production runs exactly these binaries (etcd, apiserver, scheduler, controller-manager, kubelet, kube-proxy). What differs is automation and hardening: kubeadm/kops/managed-services (EKS/GKE/AKS) bootstrap them with proper PKI, HA topologies, and lifecycle management. Managed control planes run these same processes for you (you just don't see them). Doing it by hand once means you know what those tools and services are actually managing.

"Why is etcd so central / why all the fuss about backing it up?" Because etcd holds 100% of cluster state - every object, as you saw under /registry/. Everything else is stateless and reconstructible. So etcd is your single point of truth and your single point of catastrophic failure: a corrupted etcd with no backup is an unrecoverable cluster. This is why etcd runs as an HA Raft cluster (survive node loss) and why etcdctl snapshot save on a schedule is non-negotiable. The most important cluster backup is the etcd snapshot.

"The apiserver is the only thing talking to etcd - why that design?" Centralizing etcd access in the apiserver gives one place for validation, authorization, admission (the webhook workshop!), versioning, and the watch/cache machinery every controller depends on. If components talked to etcd directly, you'd have no consistent policy enforcement and no shared watch infrastructure. The apiserver as the sole etcd client is why admission control and RBAC can exist - they're enforced at that single chokepoint.

"Everything's a client of the apiserver - even the scheduler?" Yes, and this is the unifying insight of the whole workshop series. The scheduler, controller-manager, kubelet, your custom controller, your custom scheduler, kubectl, Argo CD - all of them are just API clients running watch-and-act loops against the apiserver. There's no privileged inner ring. This is why you could build a scheduler and a controller as ordinary programs: they're the same kind of thing as the "real" components. The control plane is a set of peers coordinating through one API and one database.

"Should I ever bootstrap by hand in real life?" Almost never - use kubeadm, a managed service, or a tool. But doing it once is the single best way to understand Kubernetes: it dissolves the "magic cluster" mental model into "processes + etcd + an API," which is what makes you able to debug a broken control plane (apiserver won't start? check its etcd connection and certs; Pods stuck Pending? is the scheduler running? Pods won't start? is the kubelet healthy?). The hand-build is for understanding, not production.

What this gave you¶

• You assembled a working Kubernetes control plane from individual processes: etcd + apiserver + scheduler + kubelet, no installer.
• You saw cluster state is etcd - read Kubernetes objects directly as key/values under /registry/.
• You watched a Pod travel the full pipeline you built: kubectl -> apiserver -> etcd -> scheduler -> kubelet -> running container.
• You proved the resilience model: killed and restarted stateless components with zero data loss, because all state is in etcd.
• You understand why etcd is the one thing to back up, why the apiserver is the sole etcd client (enabling RBAC/admission), and why everything is "just an API client."
• You can now debug a broken control plane, because you know it's processes + a database, not magic.

The workshop series, complete¶

You've now built, by hand, the core of every layer of Kubernetes: - a controller (the reconcile loop - the atom of the platform), - an operator with a CRD (extending the API), - admission webhooks (intercepting the API), - a scheduler (deciding placement), - pod networking / a CNI (connectivity), - a GitOps sync loop (git as truth), - an autoscaler (feedback control), - and now the control plane itself (processes + etcd).

The thread through all of them: Kubernetes is a set of control loops coordinating through one API backed by one database. Every component - built-in or custom, control-plane or yours - is a client watching the apiserver and acting to reconcile desired state with actual. You don't just use Kubernetes now; you understand how it's built, because you've built each piece.

Back to the Hard-Way Capstone month, or revisit the controller workshop where the series began.

Workshop - Build a Kubernetes controller from scratch¶

Companion to Kubernetes -> Month 03 -> Week 9: client-go Internals and a Bare Controller. The chapter explains the informer + workqueue pattern. This workshop has you build a real, running controller in Go and then watch it reconcile - including the moment that makes Kubernetes finally click: you delete something it manages, and it heals it back. By the end you'll have a working prototype and the one mental model the entire platform is built on.

~90 minutes. Needs: a local cluster (kind or k3d - free, Kubernetes-in-Docker), Go 1.21+, and kubectl. No cloud, no cost.

What you'll build, and the idea it makes concrete¶

You'll build a controller that watches ConfigMaps labeled workshop.io/managed=true and ensures each one has a companion ConfigMap named <name>-synced holding a copy of its data. Small on purpose - because the point isn't the feature, it's watching the reconcile loop work.

Here's the one idea, and why building beats reading it:

Kubernetes is declarative and level-triggered. You declare desired state; controllers continuously drive actual state toward it. A controller doesn't react to "an event happened" - it asks "what should exist? what does exist? make them match," over and over, forever. That's why you can delete a Pod and the Deployment brings it back, why kubectl apply is idempotent, why the system self-heals.

You can read that paragraph ten times and not feel it. You'll feel it in Step 7 when you delete a ConfigMap and watch your own code recreate it within a second. Building a controller is the "build a container by hand" of Kubernetes - it dispels the magic by making you the magician.

Step 0: spin up a cluster¶

$ kind create cluster --name workshop
$ kubectl cluster-info --context kind-workshop
$ kubectl get nodes
NAME                     STATUS   ROLES           AGE   VERSION
workshop-control-plane   Ready    control-plane   30s   v1.29.x

A full Kubernetes cluster, running in Docker on your laptop, in ~20 seconds. (k3d is equally fine: k3d cluster create workshop.) Everything in this workshop runs against it.

Step 1: the mental model - level vs edge triggered¶

Before code, fix the distinction that everything hinges on:

• Edge-triggered (the wrong model for K8s): "when X changes, do Y." Fragile - if you miss the event (controller was down, a message dropped), you never act, and state drifts forever.
• Level-triggered (how K8s works): "whenever you run, make actual match desired - regardless of what changed or whether you saw the change." Robust - a missed event is harmless because the next reconcile fixes it anyway.

A Kubernetes controller is a level-triggered reconcile loop:

loop forever:
    desired = what the spec says should exist
    actual  = what's really in the cluster
    if actual != desired:
        make actual match desired

Events (a ConfigMap was created/changed/deleted) are just hints to reconcile sooner - not instructions about what to do. The reconcile function always recomputes from scratch. Hold this; the code is a direct expression of it.

Step 2: scaffold the project¶

$ mkdir mirror-controller && cd mirror-controller
$ go mod init workshop/mirror-controller
$ go get k8s.io/client-go@v0.29.3 k8s.io/api@v0.29.3 k8s.io/apimachinery@v0.29.3 k8s.io/klog/v2

Three libraries do the work: client-go (the Kubernetes Go client - informers, workqueues, clients), api (the typed objects like ConfigMap), apimachinery (the machinery: object metadata, errors, label selectors).

Step 3: connect to the cluster¶

First, just prove you can talk to the API. Create main.go:

package main

import (
    "context"
    "fmt"
    "os"
    "path/filepath"

    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/clientcmd"
)

func main() {
    // Out-of-cluster: read your kubeconfig (the same file kubectl uses).
    kubeconfig := filepath.Join(os.Getenv("HOME"), ".kube", "config")
    config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
    if err != nil {
        panic(err)
    }
    clientset, err := kubernetes.NewForConfig(config)
    if err != nil {
        panic(err)
    }

    // Sanity check: list ConfigMaps in the default namespace.
    cms, err := clientset.CoreV1().ConfigMaps("default").List(context.TODO(), metav1.ListOptions{})
    if err != nil {
        panic(err)
    }
    fmt.Printf("connected. %d configmaps in default\n", len(cms.Items))
}

$ go run .
connected. 1 configmaps in default

That clientset is your typed door to every Kubernetes API. clientset.CoreV1().ConfigMaps(ns).Get/List/Create/Update/Delete(...) - the verbs you'd guess. You're now a Kubernetes client. But polling with List in a loop would hammer the API. Controllers use informers instead.

Step 4: the informer - watch, don't poll¶

An informer maintains a local, always-current cache of objects by watching the API, and calls your handlers when things change. It's the efficient "watch" that makes controllers cheap. Replace main with the informer wiring:

package main

import (
    "os"
    "path/filepath"
    "time"

    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/klog/v2"
)

func main() {
    kubeconfig := filepath.Join(os.Getenv("HOME"), ".kube", "config")
    config, _ := clientcmd.BuildConfigFromFlags("", kubeconfig)
    clientset, _ := kubernetes.NewForConfig(config)

    // A factory builds informers that share one watch connection + cache.
    // resync period 30s: re-deliver every cached object every 30s (belt-and-suspenders).
    factory := informers.NewSharedInformerFactory(clientset, 30*time.Second)
    cmInformer := factory.Core().V1().ConfigMaps().Informer()

    // For now, just log what the informer sees, to watch it work.
    cmInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    func(obj interface{}) { klog.Infof("ADD    %s", key(obj)) },
        UpdateFunc: func(old, new interface{}) { klog.Infof("UPDATE %s", key(new)) },
        DeleteFunc: func(obj interface{}) { klog.Infof("DELETE %s", key(obj)) },
    })

    stop := make(chan struct{})
    defer close(stop)
    factory.Start(stop)                              // start watching
    cache.WaitForCacheSync(stop, cmInformer.HasSynced) // wait for the initial list
    klog.Info("cache synced, watching...")
    <-stop // block forever
}

func key(obj interface{}) string {
    k, _ := cache.MetaNamespaceKeyFunc(obj)  // "namespace/name"
    return k
}

Run it, then in another terminal create a ConfigMap and watch the informer notice:

$ go run .
cache synced, watching...
ADD    kube-system/kube-root-ca.crt
ADD    default/some-existing-cm
...

# other terminal:
$ kubectl create configmap demo --from-literal=hello=world
# back in the controller log, instantly:
UPDATE default/demo
ADD    default/demo
$ kubectl delete configmap demo
DELETE default/demo

You're watching the informer's event stream in real time. But notice the problem: these handlers run inline on the watch thread, and if reconcile is slow or errors, you'd block the stream or drop work. That's what the workqueue fixes - and it's where the level-triggered design lives.

Step 5: the workqueue - the heart of the pattern¶

The handlers shouldn't do the work. They should just drop a key (namespace/name) onto a queue. Separate workers pull keys and reconcile. This decouples "something changed" from "do the work," gives you retries with backoff, and dedupes (the same key queued twice = processed once). This is the controller pattern.

The key insight that makes it level-triggered: the queue holds keys, not events. A worker pulling key default/foo doesn't know or care whether foo was added, updated, or its companion was deleted - it just reconciles foo from scratch. Here's the full controller:

package main

import (
    "context"
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "os"
    "path/filepath"
    "strings"
    "time"

    corev1 "k8s.io/api/core/v1"
    apierrors "k8s.io/apimachinery/pkg/api/errors"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    listers "k8s.io/client-go/listers/core/v1"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/client-go/util/workqueue"
    "k8s.io/klog/v2"
)

const (
    managedLabel = "workshop.io/managed"
    syncedSuffix = "-synced"
    checksumAnn  = "workshop.io/source-checksum"
)

type Controller struct {
    client   kubernetes.Interface
    lister   listers.ConfigMapLister   // reads from the cache, never the API (fast)
    synced   cache.InformerSynced
    queue    workqueue.RateLimitingQueue
}

func main() {
    kubeconfig := filepath.Join(os.Getenv("HOME"), ".kube", "config")
    config, _ := clientcmd.BuildConfigFromFlags("", kubeconfig)
    client, _ := kubernetes.NewForConfig(config)

    factory := informers.NewSharedInformerFactory(client, 30*time.Second)
    cmInformer := factory.Core().V1().ConfigMaps()

    c := &Controller{
        client: client,
        lister: cmInformer.Lister(),
        synced: cmInformer.Informer().HasSynced,
        queue:  workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter()),
    }

    // Handlers enqueue a SOURCE key. The trick: a change to a companion
    // (named foo-synced) enqueues its source (foo), so deleting a companion
    // triggers reconcile of the source -> self-healing.
    cmInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc:    c.enqueue,
        UpdateFunc: func(old, new interface{}) { c.enqueue(new) },
        DeleteFunc: c.enqueue,
    })

    stop := make(chan struct{})
    defer close(stop)
    factory.Start(stop)
    c.Run(2, stop) // 2 workers
}

// enqueue maps any observed ConfigMap to the SOURCE key to reconcile.
func (c *Controller) enqueue(obj interface{}) {
    cm, ok := obj.(*corev1.ConfigMap)
    if !ok { // tombstone on delete
        if t, isT := obj.(cache.DeletedFinalStateUnknown); isT {
            cm, _ = t.Obj.(*corev1.ConfigMap)
        }
        if cm == nil { return }
    }
    name := cm.Name
    if strings.HasSuffix(name, syncedSuffix) {
        name = strings.TrimSuffix(name, syncedSuffix) // companion -> source
    } else if cm.Labels[managedLabel] != "true" {
        return // not ours, and not a companion: ignore
    }
    c.queue.Add(cm.Namespace + "/" + name)
}

func (c *Controller) Run(workers int, stop <-chan struct{}) {
    defer c.queue.ShutDown()
    if !cache.WaitForCacheSync(stop, c.synced) {
        return
    }
    klog.Info("cache synced, controller running")
    for i := 0; i < workers; i++ {
        go func() {
            for c.processNext() {
            }
        }()
    }
    <-stop
}

func (c *Controller) processNext() bool {
    key, quit := c.queue.Get()
    if quit {
        return false
    }
    defer c.queue.Done(key)

    if err := c.reconcile(key.(string)); err != nil {
        klog.Errorf("reconcile %s failed, requeueing: %v", key, err)
        c.queue.AddRateLimited(key) // retry with exponential backoff
        return true
    }
    c.queue.Forget(key) // success: reset the backoff
    return true
}

Step 6: the reconcile function - desired vs actual, made literal¶

Here's where the mental model becomes code. reconcile is handed a source key. It computes desired state and makes the cluster match - every time, from scratch:

func (c *Controller) reconcile(key string) error {
    ns, name, _ := cache.SplitMetaNamespaceKey(key)

    // 1. What's the SOURCE? (read from cache via the lister)
    source, err := c.lister.ConfigMaps(ns).Get(name)
    if apierrors.IsNotFound(err) {
        // Source is gone. The companion has an owner reference to it,
        // so Kubernetes garbage-collects the companion automatically.
        klog.Infof("source %s gone; companion will be GC'd", key)
        return nil
    }
    if err != nil {
        return err
    }
    if source.Labels[managedLabel] != "true" {
        return nil // not managed
    }

    // 2. What SHOULD the companion look like? (desired state)
    sum := checksum(source.Data)
    companionName := name + syncedSuffix
    desired := &corev1.ConfigMap{
        ObjectMeta: metav1.ObjectMeta{
            Name:      companionName,
            Namespace: ns,
            Annotations: map[string]string{checksumAnn: sum},
            // Owner reference: ties the companion's lifecycle to the source.
            // Delete the source -> Kubernetes deletes this automatically.
            OwnerReferences: []metav1.OwnerReference{{
                APIVersion: "v1", Kind: "ConfigMap",
                Name: source.Name, UID: source.UID,
                Controller: boolPtr(true),
            }},
        },
        Data: source.Data,
    }

    // 3. What EXISTS? Make actual match desired.
    existing, err := c.lister.ConfigMaps(ns).Get(companionName)
    if apierrors.IsNotFound(err) {
        _, err = c.client.CoreV1().ConfigMaps(ns).Create(context.TODO(), desired, metav1.CreateOptions{})
        klog.Infof("created companion %s/%s", ns, companionName)
        return err
    }
    if err != nil {
        return err
    }
    // It exists - has the source drifted since we last synced?
    if existing.Annotations[checksumAnn] != sum {
        updated := existing.DeepCopy()
        updated.Data = source.Data
        updated.Annotations[checksumAnn] = sum
        _, err = c.client.CoreV1().ConfigMaps(ns).Update(context.TODO(), updated, metav1.UpdateOptions{})
        klog.Infof("updated companion %s/%s (source changed)", ns, companionName)
        return err
    }
    // Already correct. Reconcile is a no-op. (This is the common case - and
    // it's why running reconcile a thousand times is harmless: idempotent.)
    return nil
}

func checksum(data map[string]string) string {
    h := sha256.New()
    for k, v := range data {
        fmt.Fprintf(h, "%s=%s;", k, v)
    }
    return hex.EncodeToString(h.Sum(nil))[:12]
}

func boolPtr(b bool) *bool { return &b }

Read reconcile against the mental model: get desired (the companion that should exist), get actual (what does exist), make them match - create if missing, update if drifted, do nothing if correct. It never asks "what event brought me here?" It just drives toward desired state. That is a Kubernetes controller. Everything else - Deployments, the scheduler, cert-manager, Argo - is this loop at larger scale.

Step 7: run it and watch reconciliation - the payoff¶

$ go run .
cache synced, controller running

Now, in another terminal, watch the cluster while you poke it. Open a watch so you see reconciliation live:

$ kubectl get configmaps -w           # -w = watch; leave this running

Create a managed ConfigMap:

$ kubectl create configmap colors --from-literal=sky=blue
$ kubectl label configmap colors workshop.io/managed=true

In the watch terminal, a colors-synced appears within a second - your controller saw the label, reconciled, and created the companion:

NAME            DATA   AGE
colors          1      3s
colors-synced   1      1s        <- your controller created this

$ kubectl get configmap colors-synced -o jsonpath='{.data}{"\n"}'
{"sky":"blue"}                    # it copied the data

Now change the source and watch the copy follow:

$ kubectl patch configmap colors --type merge -p '{"data":{"sky":"orange"}}'
# controller log: updated companion default/colors-synced (source changed)
$ kubectl get configmap colors-synced -o jsonpath='{.data}{"\n"}'
{"sky":"orange"}                  # the copy synced

Step 8: the moment Kubernetes clicks - watch it self-heal¶

This is the whole workshop in one action. Delete the companion and watch your controller bring it back:

$ kubectl delete configmap colors-synced
configmap "colors-synced" deleted
# controller log, instantly:
#   created companion default/colors-synced
$ kubectl get configmap colors-synced
NAME            DATA   AGE
colors-synced   1      1s         <- it's BACK, 1 second old

You deleted it. It came back. You didn't tell the controller "recreate it" - you deleted the companion, the informer noticed, enqueued the source, reconcile ran, saw the companion was missing, and recreated it. This is exactly why deleting a Pod doesn't work (the Deployment recreates it), why kubectl apply is safe to re-run, why Kubernetes is self-healing. The desired state is the source of truth; the controller relentlessly enforces it. You just built that.

And the owner-reference payoff - delete the source, and the companion is garbage-collected automatically:

$ kubectl delete configmap colors
configmap "colors" deleted
$ kubectl get configmap colors-synced
Error from server (NotFound): configmaps "colors-synced" not found   # auto-cleaned

You didn't write deletion logic for that path - the owner reference told Kubernetes "this companion belongs to that source," and the garbage collector did the rest. Owner references are how every controller manages cleanup.

Step 9: break it - see the failure mode and retry¶

Watch the workqueue's retry/backoff handle errors. Temporarily make reconcile fail: add return fmt.Errorf("boom") at the top of reconcile, re-run, create a managed ConfigMap. The log shows the backoff:

reconcile default/colors failed, requeueing: boom
reconcile default/colors failed, requeueing: boom    # ~1s later
reconcile default/colors failed, requeueing: boom    # ~2s later (exponential)
reconcile default/colors failed, requeueing: boom    # ~4s later...

AddRateLimited requeues with exponential backoff, so a transiently-failing reconcile retries patiently instead of hot-looping. Remove the boom line and the next retry succeeds and Forgets the key (resetting backoff). This resilience - keep trying, back off, never give up - is why controllers survive flaky APIs and transient errors. It's free from the workqueue.

Step 10: the hardening slice - minimum RBAC¶

Running locally you used your admin kubeconfig. To run in the cluster, the controller needs a ServiceAccount with the least privilege that works - exactly what it does and no more:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: mirror-controller
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get", "list", "watch", "create", "update", "delete"]  # exactly our verbs

The discipline: grant the verbs the controller actually calls (we get/list/watch to observe, create/update/delete to reconcile) and nothing else. Over-broad RBAC on a controller is a real security hole - a compromised controller can do whatever its ServiceAccount allows. Minimum privilege is the rule.

Now extend it (the real Week 9 lab)¶

You've built the core. Extend it into the full lab to cement the pattern:

1. Cross-namespace mirror. Instead of a companion in the same namespace, mirror managed ConfigMaps into every namespace matching a prefix. You'll watch Namespaces too (a second informer) and enqueue affected sources when a new namespace appears. (Owner references don't cross namespaces - you'll handle cleanup yourself, which teaches why.)
2. Leader election. Run two replicas; add tools/leaderelection so only one acts (a Lease is the lock). Kill the leader, watch the standby take over - HA controllers.
3. Metrics + health. Expose queue depth and reconcile duration on /metrics, add /healthz and /readyz. This is what makes a controller operable.
4. Graduate to controller-runtime. Rebuild this in controller-runtime/kubebuilder (Week 10) and see how much of the boilerplate it hides - now that you know what it's hiding, because you built it by hand.

What you might wonder¶

"Why client-go directly instead of controller-runtime/kubebuilder?" Because building it raw shows you the machinery - informer, lister, workqueue, reconcile - that every framework hides. controller-runtime is what you'll use in production (it's less code), but if you start there, the reconcile loop is magic. Build it once by hand (this workshop), then let the framework do it (the operator workshop). You can only appreciate what kubebuilder saves you after you've written the boilerplate yourself.

"Why a workqueue? Why not just do the work in the event handler?" Three reasons, all of which you saw: decoupling (handlers stay fast, work happens on workers), retries with backoff (AddRateLimited - Step 9), and deduplication (the same key enqueued repeatedly is processed once). Doing work inline blocks the watch stream and gives you none of this. The workqueue is the controller pattern.

"Is reconcile really called over and over even when nothing changed?" Yes - the 30s resync re-delivers every object, so reconcile runs periodically even with no changes (plus on every event). That's the point: it's level-triggered, so a no-op reconcile (Step 6's "already correct" path) must be cheap and harmless. Idempotency is non-negotiable - reconcile must produce the same result no matter how many times it runs. A controller that isn't idempotent corrupts state on the second run.

"Why read from the lister (cache) instead of the API?" Calling the API in the hot path would hammer the apiserver - a controller watching thousands of objects reconciling constantly would melt it. The informer keeps a local cache; the lister reads from it (microseconds, no network). You only call the API to change things (create/update/delete), never to read in reconcile. This read-from-cache / write-to-API split is a core scaling principle.

"How does this scale to real controllers like Deployments?" Identically. The Deployment controller watches Deployments and ReplicaSets, and reconciles: desired replicas vs actual Pods, create/delete to match. The ReplicaSet controller watches ReplicaSets and Pods. The scheduler watches unscheduled Pods and binds them. Every one is the loop you just built - watch, diff desired vs actual, act, requeue. Once you've built one controller, you understand the entire control plane's shape.

What this gave you¶

• You built a real, running Kubernetes controller in Go from raw client-go - informer, lister, workqueue, reconcile.
• You watched it reconcile: create a source, the companion appears; change the source, the copy follows.
• You watched it self-heal - deleted the companion, your code brought it back - the moment that makes "declarative, level-triggered, self-healing" concrete instead of a slogan.
• You used owner references for automatic garbage collection.
• You saw the workqueue's retry/backoff handle failures, and the read-from-cache / write-to-API scaling split.
• You know minimum-privilege RBAC for a controller.
• You understand that every controller in Kubernetes - Deployments, the scheduler, every operator - is this exact loop.

This is the foundation for the rest of the Kubernetes workshops. Next, you'll extend the API itself: build an operator with a Custom Resource Definition, where you define a new kind of Kubernetes object and the controller that gives it meaning.

Back to the Controllers & Operators month for the conceptual frame.

Workshop - Build a custom scheduler¶

Companion to Kubernetes -> Month 01 -> Week 3: The Scheduler. The chapter explains how the scheduler assigns Pods to nodes through filtering and scoring. This workshop has you build a working scheduler - a real program that watches for unscheduled Pods and binds them to nodes by your own logic - and watch your placement decisions take effect. By the end you'll understand that scheduling is "just" another control loop, and you'll have replaced one of the most mysterious-seeming control-plane components with ~120 lines of Go.

~90 minutes. Needs: kind/k3d (ideally a multi-node cluster), Go 1.21+, kubectl. Prerequisite: the controller-from-scratch workshop - a scheduler is a controller with a specific job.

What you'll build, and the idea it makes concrete¶

You'll build a minimal scheduler that watches for Pods requesting schedulerName: workshop-scheduler, picks a node by a simple policy (least-loaded by Pod count), and binds the Pod to it. Then you'll watch your scheduler place Pods - and watch a Pod sit Pending forever when your scheduler isn't running, proving you are the thing making placement happen.

The idea this makes concrete:

Scheduling is not magic, and not part of the kubelet - it's a separate control loop. The default kube-scheduler watches for Pods with no node assigned (spec.nodeName == ""), runs filter (which nodes can run this?) then score (which is best?), and writes the chosen node back via a Bind call. The kubelet on that node then notices "a Pod is assigned to me" and runs it. Placement and execution are decoupled. You can run multiple schedulers side by side, and a Pod picks one by schedulerName.

The controller workshop showed reconcile on ConfigMaps. This shows the same loop doing the job that feels most like core-Kubernetes-magic - and revealing it's the same pattern: watch, decide, act.

Step 0: how scheduling actually works¶

The mental model, because the magic dissolves once you see it:

1. You create a Pod. The apiserver stores it with spec.nodeName = "" (unscheduled).
2. A SCHEDULER (watching for unscheduled Pods) notices it.
3. Scheduler FILTERS: which nodes have enough CPU/mem, match nodeSelector, tolerate taints?  -> feasible nodes
4. Scheduler SCORES: rank the feasible nodes by some policy -> best node
5. Scheduler BINDS: writes spec.nodeName = <best node> (a POST to pods/<name>/binding).
6. The KUBELET on that node sees a Pod assigned to it, pulls images, starts containers.

Two facts that surprise people: - The scheduler never starts a container. It only decides and records the decision (the bind). The kubelet does the running. Decision and execution are different components. - A Pod with no scheduler to handle it stays Pending forever. There's no fallback - if you set a schedulerName no scheduler is watching, the Pod just waits. You'll see this directly.

Your scheduler implements steps 2-5.

Step 1: a multi-node cluster (so placement is visible)¶

Placement is only interesting with more than one node. kind can make a multi-node cluster:

$ cat <<EOF | kind create cluster --name sched-workshop --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
EOF
$ kubectl get nodes
NAME                          STATUS   ROLES           AGE
sched-workshop-control-plane  Ready    control-plane   40s
sched-workshop-worker         Ready    <none>          25s
sched-workshop-worker2        Ready    <none>          25s
sched-workshop-worker3        Ready    <none>          25s

Three worker nodes - your scheduler will choose among them.

Step 2: the scheduler loop - watch unscheduled Pods¶

A scheduler is a controller whose "desired state" is "every Pod assigned to a node." Set up the project and the watch:

$ mkdir mini-scheduler && cd mini-scheduler
$ go mod init workshop/mini-scheduler
$ go get k8s.io/api@v0.29.3 k8s.io/apimachinery@v0.29.3 k8s.io/client-go@v0.29.3

package main

import (
    "context"
    "fmt"
    "path/filepath"
    "os"

    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    "k8s.io/apimachinery/pkg/fields"
    "k8s.io/client-go/informers"
    "k8s.io/client-go/kubernetes"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/tools/clientcmd"
    "k8s.io/klog/v2"
)

const schedulerName = "workshop-scheduler"

func main() {
    kubeconfig := filepath.Join(os.Getenv("HOME"), ".kube", "config")
    config, _ := clientcmd.BuildConfigFromFlags("", kubeconfig)
    client, _ := kubernetes.NewForConfig(config)

    factory := informers.NewSharedInformerFactory(client, 0)
    podInformer := factory.Core().V1().Pods()
    nodeInformer := factory.Core().V1().Nodes()
    podLister := podInformer.Lister()
    nodeLister := nodeInformer.Lister()

    podInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
        AddFunc: func(obj interface{}) {
            pod := obj.(*corev1.Pod)
            // Only handle pods that ask for US and aren't scheduled yet.
            if pod.Spec.SchedulerName != schedulerName || pod.Spec.NodeName != "" {
                return
            }
            if err := schedule(client, podLister, nodeLister, pod); err != nil {
                klog.Errorf("failed to schedule %s/%s: %v", pod.Namespace, pod.Name, err)
            }
        },
    })

    stop := make(chan struct{})
    factory.Start(stop)
    cache.WaitForCacheSync(stop, podInformer.Informer().HasSynced, nodeInformer.Informer().HasSynced)
    klog.Infof("%s running", schedulerName)
    <-stop
}

The trigger is the same as any controller: an informer fires when an unscheduled Pod (asking for our scheduler) appears. The work is schedule().

Step 3: filter and score - the decision¶

Here's the heart - the same filter-then-score the real scheduler does, simplified to one policy (fewest Pods wins):

func schedule(client kubernetes.Interface, podLister listers.PodLister,
    nodeLister listers.NodeLister, pod *corev1.Pod) error {

    nodes, err := nodeLister.List(labels.Everything())
    if err != nil {
        return err
    }

    // FILTER: which nodes can run this pod?
    var feasible []*corev1.Node
    for _, n := range nodes {
        if isReady(n) && !hasBlockingTaint(n) {   // (real schedulers also check cpu/mem/affinity)
            feasible = append(feasible, n)
        }
    }
    if len(feasible) == 0 {
        return fmt.Errorf("no feasible node for %s", pod.Name)
    }

    // SCORE: pick the node running the fewest pods (least-loaded policy).
    allPods, _ := podLister.List(labels.Everything())
    countByNode := map[string]int{}
    for _, p := range allPods {
        if p.Spec.NodeName != "" {
            countByNode[p.Spec.NodeName]++
        }
    }
    best := feasible[0]
    for _, n := range feasible[1:] {
        if countByNode[n.Name] < countByNode[best.Name] {
            best = n
        }
    }

    // BIND: record the decision. THIS is what "scheduling" actually is.
    binding := &corev1.Binding{
        ObjectMeta: metav1.ObjectMeta{Name: pod.Name, Namespace: pod.Namespace},
        Target:     corev1.ObjectReference{Kind: "Node", Name: best.Name},
    }
    err = client.CoreV1().Pods(pod.Namespace).Bind(context.TODO(), binding, metav1.CreateOptions{})
    if err != nil {
        return err
    }
    klog.Infof("bound %s/%s -> %s (had %d pods)", pod.Namespace, pod.Name, best.Name, countByNode[best.Name])

    // Emit a Scheduled event so `kubectl describe pod` shows what happened (like the real scheduler).
    return nil
}

func isReady(n *corev1.Node) bool {
    for _, c := range n.Status.Conditions {
        if c.Type == corev1.NodeReady {
            return c.Status == corev1.ConditionTrue
        }
    }
    return false
}
func hasBlockingTaint(n *corev1.Node) bool {
    for _, t := range n.Spec.Taints {
        if t.Effect == corev1.TaintEffectNoSchedule {
            return true
        }
    }
    return false
}

The Bind call is the whole job: a POST to pods/<name>/binding that writes spec.nodeName. That's it - "scheduling a Pod" is setting one field. Everything else (filter, score) is just deciding which value to write. The real scheduler has dozens of filter and score plugins; yours has one of each, but the shape is identical.

Step 4: run it and watch your scheduler place Pods¶

$ go run .
workshop-scheduler running

Create a few Pods that ask for your scheduler:

$ for i in 1 2 3 4 5 6; do
  kubectl run pod$i --image=pause --overrides='{"spec":{"schedulerName":"workshop-scheduler"}}'
done

Watch your scheduler's log - it's making decisions in real time:

bound default/pod1 -> sched-workshop-worker  (had 0 pods)
bound default/pod2 -> sched-workshop-worker2 (had 0 pods)
bound default/pod3 -> sched-workshop-worker3 (had 0 pods)
bound default/pod4 -> sched-workshop-worker  (had 1 pods)    <- least-loaded: spreads them
bound default/pod5 -> sched-workshop-worker2 (had 1 pods)
bound default/pod6 -> sched-workshop-worker3 (had 1 pods)

Your least-loaded policy spread 6 Pods evenly across 3 nodes. Confirm from the cluster's side:

$ kubectl get pods -o wide --sort-by='{.spec.nodeName}'
NAME   READY   STATUS    NODE
pod1   1/1     Running   sched-workshop-worker
pod4   1/1     Running   sched-workshop-worker
pod2   1/1     Running   sched-workshop-worker2
pod5   1/1     Running   sched-workshop-worker2
pod3   1/1     Running   sched-workshop-worker3
pod6   1/1     Running   sched-workshop-worker3

Two Pods per node, exactly as your scoring decided. And critically - kubectl describe pod pod1 shows it was scheduled and then the kubelet started it. Your scheduler decided; the kubelet executed. You watched the decoupling.

Step 5: the proof that YOU are the scheduler - watch a Pod hang Pending¶

This is the "it clicks" moment. Stop your scheduler (Ctrl-C). Now create another Pod that asks for it:

$ kubectl run orphan --image=pause --overrides='{"spec":{"schedulerName":"workshop-scheduler"}}'
$ kubectl get pod orphan
NAME     READY   STATUS    AGE
orphan   0/1     Pending   30s          <- stuck. Forever.
$ kubectl describe pod orphan | grep -A2 Events
Events:  <none>                         # NO scheduler touched it. No node assigned. It just waits.

The Pod is Pending with no events - because nothing is watching for workshop-scheduler Pods. There's no default fallback; a Pod whose named scheduler is absent waits indefinitely. Now restart your scheduler:

$ go run .
# instantly in the log:
bound default/orphan -> sched-workshop-worker (had 2 pods)
$ kubectl get pod orphan
NAME     READY   STATUS    AGE
orphan   1/1     Running   3s           <- your scheduler woke up and placed it

You watched a Pod sit unschedulable until your program ran, then get placed the instant it did. That's the proof that scheduling is a control loop you can own - the Pending Pod was waiting for you. This is also why a broken/overloaded scheduler manifests as Pods stuck Pending cluster-wide - now you know exactly why.

Step 6: break it - filter everything out¶

See the "no feasible node" path. Taint all workers so your filter rejects them:

$ kubectl taint nodes -l '!node-role.kubernetes.io/control-plane' workshop=blocked:NoSchedule
$ kubectl run blocked --image=pause --overrides='{"spec":{"schedulerName":"workshop-scheduler"}}'
# scheduler log: failed to schedule default/blocked: no feasible node for blocked
$ kubectl get pod blocked     # Pending - your filter rejected every node

The Pod is unschedulable because no node passed the filter - exactly what happens with the real scheduler when resource requests exceed every node's capacity, or affinity/taints exclude everything. Remove the taint (kubectl taint nodes ... workshop-) and your scheduler (which retries via the informer's resync) places it. This is the difference between "no scheduler" (Step 5, no events) and "scheduler ran but found nowhere to put it" (Step 6, a FailedScheduling event in the real scheduler).

Now extend it¶

1. Real resource filtering. Filter on actual CPU/memory: sum the requests of pods already on each node, compare to allocatable. This is the core of real scheduling - bin-packing by resources.
2. Respect nodeSelector / affinity / taints+tolerations. Add these filters and watch pods land only where they're allowed. You'll appreciate how much the default scheduler does.
3. Spread vs bin-pack. Add a second scoring policy (most-loaded = bin-pack, to empty nodes for scale-down) and make it configurable. This is the real spreading-vs-consolidation tradeoff.
4. The Scheduler Framework plugin path. Instead of a standalone scheduler, write a plugin for the real kube-scheduler (a Filter or Score plugin). This is how production custom scheduling is done - you extend the battle-tested scheduler rather than replace it.

What you might wonder¶

"Should I ever run a custom scheduler in production?" Rarely as a replacement - the default scheduler is sophisticated (resource fit, affinity, topology spread, preemption) and hard to beat. But custom scheduling is real for specialized needs: gang scheduling (all-or-nothing for ML training jobs), hardware-topology-aware placement (GPUs/NUMA), or batch/HPC. The right path is almost always a Scheduler Framework plugin (extend the default), not a from-scratch scheduler. Building one here is to understand the default - so you can reason about why a Pod landed where it did, and why one is stuck Pending.

"Why is binding a separate API call, not just updating the Pod?" The binding subresource (pods/<name>/binding) exists specifically so scheduling is an atomic, auditable, permission-scoped operation - a component can be granted "bind pods" without "edit pods." It also lets the apiserver enforce that a Pod is bound exactly once. It's the same subresource-design reason status is separate from spec.

"How does the real scheduler avoid placing two pods on a node that only fits one?" It tracks assumed Pods (Pods it has bound but that the cache hasn't caught up on yet) so back-to-back decisions account for in-flight bindings - avoiding the race where it over-commits a node in the gap before the bind is observed. Your mini-scheduler has this race (it reads pod counts from the cache); the real one handles it. A good "now extend it" thought experiment.

"What's preemption?" When a high-priority Pod can't fit, the scheduler can evict lower-priority Pods to make room (then schedule the high-priority one). It's why PriorityClass matters. Your mini-scheduler doesn't preempt - it just fails when nothing fits. Preemption is one of the most complex parts of the real scheduler.

"Multiple schedulers really run at once?" Yes - that's why schedulerName exists. The default scheduler handles Pods with schedulerName: default-scheduler (the default); yours handles workshop-scheduler. They coexist, each watching for its own Pods. This is how you can run a specialized scheduler for batch jobs alongside the default for services, in the same cluster.

What this gave you¶

• You built a working scheduler: watch unscheduled Pods, filter feasible nodes, score, bind.
• You learned scheduling is recording a decision (the bind sets spec.nodeName) - the scheduler never runs containers; the kubelet does.
• You watched your least-loaded policy spread Pods across nodes, confirmed from the cluster.
• You watched a Pod hang Pending with no events until your scheduler ran - proving scheduling is a loop you own, and explaining cluster-wide Pending.
• You saw the "no feasible node" path and how it differs from "no scheduler."
• You know when custom scheduling is warranted (gang/topology/batch) and that the Scheduler Framework plugin path beats a from-scratch replacement.

Next: go below the control plane to the network - build pod networking and see how packets actually flow between Pods.

Back to the Control Plane month.

Workshop - Build a GitOps sync loop¶

Companion to Kubernetes -> Month 05 -> Week 17: GitOps (ArgoCD and Flux). The chapter explains GitOps - git as the source of truth, continuously reconciled into the cluster. This workshop has you build a tiny GitOps engine - a loop that pulls manifests from a git repo and applies them, detects drift, and heals it - then watch it revert a manual change you make to the cluster. By the end you'll understand precisely what Argo CD and Flux do, because you'll have built the core in ~80 lines.

~75 minutes. Needs: kind/k3d, Go 1.21+, git, kubectl. Prerequisite: the controller-from-scratch workshop - GitOps is the reconcile loop with git as the desired state.

What you'll build, and the idea it makes concrete¶

You'll build a controller whose "desired state" isn't a Kubernetes object - it's a git repository. The loop: clone/pull the repo, apply the manifests it contains, and on every tick re-apply so any drift (someone kubectl edit-ed a Deployment, someone deleted a Service) is corrected back to what git says. Then you'll kubectl scale a Deployment by hand and watch your engine revert it within seconds.

The idea this makes concrete:

GitOps is the reconcile loop you already know, with git as the source of truth. Instead of "desired state lives in a Website CR," desired state lives in a git repo, and the controller continuously drives the cluster toward what's committed. The consequences are the whole value proposition: git is the audit log (every change is a commit), rollback is git revert, the cluster self-heals toward git (manual kubectl changes get reverted), and no human ever runs kubectl apply against production - they open a PR. Argo CD and Flux are this loop, industrialized with a UI, multi-repo/multi-cluster support, health checks, and sync waves.

The controller workshop reconciled a CR into objects. This reconciles a git repo into a whole cluster - same loop, different desired-state source, and it's the pattern modern platform teams run everything on.

Step 0: the GitOps model¶

Fix the shift in thinking. Traditional ("push") deployment: a human or CI runs kubectl apply / helm install to the cluster. GitOps ("pull") deployment: an agent in the cluster continuously pulls from git and reconciles:

PUSH (traditional):                      PULL (GitOps):
  human/CI --kubectl apply--> cluster      git repo <--pull-- agent-in-cluster --apply--> cluster
  - imperative, point-in-time              - declarative, continuous
  - drift accumulates silently             - drift is detected and reverted
  - "what's actually deployed?" = unknown  - "what's deployed?" = what's in git, always
  - rollback = remember the old command    - rollback = git revert
  - audit = who ran what when? = murky      - audit = git history

The agent runs the same reconcile loop as any controller: desired = manifests in git, actual = objects in the cluster, act = apply to make them match, forever. You're building that agent.

Step 1: a git repo of manifests (the source of truth)¶

Make a local git repo holding the manifests your engine will deploy:

$ mkdir gitops-repo && cd gitops-repo && git init
$ mkdir manifests
$ cat > manifests/app.yaml <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 2
  selector: {matchLabels: {app: web}}
  template:
    metadata: {labels: {app: web}}
    spec:
      containers:
      - {name: web, image: nginx:1.27-alpine}
EOF
$ git add . && git commit -m "deploy web at 2 replicas"
$ cd ..

This repo is now the declared truth: "the cluster should run web at 2 replicas." Anyone wanting to change production changes this - via a commit/PR - not the cluster directly.

Step 2: the sync loop¶

The engine: every N seconds, pull the repo and apply its manifests. Set up the project:

$ mkdir mini-gitops && cd mini-gitops && go mod init workshop/mini-gitops
$ go get k8s.io/client-go@v0.29.3 k8s.io/apimachinery@v0.29.3 k8s.io/cli-runtime@v0.29.3

The core loop (using kubectl apply's server-side-apply semantics via the dynamic client; for the workshop we shell out to kubectl apply for clarity, then note the API path):

package main

import (
    "os/exec"
    "time"

    "k8s.io/klog/v2"
)

const (
    repoURL  = "/path/to/gitops-repo"   // local for the workshop; a real URL in production
    repoDir  = "/tmp/gitops-checkout"
    manifest = "manifests"
    interval = 15 * time.Second
)

func main() {
    klog.Info("mini-gitops starting")
    for {
        if err := syncOnce(); err != nil {
            klog.Errorf("sync failed: %v", err)
        }
        time.Sleep(interval)
    }
}

func syncOnce() error {
    // 1. PULL: get the latest desired state from git.
    if err := pull(); err != nil {
        return err
    }
    // 2. APPLY: drive the cluster toward what git says. apply is idempotent and
    //    declarative - re-applying the same manifests reverts any drift back to git.
    cmd := exec.Command("kubectl", "apply", "-f", repoDir+"/"+manifest, "--prune",
        "-l", "gitops.workshop.io/managed=true")
    out, err := cmd.CombinedOutput()
    klog.Infof("apply:\n%s", out)
    return err
}

func pull() error {
    // clone if missing, else fetch+reset to origin (git is the truth, discard local)
    if _, err := exec.Command("test", "-d", repoDir+"/.git").Output(); err != nil {
        return exec.Command("git", "clone", repoURL, repoDir).Run()
    }
    if err := exec.Command("git", "-C", repoDir, "fetch", "origin").Run(); err != nil {
        return err
    }
    return exec.Command("git", "-C", repoDir, "reset", "--hard", "origin/HEAD").Run()
}

Two design choices that are GitOps: - git reset --hard origin/HEAD - the local checkout is disposable; git is the truth. Never trust local state. - kubectl apply --prune - apply also deletes cluster objects that are no longer in git (matched by the label). Without prune, removing a manifest from git wouldn't remove it from the cluster - git wouldn't be the complete truth. Prune makes the cluster exactly mirror git: present in git -> exists; absent from git -> deleted.

(Add the label gitops.workshop.io/managed=true to your manifests so prune knows what it owns - or use Argo/Flux's ownership tracking in production.)

Step 3: run it and watch git become the cluster¶

$ go run .
mini-gitops starting
apply:
deployment.apps/web created

Your engine pulled the repo and created the Deployment. The cluster now matches git:

$ kubectl get deployment web
NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    2/2     2            2           10s        <- 2 replicas, as git declares

You didn't run kubectl apply - your engine did, from git. This is the GitOps inversion: the deploy happened because the repo said so, pulled by an in-cluster agent.

Step 4: the payoff - watch it heal drift¶

This is the moment GitOps clicks. Manually change the cluster, as a panicked engineer might at 3 AM:

$ kubectl scale deployment web --replicas=5
deployment.apps/web scaled
$ kubectl get deployment web
NAME   READY   UP-TO-DATE   AVAILABLE
web    5/5     5            5            <- you scaled it to 5

Now wait one sync interval (~15s) and watch your engine's log and the cluster:

# engine log:
apply:
deployment.apps/web configured        <- it noticed drift and re-applied git's spec

$ kubectl get deployment web
NAME   READY   UP-TO-DATE   AVAILABLE
web    2/2     2            2            <- back to 2. Your manual change was REVERTED.

Your manual kubectl scale was undone, because git says 2 and the engine relentlessly enforces git. This is GitOps's superpower and its discipline: the cluster cannot drift from git, because anything not in git gets reverted. The only way to make web run 5 replicas is to commit that change:

$ cd gitops-repo
$ sed -i 's/replicas: 2/replicas: 5/' manifests/app.yaml
$ git commit -am "scale web to 5"
$ cd -
# wait one interval; engine log: deployment.apps/web configured
$ kubectl get deployment web
NAME   READY   UP-TO-DATE   AVAILABLE
web    5/5     5            5            <- now it stays at 5, because GIT says 5

The change stuck because it's in git. This is the entire GitOps contract: the cluster is a function of the repo. Want to change production? Commit. Want to roll back? Revert the commit and watch the engine restore the previous state. Want to know what's deployed? Read the repo. Want an audit trail? It's the git log.

Step 5: rollback is git revert¶

Watch the rollback story - the operational win that sells GitOps to teams:

$ cd gitops-repo
$ git revert --no-edit HEAD          # undo the "scale to 5" commit
$ cd -
# wait one interval
$ kubectl get deployment web
NAME   READY   AVAILABLE
web    2/2     2                       <- reverted to 2, because git reverted