Workshop - Build pod networking from scratch (a minimal CNI)¶

Companion to Kubernetes -> Month 04 -> Weeks 13: The CNI Spec and Pod Networking. The chapter explains the CNI contract and how Pods get IPs and connectivity. This workshop has you build the networking by hand - wire up a network namespace with a veth pair and routing exactly as a CNI plugin does, then write a tiny CNI plugin script and watch the kubelet call it to network a real Pod. By the end "every Pod gets its own IP and can reach every other Pod" stops being a slogan and becomes plumbing you've installed yourself.

~90 minutes. Needs: a Linux host (CNI is Linux networking), kind/k3d, root, and the ip command. Builds on the Linux build-a-container-by-hand investigation (namespaces) - pod networking is namespaces + virtual ethernet + routing.

What you'll build, and the idea it makes concrete¶

First you'll wire two network namespaces together by hand (the exact moves a CNI plugin makes), prove they can ping each other, then package those moves into a minimal CNI plugin and watch the kubelet invoke it to give a real Pod connectivity. The idea:

Pod networking is not magic and not built into Kubernetes - it's delegated. Kubernetes defines a contract (CNI - Container Network Interface): "when I create a Pod, I'll call a plugin executable with the Pod's network namespace; the plugin's job is to give it an interface, an IP, and routes." Kubernetes itself ships no networking - it relies entirely on a CNI plugin (Calico, Cilium, Flannel, ...) to fulfill that contract. The "every Pod gets an IP, every Pod can reach every other Pod with no NAT" model is implemented by the plugin, using plain Linux primitives: network namespaces, veth pairs, bridges, and routes.

The container-by-hand investigation showed a Pod is a process in namespaces. This shows how that isolated process gets on the network - the piece that investigation deliberately left as "an empty network world."

Step 0: the model - veth pairs bridge namespaces¶

The core Linux primitive is the veth pair: a virtual ethernet cable with two ends. Put one end inside a Pod's network namespace and the other on the host, and you've connected the Pod to the host's networking. Add a bridge and routes, and Pods can reach each other:

   Pod A netns            host             Pod B netns
   +-----------+                           +-----------+
   | eth0      |                           | eth0      |
   |10.244.0.2 |                           |10.244.0.3 |
   +-----+-----+                           +-----+-----+
         |veth pair                              |veth pair
         |                                       |
      veth-a ----+      cni0 bridge      +---- veth-b
                 +------[10.244.0.1]------+
                          |
                       (routing to other nodes / internet)

A CNI plugin's whole job, per Pod: create a veth pair, move one end into the Pod's netns and name it eth0, assign it an IP, attach the host end to a bridge, and set up routes. That's it. You're about to do each step by hand.

Step 1: wire two namespaces by hand¶

Create two network namespaces (standing in for two Pods) and connect each to a bridge. All as root on a Linux host:

# Create a bridge (the "node" network all pods attach to)
$ sudo ip link add cni0 type bridge
$ sudo ip addr add 10.244.0.1/24 dev cni0
$ sudo ip link set cni0 up

# "Pod A": a network namespace
$ sudo ip netns add podA
# veth pair: host end veth-a, pod end will become eth0
$ sudo ip link add veth-a type veth peer name eth0-a
# move the pod end into podA's namespace
$ sudo ip link set eth0-a netns podA
# attach the host end to the bridge
$ sudo ip link set veth-a master cni0
$ sudo ip link set veth-a up
# inside podA: name it eth0, give it an IP, bring it up, add a default route
$ sudo ip netns exec podA ip link set eth0-a name eth0
$ sudo ip netns exec podA ip addr add 10.244.0.2/24 dev eth0
$ sudo ip netns exec podA ip link set eth0 up
$ sudo ip netns exec podA ip link set lo up
$ sudo ip netns exec podA ip route add default via 10.244.0.1

Repeat for "Pod B" (podB, veth-b/eth0-b, IP 10.244.0.3). Now the payoff - watch the two "Pods" reach each other:

$ sudo ip netns exec podA ping -c2 10.244.0.3
PING 10.244.0.3: 56 data bytes
64 bytes from 10.244.0.3: icmp_seq=1 ttl=64 time=0.07 ms
64 bytes from 10.244.0.3: icmp_seq=2 ttl=64 time=0.05 ms      # podA reaches podB!

You just built pod-to-pod networking by hand. Two isolated network namespaces, each with its own IP, talking over a bridge - exactly the "flat pod network" Kubernetes promises. No Kubernetes involved yet; this is pure Linux. A CNI plugin does precisely these ip commands, programmatically, every time a Pod is created. Inspect what you built:

$ sudo ip netns exec podA ip addr show eth0     # the pod's own eth0 at 10.244.0.2
$ bridge link show                              # veth-a and veth-b attached to cni0

(Clean up: sudo ip netns del podA podB; sudo ip link del cni0.)

Step 2: the CNI contract - how Kubernetes calls a plugin¶

Now connect this to Kubernetes. The kubelet, when starting a Pod, calls a CNI plugin executable with: - An operation (ADD when creating, DEL when destroying) via the CNI_COMMAND env var. - The Pod's network namespace path via CNI_NETNS. - The container ID, interface name (CNI_IFNAME, usually eth0), via env vars. - A JSON config on stdin (from /etc/cni/net.d/).

The plugin does the Step 1 moves for that Pod's netns, then prints a JSON result (the assigned IP) to stdout. That's the entire contract: an executable in /opt/cni/bin/, config in /etc/cni/net.d/, env vars + stdin in, JSON out. Kubernetes ships no networking - it just calls this executable.

Step 3: write a minimal CNI plugin¶

A CNI plugin can be any executable - even a shell script. Here's a minimal one that does Step 1's moves for whatever netns the kubelet hands it (/opt/cni/bin/mini-cni):

#!/bin/bash
# Minimal CNI plugin. Reads CNI_COMMAND, CNI_NETNS, CNI_IFNAME from env; config on stdin.
set -e
config=$(cat)                                  # JSON config from /etc/cni/net.d
case "$CNI_COMMAND" in
ADD)
    # Pick an IP (toy IPAM: random host octet; real plugins track allocations).
    ip_octet=$(( (RANDOM % 200) + 10 ))
    pod_ip="10.244.0.${ip_octet}"
    host_veth="veth$$"                          # unique host-side name

    # Create veth pair; move one end into the pod's netns as eth0.
    ip link add "$host_veth" type veth peer name "$CNI_IFNAME" netns "$CNI_NETNS" 2>/dev/null || true
    ip link set "$host_veth" master cni0
    ip link set "$host_veth" up
    ip -n "$CNI_NETNS" addr add "${pod_ip}/24" dev "$CNI_IFNAME"
    ip -n "$CNI_NETNS" link set "$CNI_IFNAME" up
    ip -n "$CNI_NETNS" link set lo up
    ip -n "$CNI_NETNS" route add default via 10.244.0.1

    # CNI requires a JSON result on stdout describing what we assigned.
    cat <<EOF
{
  "cniVersion": "1.0.0",
  "interfaces": [{"name": "$CNI_IFNAME", "sandbox": "$CNI_NETNS"}],
  "ips": [{"address": "${pod_ip}/24", "interface": 0, "gateway": "10.244.0.1"}]
}
EOF
    ;;
DEL)
    # Tear down: the veth in the netns is removed when the netns goes away;
    # a real plugin also frees the IP and removes the host veth.
    echo '{"cniVersion":"1.0.0"}'
    ;;
*)
    echo "unknown CNI_COMMAND $CNI_COMMAND" >&2; exit 1 ;;
esac

This is a real, if naive, CNI plugin - same contract as Calico or Cilium, just without IP-address-management, cross-node routing, or network policy. It does the veth+IP+route dance from Step 1, driven by the env vars the kubelet sets. (Note ip link add ... peer ... netns creates the peer directly in the target namespace - the modern one-step form of Step 1's two commands.)

Step 4: install it and watch the kubelet network a Pod¶

On a kind node (or a single-node cluster you control), install the plugin and its config, then create a Pod and watch it get networked by your plugin. (On kind, exec into the node: docker exec -it <node> bash.)

# install the plugin binary and config on the node
$ cp mini-cni /opt/cni/bin/mini-cni && chmod +x /opt/cni/bin/mini-cni
$ cat > /etc/cni/net.d/10-mini.conf <<EOF
{ "cniVersion": "1.0.0", "name": "mini", "type": "mini-cni" }
EOF
# create the bridge the plugin attaches to
$ ip link add cni0 type bridge 2>/dev/null; ip addr add 10.244.0.1/24 dev cni0; ip link set cni0 up

Now create a Pod and watch it come up with an IP your plugin assigned:

$ kubectl run netpod --image=nginx
$ kubectl get pod netpod -o wide
NAME     READY   STATUS    IP            NODE
netpod   1/1     Running   10.244.0.137  <node>      <- an IP YOUR plugin assigned

The kubelet started the Pod's sandbox (a network namespace), called /opt/cni/bin/mini-cni ADD with that netns, your plugin wired up the veth + IP + route, and reported the IP back - which the kubelet recorded as the Pod's IP. The kubelet did the lifecycle; your plugin did the networking. Confirm connectivity:

$ kubectl exec netpod -- ip addr show eth0    # the eth0 your plugin created
$ kubectl exec netpod -- ping -c2 10.244.0.1  # reaches the bridge gateway

You watched Kubernetes delegate networking to an executable you wrote. That's CNI.

Step 5: see what the real plugins add (and why)¶

Your mini-CNI works on one node. Real plugins solve the hard parts your toy skips - and seeing the gaps teaches what they're for:

• IPAM (IP Address Management). Your plugin picks a random octet and could collide. Real plugins maintain an allocation database per node so every Pod gets a unique, leak-free IP. (CNI even has dedicated IPAM plugins like host-local.)
• Cross-node routing. Your Pods can reach each other on the same node (same bridge). For Pod-on-node-A to reach Pod-on-node-B, you need routes between node subnets - Flannel does this with a VXLAN overlay, Calico with BGP, Cilium with eBPF. This is the "flat network across nodes" part, and it's the bulk of what a CNI does.
• Network policy. Your plugin allows all traffic. Enforcing NetworkPolicy (the Cilium/eBPF investigation) is a CNI feature - and why not all CNIs support it.
• Performance. iptables-based vs eBPF-based dataplanes (the netfilter and Cilium investigations) - real plugins differ enormously here at scale.

So "which CNI should I use?" is really "which set of these do I need, and how should they be implemented?" You now understand the question because you built the floor they all stand on.

Now extend it¶

1. Real IPAM. Replace the random octet with a file-based allocator (track assigned IPs in a file, never reuse a live one, free on DEL). You'll feel why IPAM is a whole subsystem.
2. Cross-node connectivity. On a 2-node kind cluster, add static routes between the two node Pod-subnets so a Pod on node1 can ping a Pod on node2. This is Flannel-host-gw in miniature.
3. Use the standard plugins. Chain the official bridge + host-local IPAM plugins (in /opt/cni/bin already) via a CNI config, instead of your script. See how the reference plugins do it properly.
4. Trace the dataplane. Use the netfilter investigation to watch a Pod's packets traverse the host's hooks - connecting CNI (which builds the path) to netfilter (which the packets traverse).

What you might wonder¶

"Kubernetes really ships no networking?" Correct - and it's deliberate. The kubelet creates the Pod's network namespace and then calls whatever CNI plugin is configured; if none is installed, Pods stay ContainerCreating with a "no CNI config" error. This delegation is why you choose a CNI (Calico/Cilium/Flannel/...) when building a cluster - it's a required component Kubernetes doesn't provide. kind/minikube install one for you, which is why Pods "just work" there.

"veth pair vs bridge vs overlay - how do they relate?" veth pair = the cable connecting one Pod to the host. Bridge (cni0) = the switch all Pods on a node plug into (intra-node connectivity). Overlay (VXLAN) or routing (BGP) = how nodes connect their Pod subnets (inter-node connectivity). Your workshop built the first two; real CNIs add the third. They're layers, not alternatives.

"Why is the Pod IP on the Pod, not the node?" That's the Kubernetes networking model: every Pod gets a real, routable IP in a flat space, and Pods communicate without NAT (unlike Docker's default bridge, which NATs containers behind the host IP). This is why services can target Pod IPs directly and why it feels like every Pod is a little VM on the network. The CNI plugin implements this flat model.

"How does this connect to Services and kube-proxy?" CNI gives Pods IPs and connectivity (this workshop). Services give a stable virtual IP that load-balances across a set of Pod IPs - implemented by kube-proxy (iptables - the netfilter investigation) or Cilium (eBPF). CNI is the foundation (Pods can talk); Services are built on top (stable addressing + load balancing). Different layers, often different components.

"Should I ever write a CNI plugin?" Almost never from scratch - Calico/Cilium/Flannel are mature and cover ~all needs. You'd write CNI logic for exotic environments (a custom cloud, special hardware) or as a meta-plugin (chaining others). Building one here is to understand pod networking - so you can debug "why can't this Pod reach that one?" by knowing exactly what the plugin set up (veth? IP? route? policy?).

What this gave you¶

• You wired pod-to-pod networking by hand: veth pairs, a bridge, IPs, routes - and watched two namespaces ping each other.
• You know the CNI contract: Kubernetes calls a plugin executable (ADD/DEL, netns, JSON in/out) and ships no networking itself.
• You wrote a minimal CNI plugin and watched the kubelet invoke it to give a real Pod an IP and connectivity.
• You know what real plugins add - IPAM, cross-node routing, network policy, dataplane performance - and why those are the hard parts.
• You can reason about "why can't this Pod reach that one?" in terms of the actual plumbing.
• You connected CNI (builds the path) to netfilter and kube-proxy/Services (what runs on it).

Next: the platform layer - build a GitOps sync loop that continuously reconciles your cluster from a git repo, the pattern Argo CD and Flux industrialize.

Back to the Networking & Storage month.

Comments