Containerd From The Bottom Up

Chapter 17: Lab Safety And Shape

A command that creates a namespace, writes a cgroup file, adds a veth pair, or starts containerd-shim-runc-v2 mutates host state. Part VI runs every mutating experiment inside a disposable Linux VM.

Mounts can leak through shared propagation. cgroup files can affect real processes. A veth pair and route can change packet flow. CNI plugins can write IPAM state and firewall rules. runc and containerd create runtime directories, cgroups, namespaces, mounts, and processes.

Safety Classes

The experiments use three safety classes.

Class Meaning Examples
Inspect-only Reads state without changing it. readlink /proc/self/ns/pid, findmnt, ctr -n lab containers ls.
User-namespace scoped Uses a user namespace to reduce host privilege. unshare --user --map-root-user labs.
VM-only mutation Changes kernel or runtime state. Mounts, cgroups, veth, bridges, CNI, runc, containerd tasks.

Root inside a user namespace is not root on the host: distribution policy, kernel configuration, subordinate ID mappings, and capability rules all affect what works. Lab steps that depend on predictable behavior call out a VM.

The Lab Shape

Every mutating experiment follows the same shape:

  1. State the question.
  2. Name the scope.
  3. Inspect the starting state.
  4. Make the smallest change.
  5. Inspect the changed state.
  6. Clean up.
  7. Verify cleanup.

An experiment that creates a namespace but never looks at /proc/<pid>/ns teaches an incantation. An experiment that starts a containerd task but never reads pgrep containerd-shim-runc-v2 hides the runtime boundary.

Namespace Tools

unshare(1) creates new namespaces for a program. nsenter(1) enters namespaces that already exist. In util-linux, both tools map command-line choices to kernel namespace flags.

unshare carries the namespace table:

{ .type = CLONE_NEWNET,   .name = "ns/net"  },
{ .type = CLONE_NEWPID,   .name = "ns/pid_for_children" },
{ .type = CLONE_NEWNS,    .name = "ns/mnt"  },

nsenter carries the same idea for target namespace files:

{ .nstype = CLONE_NEWNET,   .name = "ns/net",  .fd = -1 },
{ .nstype = CLONE_NEWPID,   .name = "ns/pid",  .fd = -1 },
{ .nstype = CLONE_NEWNS,    .name = "ns/mnt",  .fd = -1 },

The header comment states the purpose:

 * nsenter(1) - command-line interface for setns(2)

unshare(1) and nsenter(1) are lab handles for the same setns(2) and clone(2) flags that runtimes use.

Persistent References

A namespace can outlive the process that first created it if something still holds a reference. A bind mount of a namespace file is one way to keep that reference. That is useful when an experiment needs two shells to inspect one namespace, but it also creates a cleanup obligation.

The cleanup step should not stop at "exit the shell." It should verify that namespace bind mounts, processes, links, routes, cgroup directories, CNI cache entries, runc state, and containerd tasks are gone.

For Part VI, the shortest useful cleanup rule is:

If the setup step names an object, the cleanup step names the same object.

Every lab object in Part VI carries a cdb- prefix: cdb-net-a, cdb-br0, cdb-runc, cdb-task, cdb-lab.slice. The prefix is what lets one find, grep, or ip link filter the lab's residue out of the host's existing state.

The First Check

Every lab VM should start with one inspect-only check that records the environment. The exact commands can vary by distribution, but the state being checked does not:

uname -a
readlink /proc/self/ns/{mnt,pid,net,user,cgroup}
findmnt -no TARGET,FSTYPE,OPTIONS /sys/fs/cgroup
ip -br link

That gives the reader the kernel, current namespace identities, cgroup mount type, and starting link list. If a later cleanup step fails, the lab has a baseline to compare against.

Sources And Further Reading