Appendix A: Command-Line Tools

A reference for the tools the book reaches for repeatedly. Standard Unix utilities (cat, ls, grep, sed, awk, tar, sha256sum) are omitted; the book uses them for plumbing, not as subjects.

Container Engines And CLIs

docker

A user-facing wrapper over dockerd, which talks to containerd over a Unix socket. Almost everything in this book that uses docker could equally use nerdctl or podman; the syntax is shared by convention, not by spec.

• docker run --rm -it <image> <cmd> — start, attach a TTY, remove on exit.
• docker inspect -f '{{.State.Pid}}' <name> — get a running container's host PID, the entry point for reaching into /proc/<pid>/.
• docker buildx build --output type=oci,dest=out.tar . — build an OCI image archive without pushing.
• docker run --privileged — drop the bounding capability set, the seccomp profile, the AppArmor or SELinux profile, and the device cgroup restrictions. Removes most of what makes the container a container.

podman

A daemonless, rootless-first container engine. CLI is docker-compatible, so alias docker=podman works for most flows. Without a daemon, every invocation forks runc directly; the container's parent is the calling shell. Rootless storage lives under ~/.local/share/containers/storage/.

• podman info --format '{{.Store.GraphDriverName}}' — kernel overlay versus fuse-overlayfs.

nerdctl

containerd's docker-compatible CLI. Same arguments your fingers already know (run -it, exec, logs, pull), but talks directly to containerd without a daemon.

• nerdctl --namespace k8s.io <cmd> — operate on the namespace the kubelet uses.

ctr

containerd's native debug CLI. Closer to the gRPC API than nerdctl, less ergonomic, but the right tool when you want to see exactly what containerd is doing.

• ctr namespaces ls — containerd's logical namespaces (default, moby, k8s.io), separate from kernel namespaces.
• ctr -n k8s.io content ls — every blob in the content store for that namespace.
• ctr -n k8s.io snapshot ls — committed image layers and active container roots.
• ctr -n k8s.io leases ls — leases keep arbitrary content alive during in-flight operations.

crictl

The Kubernetes CRI debug CLI. Speaks the same gRPC interface the kubelet uses, so it sees what the kubelet sees.

• crictl ps — running containers from the runtime's view.
• crictl pods --namespace kube-system — pod sandboxes in a Kubernetes namespace.

kubectl

Kubernetes cluster CLI. Talks to kube-apiserver over HTTPS, never directly to a node's runtime — anything touching containerd from a node uses crictl or nerdctl.

• kubectl get pods -o wide — see which node a pod landed on.
• kubectl exec -it <pod> -- sh — under the hood, a setns chain plus an execve.

buildah

Image-building CLI from the Containers project. Builds OCI images without a daemon and without root, and lets you script the layer-by-layer construction of an image as a sequence of buildah commands instead of a Dockerfile.

skopeo

Image transport tool. Copies, inspects, and verifies images between registries, OCI layouts on disk, Docker daemons, and archive files.

• skopeo copy docker://alpine:3.20 oci:./alpine-oci:3.20 — pull into an OCI layout for inspection.
• skopeo inspect docker://alpine:3.20 — print the manifest and config without pulling.

OCI Runtimes

runc

The reference OCI runtime. Reads an OCI bundle (a directory containing config.json and a rootfs/), creates the namespaces and cgroups, applies the security profiles, and execs the user process. Almost every container on Linux runs under runc or a drop-in replacement.

• runc create <id> / runc start <id> — the two-step creation flow described in chapter 9.
• runc list — running containers from runc's state directory (/run/runc/).
• runc spec — emit a default config.json template.

crun

A C reimplementation of runc. Faster startup, lower memory per running container, drop-in compatible. The default runtime in podman.

youki

A Rust reimplementation of the OCI runtime. Less battle-tested than runc or crun but tracks the spec.

Linux Primitive Tools

unshare(1)

User-space wrapper for unshare(2). Creates new namespaces and execs a command inside them.

• unshare --user --map-root-user --pid --fork --mount-proc --mount --uts --ipc --net -- ... — the full hand-rolled-container flag stack.
• --fork — required with --pid, because the calling process cannot enter the new PID namespace itself.
• --mount-proc — remount /proc so ps reflects the new PID space.

nsenter(1)

Calls setns(2) on a target namespace and execs a command. The mechanism behind kubectl exec, crictl exec, and docker exec.

• nsenter -t <pid> -m -p -- <cmd> — enter a process's mount and PID namespaces.
• Flag letters mirror namespace types: -m mount, -u UTS, -i IPC, -n net, -p PID, -U user, -C cgroup, -T time.

lsns(1)

Lists namespaces by walking /proc/*/ns/. The fastest way to figure out which container is using which namespace.

• lsns -t net — only network namespaces.
• lsns -p <pid> — namespaces a specific process belongs to.

Capabilities, Seccomp, MAC

capsh(1)

libcap's capability-set inspector and launcher. Decodes capability bitmaps and runs commands with chosen sets.

• capsh --decode=0x<hex> — turn the hex blobs from /proc/<pid>/status into capability names.
• capsh --print — dump the calling shell's capability state.

setpriv(1)

Drop privileges before exec. The right tool for testing capability and no_new_privs configurations without writing C.

• setpriv --no-new-privs <cmd> — sets prctl(PR_SET_NO_NEW_PRIVS, 1) then execs.
• setpriv --bounding-set -all,+chown <cmd> — run with a custom bounding set.
• setpriv --inh-caps, --ambient-caps — manipulate inheritable and ambient sets directly.

getcap(8) / setcap(8)

Read and write file capabilities (the security.capability xattr). The mechanism that lets ping work for non-root users by carrying cap_net_raw=ep on the binary.

• getcap -r /usr/bin /usr/sbin — every binary with file capabilities under those paths.
• setcap cap_net_bind_service=ep <bin> — grant a binary the right to bind low ports.

aa-status(8)

AppArmor status. Confirms the LSM is loaded and lists which profiles are in enforce versus complain mode.

getenforce(8) / setenforce(8) / sestatus(8)

SELinux equivalents. getenforce returns Enforcing, Permissive, or Disabled; sestatus is the verbose form. setenforce 0 is a debugging crutch — useful for ruling SELinux in or out as the cause of a failure, almost never the fix.

ausearch(8)

Searches the kernel audit log. The single most useful tool for diagnosing SELinux denials.

• ausearch -m AVC -ts recent — recent Access Vector Cache denials (SELinux's term for "policy said no").

bpftool(8)

Inspects and manipulates eBPF programs and maps. cgroup v2's device control is implemented as an attached BPF program, so bpftool is the only way to see what device policy a container has.

• bpftool cgroup tree — every cgroup with attached BPF programs, by attach type.
• bpftool cgroup show <path> — programs attached to a specific cgroup.
• bpftool prog dump xlated id <id> — disassemble a loaded program.

Cgroups And systemd

systemctl(1)

systemd's control CLI.

• systemctl status <unit> — including the unit's CGroup: line, which is where its processes live in /sys/fs/cgroup.
• systemctl set-property <unit> MemoryMax=1G — write a resource limit through systemd. Going through systemd matters because systemd owns the tree.

systemd-run(1)

Launches a transient unit and tracks it under a chosen slice. The convenient way to put a process into a specific cgroup without writing files by hand.

• systemd-run --slice=demo.slice --unit=oneshot.service sleep 60 — run sleep as a tracked service.
• systemd-run --scope -p MemoryMax=100M -- <cmd> — run a one-shot scope with a memory limit.

systemd-cgls(1)

Pretty-prints the cgroup tree as systemd has shaped it: slices, scopes, and services in their hierarchy.

Filesystem And Mount

findmnt(8)

Lists mounts as a tree. The right tool for "what propagation type is / set to?" and "what is mounted under /sys/fs/cgroup?"

• findmnt -o TARGET,SOURCE,FSTYPE,OPTIONS,PROPAGATION — the most useful column set.

mount(8) / umount(8)

Front-ends to mount(2) and umount2(2). Relevant invocations:

• mount -t overlay overlay -o lowerdir=...,upperdir=...,workdir=... /merged — assemble a layered filesystem (chapter 6).
• mount --make-rprivate / — break propagation from / so a new mount namespace's setup does not leak back to the host.
• mount --bind <src> <dst> and mount --rbind <src> <dst> — make <src> appear at <dst>, recursively in the --rbind form.

fuse-overlayfs

Userspace OverlayFS that does not require CAP_SYS_ADMIN for trusted.* xattrs. The fallback rootless container engines use on kernels older than 5.11.

jq

CLI JSON processor. Used throughout the book to walk OCI image manifests, runtime config.json files, and docker inspect output.

• jq '.layers[].digest' manifest.json — extract every layer digest.
• jq -r '.config.digest' manifest.json | sed 's/sha256://' — strip the digest's algorithm prefix.

Networking

ip(8)

The iproute2 swiss-army knife, replacing every legacy networking utility. Used in nearly every chapter that touches the network namespace.

• ip link — list and manipulate network devices (add, del, set <dev> netns <ns>).
• ip addr — list and configure addresses.
• ip route — routing table.
• ip netns add|del|exec|list — network-namespace management. ip netns add foo bind-mounts the namespace under /var/run/netns/foo so it persists past the calling process.
• ip -n <ns> <subcmd> — equivalent to ip netns exec <ns> ip <subcmd>, less typing.

bridge(8)

iproute2's bridge subcommand. Lists and configures Linux bridges, including FDB entries and VLANs.

• bridge link show — which interfaces are members of which bridges.

ss(8)

Socket statistics. The netstat replacement.

• ss -tlnp — TCP, listening, no name resolution, with the owning process.
• ss -nlx — Unix domain sockets.

iptables(8) / nft(8)

Netfilter rule management. iptables is legacy; nft is the modern (nftables) interface. Container networking still mostly uses iptables-style rules through compatibility layers.

• iptables -t nat -L POSTROUTING -nv — the NAT rules that make outbound container traffic appear to come from the host.
• nft list ruleset — dump the entire nftables ruleset.

tcpdump(1)

Packet capture.

• tcpdump -i any -nn host 10.10.0.2 — every packet to or from a specific address, no name resolution.
• tcpdump -i veth-foo -w /tmp/cap.pcap — capture from a veth end into a pcap for analysis in Wireshark.

conntrack(8)

Inspects the kernel's connection-tracking table, the data structure netfilter NAT relies on. Reach for it when a connection works once and then breaks — a stale conntrack entry is the usual culprit.

• conntrack -L — dump the full table.

Init And Supervision

tini, dumb-init

Tiny init programs designed to run as PID 1 inside a container. They install signal handlers (so SIGTERM from docker stop actually terminates the process tree), forward signals to their child, and reap zombie processes. Without one of these or an equivalent, container shutdown and zombie reaping rely on the application doing the right thing as PID 1.

Where These Are Used