Reading runc Source Code (Part 3): Container Resource Control in SPEC.md

Introduction ​

In the previous article, we explored the Filesystem chapter of SPEC.md and learned how containers build a complete filesystem environment from scratch. This article continues into the Resources chapter, examining how containers achieve resource control through Cgroups.

If Namespaces determine "what you can see," then Cgroups determine "how much you can use." A container without resource limits is no different from an ordinary process — it could consume all the CPU and memory on the host. Cgroups are the key to solving this problem.

Cgroups: The Core of Container Resource Control ​

SPEC.md gets straight to the point about resource control:

Cgroups are used to handle resource allocation for containers. This includes system resources like cpu, memory, and device access.

Cgroup Subsystems Defined in SPEC.md ​

OS Background: The table above reflects Cgroup v1’s subsystem layout — each subsystem is mounted independently under /sys/fs/cgroup/. Modern Linux systems (Ubuntu 22.04+, Fedora 31+, etc.) have fully transitioned to Cgroup v2, which uses a Unified Hierarchy architecture where all controllers are merged into a single directory tree. All hands-on demonstrations below use Cgroup v2.

Cgroup v1 vs v2: Architectural Differences ​

In Cgroup v1, each subsystem has its own hierarchy:

/sys/fs/cgroup/
├── cpu/
│   └── docker/<container-ID>/
├── memory/
│   └── docker/<container-ID>/
├── blkio/
│   └── docker/<container-ID>/
└── freezer/
    └── docker/<container-ID>/

In Cgroup v2, all controllers are merged into a unified hierarchy:

/sys/fs/cgroup/
└── system.slice/
    └── docker-<container-ID>.scope/
        ├── cgroup.controllers    # Available controllers
        ├── cgroup.subtree_control # Controllers enabled for subtree
        ├── cpu.max               # CPU bandwidth limit
        ├── memory.max            # Memory limit
        ├── memory.current        # Current memory usage
        ├── io.max                # I/O bandwidth limit
        ├── pids.max              # Max process count
        └── cgroup.freeze         # Freeze control (replaces freezer subsystem)

The core advantage of Cgroup v2 is consistency: a process belongs to exactly one cgroup node, and all resource limits are managed in the same location, avoiding the inconsistency issues where a process could be in different positions across v1’s separate hierarchies.

Statistics Collection ​

All cgroup subsystem are joined so that statistics can be collected from each of the subsystems.

SPEC.md requires containers to join all cgroup subsystems, partly for collecting statistics across all dimensions. When you run docker stats, the CPU usage, memory consumption, network I/O and other metrics all come from statistics files in the cgroup filesystem.

Why Not Use /proc for Container Monitoring? ​

A common question is: doesn’t /proc also contain system monitoring data? Why must containers use cgroups?

The answer is: /proc provides a global view (host), while cgroups provide a local view (container).

When you run free -m inside a container, it reads /proc/meminfo and displays the host’s (or Docker VM’s) total physical memory, not the container’s memory limit. For example, with docker run -m 100m limiting the container to 100MB, free -m might show 16GB — because /proc/meminfo is not cgroup-aware.

docker run -d --name mem_test -m 100m alpine tail -f /dev/null

docker exec mem_test free -m

docker stats --no-stream mem_test

docker rm -f mem_test

Modern Java (10+) and Node.js runtimes have adapted to this — when they detect they’re running in a container, they read cgroup files instead of /proc/meminfo to determine available memory, avoiding OOM Kill from over-allocating heap memory.

Freezer: The Container Pause Button ​

Freezer does not expose any stats but is joined so that containers can be paused and resumed.

Freezer is a special subsystem. It provides no statistics; its sole purpose is to batch pause and resume all processes within a container.

Why Not Use kill -STOP? ​

Linux’s native kill -STOP (SIGSTOP signal) can also pause processes, but Freezer is superior in two ways:

• Cannot be intercepted: Applications can catch signals, but cannot detect or prevent Freezer’s kernel-level freeze. When unfrozen, the process thinks only an instant has passed
• Atomic operation: kill sends signals one by one, creating a time gap — child processes might still be running when the parent is paused. Freezer freezes the entire cgroup as a whole at the kernel level, leaving no stragglers

Hands-on: Freezing and Unfreezing Processes with Cgroup v2 ​

This experiment requires two terminal windows, and the system must use Cgroup v2 (verify with mount | grep cgroup2).

Terminal A: Run the target process

bash -c 'while true; do echo "Process running... $(date +%T)"; sleep 2; done' &
PID=$!
echo "Target process PID: $PID"

Terminal B: Freeze and unfreeze

sudo mkdir -p /sys/fs/cgroup/my_freezer

echo $PID | sudo tee /sys/fs/cgroup/my_freezer/cgroup.procs

echo 1 | sudo tee /sys/fs/cgroup/my_freezer/cgroup.freeze

cat /sys/fs/cgroup/my_freezer/cgroup.freeze

echo 0 | sudo tee /sys/fs/cgroup/my_freezer/cgroup.freeze

kill -9 $PID
sudo rmdir /sys/fs/cgroup/my_freezer

In Cgroup v2, Freezer is no longer an independent subsystem but is controlled through the cgroup.freeze file in the unified hierarchy — write 1 to freeze, 0 to unfreeze. This is more concise than v1’s FROZEN/THAWED string interface.

docker pause and docker unpause use exactly this mechanism under the hood. In Kubernetes, while there’s no direct pause command, CRIU (Checkpoint/Restore In Userspace) leverages Freezer for live container migration: freeze the container, save a memory snapshot, transfer to another machine, then restore.

Memory Control: The Truth Behind OOM Kill ​

The memory subsystem is the most commonly felt part of container resource control — when a container’s memory exceeds its limit, the kernel’s OOM Killer terminates the process. This is the underlying cause of the OOMKilled status in Kubernetes.

Hands-on: Triggering OOM Kill with Cgroup v2 ​

Terminal A: Run a memory-hungry process

python3 -c "
import time
a = []
while True:
    a.append(' ' * 1024 * 1024)
    print(f'Allocated {len(a)} MB...')
    time.sleep(0.5)
" &
PID=$!
echo "Hungry process PID: $PID"

Terminal B: Set a 50MB memory limit

sudo mkdir -p /sys/fs/cgroup/my_mem_limit

cat /sys/fs/cgroup/cgroup.subtree_control

echo 52428800 | sudo tee /sys/fs/cgroup/my_mem_limit/memory.max

echo 0 | sudo tee /sys/fs/cgroup/my_mem_limit/memory.swap.max

echo $PID | sudo tee /sys/fs/cgroup/my_mem_limit/cgroup.procs

sudo rmdir /sys/fs/cgroup/my_mem_limit

Comparison of memory control interfaces between Cgroup v1 and v2:

Startup Synchronization: Preventing Process Escape from Cgroups ​

The most critical paragraph in SPEC.md describes the multi-process synchronization mechanism during container startup:

The parent process of the container’s init must place the init pid inside the correct cgroups before the initialization begins. This is done so that no processes or threads escape the cgroups. This sync is done via a pipe (specified in the runtime section below) that the container’s init process will block waiting for the parent to finish setup.

This addresses a serious race condition problem:

Dangerous scenario: If runc (parent process) lets the init process run first and then adds it to the cgroup — during this time gap, the init process may have already forked child processes. These children were born before cgroup configuration was complete, so they are not subject to any resource limits, successfully "escaping" the sandbox.

runc’s solution: Precise synchronization between parent and child processes via a pipe:

runc (parent)                           init (child)
     |                                       |
     |--- create pipe (FD 3) ------>         |
     |--- fork() creates init ------>        |
     |                                       |--- read pipe (blocks)
     |--- write init PID to cgroup.procs     |
     |--- write uid/gid mappings             |
     |--- close pipe write end ------>       |
     |                                       |--- pipe reads EOF, unblocks
     |                                       |--- mount filesystems, pivot_root
     |                                       |--- exec user process

The specific steps:

1. Create pipe: runc creates an IPC pipe, passing it to the init process via file descriptor 3
2. Create init process: runc forks the container’s init process
3. Init blocks: The init process immediately reads from the pipe and blocks since there’s no data
4. Configure cgroup: While init is blocked, runc writes its PID to the cgroup’s cgroup.procs file
5. Send go signal: runc closes the pipe’s write end, init reads EOF and unblocks
6. Init begins execution: The init process starts executing the user-specified command

Through this "pipe blocking" synchronization, runc guarantees: no container process can execute even a single line of application code before cgroup rules take effect.

CPU Control: Bandwidth Limiting and Core Pinning ​

While SPEC.md doesn’t elaborate on CPU control details, CPU is one of the most important dimensions of container resource management. Cgroups control CPU through two mechanisms:

CPU Bandwidth Control (cpu controller) ​

In Cgroup v2, the cpu.max file controls CPU bandwidth in the format $MAX $PERIOD (in microseconds):

echo "50000 100000" | sudo tee /sys/fs/cgroup/my_cgroup/cpu.max

echo "max 100000" | sudo tee /sys/fs/cgroup/my_cgroup/cpu.max

This corresponds to Docker’s --cpus flag: docker run --cpus=0.5 effectively sets cpu.max to 50000 100000.

CPU Core Pinning (cpuset controller) ​

The cpuset.cpus file pins processes to specific CPU cores:

echo "0-1" | sudo tee /sys/fs/cgroup/my_cgroup/cpuset.cpus

echo "0" | sudo tee /sys/fs/cgroup/my_cgroup/cpuset.cpus

In latency-sensitive scenarios (high-frequency trading, real-time audio/video), CPU pinning reduces performance overhead from context switching and cache invalidation.

I/O Control: Block Device Bandwidth Limiting ​

In Cgroup v2, the io.max file controls block device I/O bandwidth in the format $MAJOR:$MINOR rbps=$BYTES wbps=$BYTES:

lsblk -o NAME,MAJ:MIN

echo "8:0 rbps=10485760 wbps=10485760" | sudo tee /sys/fs/cgroup/my_cgroup/io.max

Compared to v1’s blkio subsystem, v2’s io.max consolidates read and write limits into a single file, making it more concise.

PID Control: Preventing Fork Bombs ​

The pids.max file limits the maximum number of processes in a cgroup:

echo 100 | sudo tee /sys/fs/cgroup/my_cgroup/pids.max

This is the key defense against fork bombs (:(){ :|:& };:). Without this limit, a malicious container could exhaust the host’s PID resources through infinite forking, preventing the entire machine from creating new processes.

Cgroup Practice with Docker ​

The manual cgroup operations above are performed automatically by the container runtime in practice. Docker provides rich parameters for setting resource limits:

docker run -d \
  --name resource_demo \
  --cpus=1.5 \              # CPU bandwidth: up to 1.5 cores
  --cpuset-cpus="0,1" \     # CPU pinning: only use CPU 0 and 1
  -m 256m \                 # Memory limit: 256MB
  --memory-swap 256m \      # Disable swap (swap limit = memory limit)
  --pids-limit 200 \        # Max processes: 200
  --device-read-bps /dev/sda:10mb \  # Disk read bandwidth limit
  alpine tail -f /dev/null

cat /sys/fs/cgroup/system.slice/docker-$(docker inspect --format='{{.Id}}' resource_demo).scope/cpu.max
cat /sys/fs/cgroup/system.slice/docker-$(docker inspect --format='{{.Id}}' resource_demo).scope/memory.max

docker rm -f resource_demo

Complete Container Resource Control Flow ​

Putting it all together, runc’s complete container resource control flow is:

1. Create cgroup node: Create a container-specific cgroup directory under the unified hierarchy at /sys/fs/cgroup/
2. Enable controllers: Enable required controllers (cpu, memory, io, pids, etc.) via cgroup.subtree_control
3. Configure resource limits: Write to cpu.max, memory.max, pids.max, etc. based on container configuration
4. Synchronize wait: Block the init process via pipe, ensuring cgroup configuration completes before proceeding
5. Write process: Write the init process’s PID to cgroup.procs
6. Release init: Close the pipe, allowing the init process to begin execution
7. Continuous monitoring: The runtime continuously collects statistics via memory.current, cpu.stat, etc.

The order of every step is critical — resource limits must take effect before the process begins execution, otherwise an "escape" window exists.

Summary ​

The Resources chapter of SPEC.md defines two core aspects of container resource control: what controls resources (10 cgroup subsystems covering CPU, memory, I/O, devices, process count, and more) and how to prevent escape (pipe synchronization ensures cgroups are in effect before process execution).

Containers have no magic — they are ordinary Linux processes that runc cleverly places into cgroup directories configured with various rules. Understanding this, when you encounter resources.limits in Kubernetes, the OOMKilled status, or docker stats output, you’ll know exactly what’s happening underneath.

In the next article, we’ll continue reading SPEC.md’s Security (Linux capabilities) and Runtime (init process) sections.