Reading runc Source Code (Part 4): Intel RDT Micro-Architecture Isolation in SPEC.md

Introduction ​

In container technology, Cgroups are commonly used to restrict CPU and memory resources. However, even with pinned CPU cores, different containers can still interfere with each other. For example, because the L3 cache and memory bus are shared, a CPU-intensive or memory-bandwidth-hungry "noisy neighbor" can cause a severe performance degradation for other containers on the same node.

To address this micro-architectural resource isolation problem, runc’s SPEC.md introduces support for Intel RDT. This article follows the specification to break down this hardcore feature.

What is Intel RDT? ​

The SPEC.md explicitly states at the beginning of this section:

Intel platforms with new Xeon CPU support Resource Director Technology (RDT). Cache Allocation Technology (CAT) and Memory Bandwidth Allocation (MBA) are two sub-features of RDT.

On newer Intel Xeon platforms, the hardware provides Resource Director Technology (RDT). In the context of runc, two of its sub-features are primarily used:

1. CAT (Cache Allocation Technology): For L3 cache allocation.
2. MBA (Memory Bandwidth Allocation): For memory bandwidth allocation.

CAT: Partitioning the L3 Cache ​

Cache Allocation Technology (CAT) provides a way for the software to restrict cache allocation to a defined 'subset' of L3 cache which may be overlapping with other 'subsets'. The different subsets are identified by class of service (CLOS) and each CLOS has a capacity bitmask (CBM).

Hardware Background: Traditional CPU isolation only operates at the Core level, but the L3 cache is usually shared among multiple cores or even the entire CPU socket. CAT allows software to partition the L3 cache into multiple subsets via a Class of Service (CLOS). Each CLOS corresponds to a Capacity Bitmask (CBM), such as 0x7FF. By setting different bitmasks, you can restrict a specific container to use only a few ways of the L3 cache, preventing its cache lines from being evicted by other containers.

MBA: Throttling Memory Bandwidth ​

Memory Bandwidth Allocation (MBA) provides indirect and approximate throttle over memory bandwidth for the software. A user controls the resource by indicating the percentage of maximum memory bandwidth or memory bandwidth limit in MBps unit…

If CAT is about managing cache "capacity", MBA is about managing memory "traffic". It provides an indirect and approximate memory bandwidth throttling mechanism for software. You can control this resource by specifying a percentage of the maximum memory bandwidth (e.g., 20%) or a direct limit in MB/s, preventing a single container from saturating the entire memory bus.

The resctrl Filesystem Interface ​

So, how do we interact with hardware-level Intel RDT in a Linux system? SPEC.md points out the kernel’s interaction method:

In Linux 4.10 kernel or newer, the interface is defined and exposed via "resource control" filesystem, which is a "cgroup-like" interface. Comparing with cgroups, it has similar process management lifecycle and interfaces in a container. But unlike cgroups’ hierarchy, it has single level filesystem layout.

The Linux kernel introduced support for CAT and MBA in versions 4.10 and 4.12 respectively, exposing a virtual filesystem named resctrl. Its operational logic is very similar to Cgroups (managing processes by writing PIDs to specific files), but with one significant difference: resctrl has a flat, single-level structure, and does not support nested hierarchies like Cgroups.

SPEC.md shows its typical directory tree:

mount -t resctrl resctrl /sys/fs/resctrl
tree /sys/fs/resctrl
/sys/fs/resctrl/
|-- info
|   |-- L3
|   |   |-- cbm_mask
|   |   |-- min_cbm_bits
|   |   |-- num_closids
|   |-- MB
|       |-- bandwidth_gran
|       |-- delay_linear
...
|-- schemata
|-- tasks
|-- <container_id>
    |-- ...
    |-- schemata
    |-- tasks

• info/: Contains RDT feature information supported by the current CPU, such as max L3 mask length and max supported CLOS count.
• schemata: Defines the resource allocation rules for the current group (e.g., the L3 mask and memory bandwidth percentage).
• tasks: Contains the Process IDs (PIDs) constrained by these rules. Creating a <container_id> directory is equivalent to creating a new CLOS isolation group.

OCI Configuration Mapping ​

To allow runc to leverage these features, the OCI config.json provides an intelRdt configuration field. SPEC.md gives the following example:

"intelRdt": {
    "closID": "guaranteed_group",
    "l3CacheSchema": "L3:0=7f0;1=1f",
    "memBwSchema": "MB:0=20;1=70"
}

• closID: The name of the directory created under /sys/fs/resctrl.
• l3CacheSchema: Specifies the L3 cache mask for each Socket (e.g., Node 0 gets 7f0, Node 1 gets 1f).
• memBwSchema: Specifies the memory bandwidth ratio for each Socket (e.g., Node 0 is limited to 20%, Node 1 to 70%).

When a container starts, runc is responsible for creating the corresponding closID directory in the resctrl filesystem, writing the schema rules, and adding the container’s main process PID to the tasks file.

Hands-on Demo ​

To deepen our understanding, we can bypass runc and directly simulate the container’s RDT resource limitation process on the host.

Note: This experiment requires a physical machine (or a VM supporting RDT virtualization) equipped with an RDT-capable Intel Xeon CPU and Linux kernel >= 4.12.

1. Mount the resctrl Filesystem ​

First, verify system support and mount resctrl:

$ sudo mount -t resctrl resctrl /sys/fs/resctrl

$ cat /sys/fs/resctrl/info/L3/cbm_mask
fffff

(Note: Output fffff means a 20-bit L3 cache mask, meaning the hardware partitions the L3 cache into 20 allocatable sub-blocks.)

2. Create an Isolation Group and Allocate Resources ​

We will simulate the isolation requirement in config.json by creating a group named demo_container:

$ sudo mkdir /sys/fs/resctrl/demo_container

$ echo "L3:0=f" | sudo tee /sys/fs/resctrl/demo_container/schemata
L3:0=f

$ echo "MB:0=30" | sudo tee -a /sys/fs/resctrl/demo_container/schemata
MB:0=30

3. Add a Process to the Isolation Group ​

Now let’s start a memory/cache-intensive test program (e.g., stress-ng) and add its PID to the isolation group:

$ stress-ng --cpu 2 --cache 2 &
[1] 12345

$ echo 12345 | sudo tee /sys/fs/resctrl/demo_container/tasks
12345

4. Monitor the Actual Effects of RDT Limits ​

After the limits are applied, how do we verify that stress-ng is actually being throttled in L3 cache and memory bandwidth? Here are two main monitoring methods.

Method 1: Observe Cache Miss Rate (LLC-misses) using perf

Since the L3 cache (LLC, Last Level Cache) allocated to this process has been reduced, its cache miss rate will inevitably rise. We can use the kernel’s built-in perf tool to monitor the PID:

$ sudo perf stat -e LLC-loads,LLC-load-misses -p 12345

 Performance counter stats for process id '12345':

        24,512,331      LLC-loads
        18,435,120      LLC-load-misses          #  75.21% of all LL-cache accesses

       10.001234560 seconds time elapsed

By comparing the data before and after putting the PID into demo_container, you will find that the ratio of LLC-load-misses (up to 75% in this example) increases significantly, indicating that the cache space has been successfully restricted and the process has to access main memory frequently.

Method 2: Monitor using pqos from intel-cmt-cat

Intel officially provides a toolkit called intel-cmt-cat (which includes pqos) specifically designed for managing and monitoring RDT metrics.

$ sudo pqos -p all:12345

[PID]      CORE   IPC      LLC MISSES   LLC[KB]    MBL[MB/s]  MBR[MB/s]
12345      2      0.45     854023       2048       345.5      0.0

pqos displays the Local/Remote memory bandwidth (MBL[MB/s]) consumed by the process in real-time. When you adjust the MB:0=30 value in the schemata, you can intuitively see the bandwidth rate drop on the monitoring dashboard, and the IPC (Instructions Per Cycle) will also decrease due to waiting for memory.

Through this monitoring, we not only "configured" the hardware limits but also visually "witnessed" the throttling effect at the physical level.

Conclusion ​

Compared to traditional kernel-space Cgroups isolation, Intel RDT provides an isolation solution directly based on the CPU hardware micro-architecture. It is the ultimate weapon to solve the "noisy neighbor" problem in multi-tenant container scenarios. runc’s native support for RDT allows users to configure micro-architectural resources as simply as configuring Cgroups when creating OCI containers.

In the next post, we will continue our journey through SPEC.md to explore a critical security defense line in containers—Linux Capabilities.