Containers Under the Hood: Namespaces, Cgroups, and Layers

A container is not a small virtual machine. On Linux, it is an ordinary collection of processes whose view of the system and access to resources have been deliberately constrained. The kernel still schedules its threads, handles its system calls, caches its files, and routes its packets alongside host processes. What changes is the context in which the kernel answers questions such as “which processes exist?”, “what is mounted here?”, and “how much CPU may this group consume?”

That distinction is the foundation for reasoning about both container efficiency and container risk. Namespaces provide separate views of selected kernel resources. Cgroups account for and control resource use. Capabilities, system-call filters, and security modules narrow authority. Layered filesystems package an image efficiently, while a runtime assembles those mechanisms into a process environment. None of them creates a separate kernel.

The practical thesis is simple: container isolation is a composition of explicit kernel boundaries, not a single opaque wall. To operate containers safely, inspect each boundary, know what remains shared, and test limits under failure rather than inferring isolation from an image name or a command prompt.

A Container Begins as a Process #

An image is inert data: filesystem layers plus configuration such as an entry point, environment, and default user. A container begins only when a runtime asks the kernel to create a process and arranges its execution context. A high-level engine may pull and unpack the image, create networking, and manage logs. A lower-level runtime creates namespaces, joins cgroups, configures mounts and credentials, then executes the requested program. The process ultimately reaches the same scheduler and system-call interface as any other process.

This explains why container startup can be much lighter than booting a virtual machine. There is no guest kernel to initialize and usually no emulated hardware. It also explains why process behavior still matters. PID 1 inside a container has special signal and child-reaping responsibilities. If the entry point launches a shell that does not forward termination signals, an orchestrator’s graceful stop may not reach the application. If PID 1 never waits for exited children, zombies can accumulate in its PID namespace.

The process has two identities at once. Inside its PID namespace it may be PID 1; from the host it has an ordinary host PID. A namespaced hostname can differ from the host hostname, while the kernel release reported by uname still describes the shared host kernel. A root user inside the container is likewise a set of credentials interpreted through namespace and capability rules, not evidence of a private machine.

A useful mental model follows a system call. When the process calls open, kill, mount, or socket, the kernel evaluates the call using its namespace memberships, cgroup membership, credentials, capabilities, security policy, and the target object. Isolation is the result of those checks. The runtime is important because it installs the configuration, but it does not remain between every application instruction and the kernel.

Namespaces Change What a Process Can See #

A Linux process can belong to one namespace of each supported type. Processes in the same namespace share that resource view; processes in different namespaces may receive different answers from the kernel. The main types have distinct jobs:

Namespaces are selective, not transitive. A private PID namespace does not imply a private network stack or mount table. Sharing the host network removes network-namespace separation even if every other namespace remains distinct. Bind-mounting a host directory intentionally crosses the mount boundary. Joining another container’s PID namespace makes its processes visible. Every such option changes the isolation claim and should be reviewed as an authority grant.

PID namespaces are hierarchical. A process in a parent namespace can see descendants, while a process in a child cannot see processes that exist only in the parent. This asymmetry lets the host observe and manage containers. Mount namespaces store mount relationships, but filesystem object permissions still apply, and mount propagation settings determine whether later mounts cross namespace boundaries. Network namespaces hold their own interfaces and routes; a virtual Ethernet pair commonly connects one side in the container to one side on the host, where routing or network address translation carries traffic onward.

User namespaces deserve special attention. They can map container UID 0 to a non-root host UID, reducing the authority of “root” over host objects. Rootless container engines build on this mechanism. Mapping does not make every kernel subsystem namespaced, however, and host files exposed through bind mounts still have host ownership. Correct ID maps, filesystem support, and runtime policy must agree.

Cgroups Account for Work and Enforce Limits #

Namespaces answer visibility questions; cgroups answer resource-accounting and control questions. Without cgroups, a process hidden in its own PID namespace could still consume all host CPU time or memory. Cgroup v2 presents a hierarchy in which a process belongs to one cgroup, and controllers distribute resources through that tree.

CPU control can assign relative weight among busy siblings or impose a quota over a period. A quota does not reserve a dedicated core. It limits aggregate runnable time, and a multithreaded process can exhaust the quota early in a period and then be throttled. A container reporting low application throughput while its CPU throttling counter rises may be limit-bound even when the host has idle capacity elsewhere.

Memory control accounts charged pages and can set protective and hard thresholds. Crossing a hard memory limit does not behave like a polite allocation error in every application. Reclaim may intensify; if the group cannot get below its limit, the kernel can invoke an out-of-memory decision within the cgroup and kill a process. The memory attributed to a workload can include more than its language heap, such as native allocations, page cache, thread stacks, and kernel memory. Setting a runtime heap limit equal to the container limit leaves no room for those components.

I/O controllers can weight or cap access to block devices, subject to the storage stack’s support. PID limits bound the number of tasks and can contain fork bombs. Pressure Stall Information exposes time during which work is delayed for CPU, memory, or I/O, which often explains distress earlier than a utilization percentage.

Accounting is hierarchical. A child cannot escape a stricter ancestor, and a parent aggregates descendants. This matters in orchestration: a pod-level or service-level group may be the real constraint even when a particular container’s local settings look generous. It also means that moving a process into the wrong cgroup silently invalidates dashboards and limits. Verify membership from the host rather than trusting labels alone.

Privilege Is Reduced by Several Independent Gates #

Namespaces and resource limits are not a complete security policy. Linux traditionally gave UID 0 a broad set of powers; capabilities split many of those powers into named bits such as changing network configuration, sending signals across ownership boundaries, or performing selected administration. A container runtime can begin with a reduced bounding set, drop capabilities before execution, and prevent descendants from regaining them.

The safest default is to run as a non-root user and drop every capability the application does not require. Adding a capability should follow a concrete failed operation, not a general desire to make an image work. CAP_SYS_ADMIN is especially broad and gates many unrelated operations; granting it can erase much of the intended boundary. A “privileged” container commonly receives extensive devices and capabilities and should be treated as host-level code for threat modeling.

Seccomp filters restrict which system calls, and sometimes which argument patterns, a process may invoke. A web service does not normally need kernel-module loading or uncommon debugging interfaces. A default-deny profile with a maintained allowlist can reduce kernel attack surface, though profiles must follow real application behavior across architectures and runtime versions. A denied call may return an error or terminate the process depending on policy, so observability must distinguish policy denial from an application defect.

Linux Security Modules such as SELinux or AppArmor add mandatory access controls over files, processes, sockets, and other objects. They can block an operation even when Unix permissions and capabilities would permit it. The gates compose: passing one does not bypass the others. Read-only root filesystems, no-new-privileges, controlled device access, and carefully scoped bind mounts further reduce opportunities for persistence or host interaction.

Security also starts before execution. An image is a supply-chain input. Pinning an immutable digest identifies content more precisely than a mutable tag; signature and provenance policy can establish who produced it; scanning can identify known vulnerable packages. Those controls do not replace runtime isolation, because a trusted image can contain a defect and an untrusted request can exploit it.

Layers Package Files, While OverlayFS Creates a View #

Container images are usually represented as immutable filesystem layers. Each layer records additions, changes, and deletion markers relative to its parent. Images can share identical lower layers, reducing transfer and storage when many workloads use the same base. A content digest identifies layer bytes, so registries and local stores can verify and deduplicate them.

At runtime, an overlay filesystem commonly combines multiple read-only lower directories with one writable upper directory and presents a merged mount. Reading an untouched file resolves to a lower layer. The first modification to a lower-layer file triggers copy-up into the writable layer; future reads see the upper copy. Deleting a lower file creates a whiteout so the merged view hides it without changing the immutable image.

This model has consequences. A small edit to a large file may copy the complete file into the upper layer. Write-heavy databases can suffer both performance and durability surprises when placed in a disposable overlay instead of a purpose-built volume. Deleting a secret in a later image build step does not remove it from an earlier layer; anyone with the image can inspect that layer. Build secrets must avoid entering layers in the first place.

The writable layer belongs to a particular container and is normally ephemeral. Persistent data should live in an explicitly managed volume or external service. Bind mounts expose an existing host path and therefore couple the container to host layout and permissions. Named volumes give the engine more lifecycle control but still need backup, restore, capacity, and ownership policies. An image layer is a deployment artifact, not a database backup strategy.

Layer ordering also affects efficiency. Stable dependencies placed before frequently changing application files improve cache reuse during builds. That is a build concern, not a runtime isolation guarantee: two images sharing a base layer do not share mutable file contents through the overlay upper directory.

Worked Example: Trace One Container from Image to Process #

Consider an HTTP image whose configuration requests /app/server, UID 10001, port 8080, and a read-only root filesystem with /tmp mounted as a small writable temporary filesystem. The operator sets a one-CPU quota, a 256 MiB memory maximum, and a PID limit of 128. The runtime performs a sequence conceptually like this:

1. Resolve the image manifest and verify layer digests.
2. Mount the read-only layers with a fresh writable upper directory.
3. Create new PID, mount, network, UTS, IPC, and user namespaces.
4. Connect a virtual network interface and install routes.
5. Create a cgroup, configure CPU, memory, and PID controllers, and attach the process.
6. Prepare the root mount, /proc, the temporary mount, and any declared volume.
7. Set UID, groups, capabilities, seccomp, and mandatory-access labels.
8. Execute /app/server, which becomes PID 1 in the new PID namespace.

Now follow three events. First, the server writes /tmp/request-42. That write goes to the dedicated temporary mount, not an image layer. Second, it attempts to rewrite /app/config.json. The read-only root rejects the operation, which is a desired early failure if configuration was supposed to be immutable. Third, request load pushes total memory to the cgroup maximum. The language heap is only part of the charge; buffered responses and page cache contribute. Reclaim begins, pressure rises, and continued growth may result in a cgroup OOM kill.

From inside, /proc/1/status describes the server as PID 1 and /proc/self/cgroup identifies its resource group. From the host, the same task has a different PID, appears under the runtime’s cgroup subtree, and has namespace links that can be compared with host namespace links. The operator can inspect limit files and counters under the cgroup filesystem, including CPU throttling and memory events. Exact paths and tooling vary by runtime, but the kernel objects are the source of truth.

This walkthrough separates failure domains. A write error points toward mount policy. An EPERM from a system call may come from capabilities, seccomp, or a security module. A killed process accompanied by a memory event points toward cgroup pressure. Treating all three as “the container crashed” discards the mechanism needed to repair them.

What Containers Do Not Isolate #

All containers on a host normally share its kernel. A kernel vulnerability reachable through an allowed system call can threaten the boundary. Stronger isolation may require a sandboxed runtime, a minimal virtual machine per workload group, or a separate host, chosen according to threat and cost. Containers also share physical CPU caches, memory buses, storage devices, and networks, so noisy neighbors and side channels remain possible even with correct logical limits.

Resource limits are not reservations. A CPU maximum constrains use but does not promise prompt scheduling under host contention. A memory maximum prevents one group from growing without bound but does not guarantee low latency during reclaim. Network namespace separation does not encrypt traffic. An image’s read-only root does not protect a broadly mounted host socket; access to a container-engine control socket can amount to host control.

Configuration mistakes are a major failure mode: running privileged, sharing host PID or network namespaces, mounting sensitive host paths, retaining excessive capabilities, using writable executable paths, or exposing devices. Another is semantic mismatch. Applications may assume a stable hostname, write local state that disappears on replacement, ignore termination signals, or size themselves from host resources instead of cgroup limits.

Isolation is also not tenancy policy. Secrets, identity, authorization, network policy, and data encryption need explicit designs. A process that is perfectly contained can still use valid credentials to delete remote data. Conversely, an overly strict sandbox can break required behavior during an upgrade. The goal is not maximum denial in the abstract; it is the least authority compatible with a tested workload.

Verify Boundaries and Operate from Kernel Evidence #

Verification should test each claimed boundary. Start a diagnostic workload in a non-production environment and assert that it cannot see host processes, bind host ports unexpectedly, mount filesystems, access undeclared devices, write the root filesystem, or read neighboring secrets. Confirm that the configured UID maps to the intended host identity and that dropped capabilities are absent from the effective and bounding sets. Compare namespace identifiers between the process and host to ensure separation is real.

Exercise resource controls deliberately. Drive CPU until throttling occurs and confirm the service degrades in the expected way. Allocate memory gradually and observe pressure, reclaim, and memory.events; verify restart policy and data integrity after an OOM kill. Create tasks until the PID limit rejects growth. Fill writable storage and logs to ensure capacity failures do not corrupt persistent state or exhaust the host.

Test lifecycle behavior by sending the runtime’s normal termination signal. The application should stop accepting work, finish or reject in-flight operations according to policy, flush durable state, reap children, and exit before forced termination. Restart it on another host with only its declared volumes and remote dependencies; if state is missing, the deployment contract was incomplete.

Operational dashboards should combine application signals with cgroup and runtime evidence: CPU use and throttled time, memory current/max/events, pressure, PID count, filesystem capacity, network drops, restart reason, and image digest. Alert on sustained pressure and oldest unresolved failure, not only on host averages. Record effective security settings so incident responders can distinguish an exploit attempt from a sandbox denial.

Finally, make the isolation contract reviewable. For every workload, list namespaces intentionally shared, capabilities retained, system calls allowed, mounts and devices exposed, cgroup limits, persistent data, and required credentials. Re-run boundary tests when the base image, kernel, runtime, orchestrator policy, or application behavior changes.

Containers are efficient because they do not pretend to be separate machines. They reuse the host kernel while constructing narrower views and authority for ordinary processes. That design is powerful precisely when its pieces remain visible: namespaces define views, cgroups govern resources, security gates constrain privilege, layers package files, and operational tests prove that the composition still matches its claim.