Enroot + Pyxis + SPANK: A Practical Architecture and Operations Guide for Slurm Containers

Quick definition. Enroot is the unprivileged container runtime, Pyxis is the Slurm-facing adapter, and SPANK is the Slurm plug-in framework that gives Pyxis a place in the job-launch lifecycle. Together, they let users launch HPC container workloads with familiar srun and sbatch commands rather than managing a separate container service. 1 3

Document details	Value
Audience	Slurm administrators, ML platform engineers, HPC users, and SREs
Prerequisites	Working Slurm cluster; basic Linux, Slurm, and container-image familiarity
Scope	Architecture, implementation model, deployment, usage, operational design, and troubleshooting
Version note	Validate commands and configuration keys against the Enroot, Pyxis, and Slurm versions actually deployed at your site.

Why this stack exists

GPU and MPI workloads need the portability of container images, but HPC schedulers still need to allocate nodes, CPUs, memory, GPUs, and interconnect resources. The Enroot–Pyxis–SPANK stack deliberately separates those responsibilities. Slurm remains the scheduler and resource authority. Pyxis interprets container options during Slurm launch. Enroot supplies the unprivileged sandbox and the image-to-rootfs mechanics. 1 3

Enroot is not a daemon-managed, general-purpose isolation boundary comparable to a virtual machine. Its own project describes it as an enhanced unprivileged chroot that retains filesystem separation while removing much of the isolation commonly associated with containers. This makes its lightweight model well suited to trusted HPC batch workloads, while requiring administrators to design mounts, device visibility, network policy, and shared storage carefully. 1

Design principle: use the container to make the software environment reproducible; use Slurm to make resource allocation and policy reproducible.

The architecture at a glance

Figure 1. Pyxis appears in both the client-side and compute-node-side SPANK contexts. It converts --container-* arguments into Enroot actions while Slurm continues to launch and account for the task.

The diagram highlights a fact that is easy to miss: a Pyxis launch is not simply srun calling a runtime on the login node. SPANK plug-ins can run in several contexts, including local context inside srun and remote context inside slurmstepd on each allocated compute node. That split permits Pyxis to register and interpret user options before the task is launched, then prepare the container around the task on the compute node. 3

Layer	Primary responsibility	Where it acts
Slurm	Schedules resources, starts job steps, launches tasks, records job state	Controller, srun/sbatch, slurmd, and slurmstepd
SPANK	Defines stackable job-launch extension points and lifecycle callbacks	Client and daemon execution contexts
Pyxis	Adds container arguments to Slurm commands and maps them to Enroot operations	srun/sbatch and compute-node task launch
Enroot	Imports image content, creates/starts sandboxes, applies mounts, environment, and hooks	Compute node, as the submitting user
Site configuration	Chooses storage paths, hooks, device exposure, PMI/PMIx behavior, and security posture	Every compute node, consistently

What SPANK contributes: the launch-time extension mechanism

SPANK—short for the Slurm Plug-in Architecture for Node and job (K)control—allows plug-ins to alter Slurm job-launch behavior without modifying Slurm source. A plug-in is compiled against spank.h, enabled through plugstack.conf, and loaded at runtime during a launch. SPANK plug-ins should be rebuilt when upgrading to a new Slurm major release because ABI compatibility is not guaranteed. 3

The key remote callbacks for understanding container behavior are slurm_spank_init, slurm_spank_user_init, slurm_spank_task_init, and exit callbacks. In simplified terms, the first phases set up plug-in state, user_init runs after privileges are temporarily dropped, and task_init occurs immediately before the user task is executed. A launch-time failure therefore can arise from either Slurm policy or the container preparation that is coupled to those callbacks. 3

Lifecycle stage	Meaning for a container plug-in	Operational implication
Local option processing	srun recognizes --container-* flags	Users receive a native Slurm CLI rather than a wrapper command
Remote initialization	slurmstepd loads the plug-in on the allocated node	Each node needs a compatible Pyxis binary and Enroot installation
User initialization	Pyxis can determine container state and prepare the sandbox as the job user	Runtime paths, permissions, and mount policies matter here
Per-task initialization	Task-specific namespaces and execution are finalized before execve()	MPI rank and GPU visibility must remain coherent at this boundary
Exit handling	Task status is collected and the step ends	Temporary containers may be removed; named containers require explicit lifecycle planning

What Enroot contributes: a daemonless, unprivileged runtime

Enroot imports container images, generally materializes them into a SquashFS artifact, creates a sandbox, and starts a command inside it. Its documented command surface includes import, create, start, exec, export, list, and remove. A minimal standalone flow is enroot import, enroot create, and enroot start. 1

Enroot’s paths are part of the architecture, not implementation trivia. By default, ENROOT_CACHE_PATH holds images and credentials cache, ENROOT_DATA_PATH holds user container storage, ENROOT_RUNTIME_PATH is working state, and ENROOT_TEMP_PATH provides temporary space. Environment variables override enroot.conf, allowing a site to establish sensible defaults yet override them in controlled job environments when needed. 5

Path category	Typical contents	Practical design objective
ENROOT_CACHE_PATH	Downloaded layers, image and credential cache	Persistent node-local storage where possible; avoid synchronized large-image imports through a metadata-sensitive shared filesystem
ENROOT_DATA_PATH	Container root filesystems and saved container state	Fast local storage or job-private scratch when lifecycle isolation matters
ENROOT_RUNTIME_PATH	Runtime working files	Node-local, short-lived storage; sites often choose memory-backed or local runtime paths
ENROOT_TEMP_PATH	Temporary work	Capacity sufficient for import/create operations; observe local disk pressure

Operational rule: cache sharing solves repeated downloads; it does not remove the need to manage per-job rootfs state and cleanup.

Configuration and hooks

Enroot supports system-wide and user-specific configuration. Under its configuration roots, environ.d exports environment values into containers, mounts.d defines filesystem mounts, and hooks.d contains pre-start shell hooks. Hooks run before the runtime switches to the final container root and are therefore a powerful administrative integration mechanism that deserves normal production change control. 5

The standard hook set includes GPU support through libnvidia-container and Mellanox HCA support through the Mellanox hook. The latter is significant for distributed training and MPI sites: do not assume that a container image alone supplies the host’s IB/RDMA integration. Confirm that the appropriate host driver and hook policy is active, then validate with a small multi-node test. 6

What Pyxis contributes: Slurm-native container ergonomics

Pyxis is a SPANK plug-in for Slurm that allows unprivileged users to run containerized tasks through srun. The project requires Enroot and exposes its container controls as direct Slurm options. Its documented image parameter accepts an Enroot URI or a SquashFS image path located on the remote host filesystem. 2

Pyxis option	Purpose	Example
--container-image	Selects a registry image or remote SquashFS file	--container-image=almalinux:9
--container-mounts	Binds host paths into the container	--container-mounts=/lustre:/lustre,/datasets:/datasets:ro
--container-workdir	Selects the in-container working directory	--container-workdir=/workspace
--container-env	Preserves nominated host variables before the image entrypoint	--container-env=MASTER_ADDR,MASTER_PORT
--container-mount-home / --no-container-mount-home	Controls home-directory mounting, subject to site setup	--no-container-mount-home
--container-writable	Requests a writable container filesystem	--container-writable
--container-name	Names a saved container; existing named state can be reused	--container-name=experiment-rootfs
--container-save	Exports final container state to a SquashFS file	--container-save=/scratch/me/result.sqsh
--container-unshare	Requests new net, ipc, and/or uts namespaces	--container-unshare=ipc,uts

Pyxis documents an important environment semantic: exported host variables are normally made available after the image entrypoint, and values already present in the image take precedence. --container-env is therefore the correct mechanism when a named host value must be set before the entrypoint. Avoid relying on an informal assumption that every host-side environment value automatically overrides the image. 2

Administrator implementation: a deployment checklist

The exact package commands vary by distribution, but the dependency order should not. Install and test Enroot on each compute-node image, build Pyxis against the same Slurm release deployed by the cluster, enable Pyxis through Slurm’s plugstack configuration, and restart slurmd so that the daemon loads the new configuration. Pyxis specifically warns that compiling against a different Slurm release can yield Incompatible plugin version. 2

Compute node prerequisites
├── Slurm slurmd/slurmstepd matching the cluster release
├── Enroot runtime and required helpers
├── Pyxis built against this Slurm release
├── /etc/slurm/plugstack.conf or plugstack.conf.d entry for Pyxis
├── /etc/enroot/enroot.conf plus reviewed storage paths
└── Approved Enroot hooks for GPU, HCA, PMIx, and site integrations

A conventional Pyxis install enables the supplied pyxis.conf through /etc/slurm/plugstack.conf.d/ and restarts slurmd. The Pyxis setup guide also recommends that administrators customize Enroot configuration for their cluster and documents optional PMIx/PyTorch-related hooks for multi-node workloads. 2

Implementation decision	Recommended starting point	Why it matters
Slurm/Pyxis compatibility	Build Pyxis against the installed Slurm release	Prevents plug-in load failure
Image cache	Persistent local disk per node	Reduces repeated registry pulls and shared-storage storms
Runtime and rootfs state	Local fast storage; consider job-private state for strict lifecycle isolation	Reduces startup latency and cross-job state leakage risk
Home directory	Default to least privilege; let users opt in only if justified	Avoids unintentionally exposing credentials and configuration
Writable rootfs	Prefer explicit --container-writable or a documented site policy	Persistent writable state needs capacity, ownership, and cleanup controls
GPU and IB hooks	Enable only after validation on the deployed driver stack	Containerized GPU/MPI behavior depends on the host integration
PMIx	Test bare metal first, then test containerized multi-node jobs	Preserves the scheduler’s MPI launch semantics

A minimal configuration shape

The following is illustrative rather than a copy-and-paste production policy. The authoritative Enroot configuration names and defaults are documented upstream. 5

# /etc/enroot/enroot.conf — example site policy
ENROOT_RUNTIME_PATH  /run/enroot/user-$(id -u)
ENROOT_CACHE_PATH    /local/enroot-cache/user-$(id -u)
ENROOT_DATA_PATH     /local/enroot-data/user-$(id -u)
ENROOT_MOUNT_HOME    n
ENROOT_RESTRICT_DEV  y

The Pyxis setup guide supplies a similar example and discusses local runtime space, persistent local cache, and local data paths as cluster design choices. Treat the values as workload- and filesystem-dependent: a small single-user test cluster and a busy multi-tenant GPU partition should not necessarily share the same policy. 4

User workflow 101

1. Learn the raw Enroot model

Running Enroot directly is useful for image and hook troubleshooting because it separates runtime behavior from Slurm behavior. The upstream usage guide gives this core sequence. 7

# Fetch an OCI/Docker-style image and produce a local SquashFS artifact.
enroot import docker://ubuntu

# Create an unprivileged sandbox from the artifact.
enroot create ubuntu.sqsh

# Start a command in the sandbox.
enroot start ubuntu bash -lc 'cat /etc/os-release'

2. Run the same idea through Slurm and Pyxis

With Pyxis enabled, the container declaration stays next to the scheduler request. This small test verifies image import, container launch, and basic command execution on a compute node. 2

srun --container-image=almalinux:9 \
     bash -lc 'grep PRETTY_NAME /etc/os-release'

3. Bind persistent project data deliberately

Container filesystems should not be treated as the default home for training data, checkpoints, or final results. Bind the required project locations explicitly, and use read-only mounts for immutable datasets where practical. 2

srun --gpus=1 \
     --container-image=nvcr.io#nvidia/pytorch:24.09-py3 \
     --container-mounts=/lustre/projects/team-a:/workspace,/lustre/datasets:/datasets:ro \
     --container-workdir=/workspace \
     python train.py --data /datasets/corpus

4. Submit a batch job

In an #SBATCH directive, escape the # in an NGC-style image reference because # starts a Slurm directive comment. 2

#!/bin/bash
#SBATCH --job-name=pyxis-101
#SBATCH --nodes=1
#SBATCH --gpus=1
#SBATCH --time=00:15:00
#SBATCH --output=slurm-%j.out
#SBATCH --container-image=nvcr.io\#nvidia/pytorch:24.09-py3
#SBATCH --container-mounts=/lustre/projects/team-a:/workspace
#SBATCH --container-workdir=/workspace
#SBATCH --no-container-mount-home

python - <<'PY'
import torch
print('torch:', torch.__version__)
print('cuda available:', torch.cuda.is_available())
PY

5. Scale to multi-node carefully

Pyxis supports multi-node MPI workloads through PMI2 or PMIx when Slurm provides support. The right command line depends on the site’s Slurm MPI configuration and enabled Enroot hooks; do not assume that a single-node GPU command is proof of multi-node readiness. 2

srun --nodes=2 --ntasks-per-node=4 --gpus-per-node=4 \
     --mpi=pmix \
     --container-image=nvcr.io#nvidia/pytorch:24.09-py3 \
     --container-mounts=/lustre/projects/team-a:/workspace \
     --container-workdir=/workspace \
     bash -lc 'hostname; python -c "import torch; print(torch.cuda.device_count())"'

Named containers and writable state: useful, but a lifecycle decision

Pyxis says that unnamed containers are removed after the Slurm task completes, while named containers are not. A named container can also cause an existing container to be used and skip image import. This can be a performance or iterative-development feature, but it changes the object from disposable task state into persistent mutable state. 2

Pattern	Best use	Main risk	Control
Unnamed, read-only container	Most batch inference and training runs	Re-import/start overhead	Use node-local cache and stable image tags or digests
Unnamed, writable container	Short experiments that need package installation during a task	Changes disappear when the task ends	Export explicitly with --container-save if state must be retained
Named container	Intentional reuse across compatible steps	Stale files, storage growth, and cross-step coordination hazards	Use a unique naming convention and explicit enroot remove cleanup
Global/shared writable rootfs policy	Highly controlled platform workflow	Cross-job persistence, permission confusion, cleanup and security exposure	Avoid as a default; scope storage by user/job and document teardown

Safety rule: a container name is not a locking protocol. If several concurrent steps attempt to attach to, start, or remove the same named container, treat the behavior as a distributed lifecycle problem and synchronize the workflow explicitly.

A useful mental model is a three-state race. If an owner has already stopped before an attaching step checks the container, the new step can follow a non-attach creation/reuse path. If the owner exits after the check but before the attach completes, an ENOENT-style failure is possible. If the owner survives the attach window, attachment succeeds. The exact implementation and version details must be tested locally, but the general mitigation is stable: avoid unnecessary shared named-container lifetimes, serialize a handoff when required, and remove persistent state deliberately after the workflow.

Performance engineering: where large images hurt

A 25 GB image is not merely a bigger unit of storage. On an N-node job, a naïve cold start can multiply registry download work, shared-filesystem metadata operations, decompression, and rootfs creation. The right response is normally not “put every Enroot path on the shared filesystem.” Instead, distinguish durable image cache, short-lived runtime state, and per-job rootfs state. Enroot’s documented cache, data, runtime, and temporary paths give administrators the controls to make that distinction. 5

Symptom	Likely pressure point	First diagnostic	Typical corrective direction
Many nodes import the same image at once	Registry bandwidth, shared cache metadata, cold local cache	Compare cold versus warm start and count imports per node	Pre-stage images or use a persistent node-local cache
Start time rises with rank count on one node	Rootfs work, local I/O, mount setup, hooks	Measure time to first process for 1, 2, 4, and 8 local ranks	Reduce duplicated work; evaluate the site’s image-mount mode and local storage
Rootfs storage grows without bound	Named/writable container data is retained	Inspect ENROOT_DATA_PATH by user, job, and container name	Add job teardown and retention policy
Slower distributed startup than bare metal	PMIx/IB hooks or environment integration	Run the smallest two-node PMIx and NCCL smoke tests	Verify approved PMIx and HCA hook configuration

Recent Pyxis releases include direct-SquashFS/squashfuse-related functionality. Whether it is appropriate depends on the installed Pyxis and Enroot releases, user-namespace and overlay support in the node kernel, and the workload’s startup profile. Treat it as a benchmarked site option, not a universal replacement for rootfs extraction or named-container reuse. Measure cold start, time to application-ready state, concurrent-rank behavior, and steady-state throughput before changing the default.

Troubleshooting playbook

Observation	Likely cause	What to check	First safe action
Incompatible plugin version	Pyxis was built against a different Slurm release	Pyxis build headers and deployed slurmd version	Rebuild/reinstall Pyxis against the deployed Slurm release, then restart slurmd 2
srun does not recognize --container-image	Pyxis is not enabled in the client/plugstack path	plugstack.conf includes and Pyxis install	Confirm the configured plugstack and deployment steps 2
Image import repeats or saturates storage	Cache path is nonpersistent, remote, or poorly shared	ENROOT_CACHE_PATH and per-node cache warmth	Move cache toward suitable persistent local storage; test one node first 4
Home files or credentials appear unexpectedly	System-level Enroot mount-home policy is active	ENROOT_MOUNT_HOME, mount configuration, job arguments	Use least-privilege default and --no-container-mount-home where required 2 6
GPU not visible inside the container	NVIDIA integration/hook or allocation mismatch	Slurm GPU allocation, nvidia-smi, NVIDIA hook policy	Test an allocated one-GPU job before debugging framework code 6
IB/NCCL/MPI breaks only in containers	Host HCA/PMIx integration not injected or configuration mismatch	Bare-metal versus containerized two-node test, Enroot hooks	Validate the site’s Mellanox and PMIx integration path 4
Named container attach fails intermittently	Owner/attacher lifecycle race or stale state	Container PID/state and timing around concurrent steps	Avoid shared name where possible; serialize; clean up stale state
Disk usage keeps increasing	Persistent named or writable rootfs state	ENROOT_DATA_PATH, container names, cleanup logs	Add explicit enroot remove to the workflow’s teardown 7

A production-ready baseline

A mature platform begins with a deliberately narrow default: unprivileged, unnamed, read-only containers; explicit project-data mounts; a persistent node-local image cache; job-local runtime and rootfs paths; reviewed GPU/HCA hooks; and a small smoke-test suite for basic, GPU, and two-node MPI launches. Add writable rootfs, named-container reuse, and alternative image-mount modes only where the workload has a measured need and the site has a matching cleanup, observability, and incident-response story. 1 4 6

This approach preserves the main advantage of the stack: users get a portable user space and a concise Slurm interface, while administrators retain predictable resource control and avoid turning an HPC batch system into an unbounded, stateful container platform.

---

References