gpu

gpu

Vendor-aware GPU passthrough for podman containers.

The run.gpus selector names the GPU vendors a container should see: "all" (or true) passes through every vendor the host detectably supports, while "nvidia" / "amd" / "intel" — alone or as a list — select vendors explicitly and fail loudly when the host cannot serve one. Several vendors in one selector simply merge their podman args, so a mixed-GPU host can hand all of its silicon to one container.

Per vendor, the mechanism follows the vendor's own container documentation — CDI when a host spec declares the vendor's device kind, the documented raw-device recipe otherwise:

• nvidia — CDI (nvidia.com/gpu) when present; otherwise the pre-CDI Container Toolkit OCI hook (env-triggered), and as a last resort raw /dev/nvidia* device nodes for images that carry their own driver userland.
• amd — CDI (amd.com/gpu, generated by amd-ctk) when present; otherwise the ROCm-documented /dev/kfd plus the AMD render nodes. The raw recipe is also what works on older podman releases (< 4.1) that predate CDI.
• intel — CDI (intel.com/gpu) when present; otherwise the Intel render nodes — the oneAPI/NEO stack needs nothing else.

Raw DRM grants are vendor-scoped: only the granted vendor's render nodes are mounted, never the whole /dev/dri (a mixed host's other GPUs — or its BMC's display device — stay invisible). Each granted node's /dev/dri/by-path symlink rides along read-only so PCI-ordered enumeration keeps working inside the container.

A :N suffix ("amd:1", "nvidia:0,nvidia:1") grants single devices instead of a vendor's whole fleet. On CDI hosts the index is passed to the vendor's spec (amd.com/gpu=1) and means whatever the vendor's tooling says it means. Without CDI the grant is best effort: terok orders a vendor's devices by PCI bus address — stable across launches (and across reboots for unchanged hardware) but not guaranteed to match rocm-smi/nvidia-smi numbering — mounts only the selected device nodes, and warns on stderr. Inside the container the selected card enumerates as device 0.

AMD and Intel render nodes are group-gated (render), so those two vendors additionally emit --group-add keep-groups — the rootless-podman way to keep the invoking user's host groups. That flag is honoured by crun only; check_gpu_error translates the runc failure into a hint.

GPU_VENDORS = ('nvidia', 'amd', 'intel') module-attribute

Recognised run.gpus vendor tokens, in canonical (emission) order.

__all__ = ['GPU_VENDORS', 'GpuConfigError', 'GpuGrant', 'GpuSelector', 'GpuVendor', 'check_gpu_available', 'check_gpu_error', 'detect_gpu_vendors', 'gpu_device_addresses', 'gpu_run_args', 'normalize_gpus'] module-attribute

GpuGrant = tuple[GpuVendor, int | None]

One normalized selector token: a vendor and an optional device index.

("amd", None) grants every AMD device; ("amd", 1) grants the host's second AMD device (PCI-bus-address order — see terok_sandbox.runtime.gpu for the ordering contract per passthrough tier).

GpuSelector = tuple[GpuGrant, ...] | Literal['all'] | None

Normalized run.gpus value: explicit grants, auto-detect, or off.

The vocabulary lives here, next to the field it types; the host-probing and podman-arg side of the story is terok_sandbox.runtime.gpu.

GpuVendor = Literal['nvidia', 'amd', 'intel']

GpuConfigError(message, *, hint=_CDI_HINT)

Bases: RuntimeError

GPU misconfiguration detected before or during container launch.

Store the remediation hint alongside the standard error message.

Source code in src/terok_sandbox/runtime/gpu.py

def __init__(self, message: str, *, hint: str = _CDI_HINT) -> None:
    """Store the remediation *hint* alongside the standard error *message*."""
    self.hint = hint
    super().__init__(message)

hint = hint instance-attribute

normalize_gpus(value)

Normalize a raw run.gpus value into a GpuSelector.

Accepts the shapes a YAML config or CLI flag produces: booleans, a single token ("all", "nvidia", "nvidia:0", …), a comma-separated token string, or a list of tokens. A :N suffix grants one device of that vendor instead of all of them; repeat the vendor to grant several ("nvidia:0,nvidia:1"). A whole-vendor token absorbs that vendor's indexed ones. Raises ValueError on an unknown vendor or malformed index so misconfiguration surfaces at parse time, not at launch.

Source code in src/terok_sandbox/config_schema.py

def normalize_gpus(value: bool | str | Sequence[str] | None) -> GpuSelector:
    """Normalize a raw ``run.gpus`` value into a [`GpuSelector`][terok_sandbox.config_schema.GpuSelector].

    Accepts the shapes a YAML config or CLI flag produces: booleans,
    a single token (``"all"``, ``"nvidia"``, ``"nvidia:0"``, …), a
    comma-separated token string, or a list of tokens.  A ``:N`` suffix
    grants one device of that vendor instead of all of them; repeat the
    vendor to grant several (``"nvidia:0,nvidia:1"``).  A whole-vendor
    token absorbs that vendor's indexed ones.  Raises ``ValueError`` on
    an unknown vendor or malformed index so misconfiguration surfaces
    at parse time, not at launch.
    """
    if value is None or value is False:
        return None
    if value is True:
        return "all"
    parts = [value] if isinstance(value, str) else list(value)
    tokens = [tok for part in parts for raw in part.split(",") if (tok := raw.strip().lower())]
    if not tokens:
        return None
    # dict.fromkeys deduplicates while keeping first-seen order; the
    # grants themselves are the (vendor, index) tuple keys.
    grants = list(dict.fromkeys(_parse_gpu_token(tok) for tok in tokens if tok != "all"))
    if "all" in tokens:
        return "all"
    whole_vendors = {vendor for vendor, index in grants if index is None}
    return tuple(
        (vendor, index)
        for vendor, index in sorted(
            grants, key=lambda g: (GPU_VENDORS.index(g[0]), g[1] if g[1] is not None else -1)
        )
        if index is None or vendor not in whole_vendors
    )

gpu_run_args(gpus=None)

Return podman run args passing the selected GPU grants through.

"all" resolves against detect_gpu_vendors and raises GpuConfigError when the host supports none — a project that opted into GPUs should never silently run without them. Explicit grants skip detection and raise per-vendor when their prerequisites are missing.

Source code in src/terok_sandbox/runtime/gpu.py

def gpu_run_args(gpus: GpuSelector = None) -> list[str]:
    """Return ``podman run`` args passing the selected GPU grants through.

    ``"all"`` resolves against [`detect_gpu_vendors`][terok_sandbox.runtime.gpu.detect_gpu_vendors]
    and raises [`GpuConfigError`][terok_sandbox.runtime.gpu.GpuConfigError] when the host
    supports none — a project that opted into GPUs should never silently
    run without them.  Explicit grants skip detection and raise
    per-vendor when their prerequisites are missing.
    """
    if not gpus:
        return []
    cdi_kinds = _declared_cdi_kinds()
    grouped = dict.fromkeys(_resolve_all(cdi_kinds)) if gpus == "all" else _group_grants(gpus)
    args = [
        arg
        for vendor, devices in grouped.items()
        for arg in _VENDOR_ARGS[vendor](cdi_kinds, devices)
    ]
    return _dedup_pairs(args)

detect_gpu_vendors()

Return the GPU vendors this host can pass through right now.

A vendor counts as detected when a CDI spec declares its device kind, or when its fallback-recipe markers are present — NVIDIA's legacy OCI hook or /dev/nvidiactl, AMD's /dev/kfd plus an AMD render node, an Intel render node.

Source code in src/terok_sandbox/runtime/gpu.py

def detect_gpu_vendors() -> frozenset[GpuVendor]:
    """Return the GPU vendors this host can pass through right now.

    A vendor counts as detected when a CDI spec declares its device
    kind, or when its fallback-recipe markers are present — NVIDIA's
    legacy OCI hook or ``/dev/nvidiactl``, AMD's ``/dev/kfd`` plus an
    AMD render node, an Intel render node.
    """
    return frozenset(_resolve_all(_declared_cdi_kinds(), strict=False))

check_gpu_available()

Return True when a CDI spec declares the nvidia.com/gpu kind.

Wizards call this to decide whether to offer the NVIDIA base image; the on-launch check_gpu_error path is the authoritative one and stays in place. Any failure (missing podman, missing CDI dirs, unreadable spec) collapses to False so callers can treat this as a pure yes/no signal.

Source code in src/terok_sandbox/runtime/gpu.py

def check_gpu_available() -> bool:
    """Return ``True`` when a CDI spec declares the ``nvidia.com/gpu`` kind.

    Wizards call this to decide whether to offer the NVIDIA base image;
    the on-launch [`check_gpu_error`][terok_sandbox.runtime.gpu.check_gpu_error]
    path is the authoritative one and stays in place.  Any failure
    (missing podman, missing CDI dirs, unreadable spec) collapses to
    ``False`` so callers can treat this as a pure yes/no signal.
    """
    return _NVIDIA_GPU_KIND in _declared_cdi_kinds()

gpu_device_addresses()

PCI bus addresses of each vendor's GPUs, in terok index order.

The authoritative answer to "which card does amd:1 grant" on the raw (non-CDI) tiers: position N in a vendor's tuple is the device that vendor:N selects. On CDI hosts the vendor's spec owns the numbering instead and this map is informational only.

Source code in src/terok_sandbox/runtime/gpu.py

def gpu_device_addresses() -> dict[GpuVendor, tuple[str, ...]]:
    """PCI bus addresses of each vendor's GPUs, in terok index order.

    The authoritative answer to "which card does ``amd:1`` grant" on the
    raw (non-CDI) tiers: position ``N`` in a vendor's tuple is the
    device that ``vendor:N`` selects.  On CDI hosts the vendor's spec
    owns the numbering instead and this map is informational only.
    """
    try:
        nvidia = tuple(sorted(d.name for d in _NVIDIA_PROC_GPUS.iterdir()))
    except OSError:
        nvidia = ()
    return {
        "nvidia": nvidia,
        "amd": tuple(addr for addr, _ in _vendor_render_nodes(_AMD_PCI_VENDOR)),
        "intel": tuple(addr for addr, _ in _vendor_render_nodes(_INTEL_PCI_VENDOR)),
    }

check_gpu_error(exc)

Raise GpuConfigError if exc looks like a GPU launch issue.

Recognises CDI misconfiguration, the crun-only keep-groups rejection, and legacy NVIDIA hook failures (typically the rootless no-cgroups requirement); does nothing when the error matches none of them. Defensively handles both bytes and str stderr so callers that ran subprocess with text=True are not punished with an AttributeError on .decode.

Source code in src/terok_sandbox/runtime/gpu.py

def check_gpu_error(exc: subprocess.CalledProcessError) -> None:
    """Raise [`GpuConfigError`][terok_sandbox.runtime.gpu.GpuConfigError] if *exc* looks like a GPU launch issue.

    Recognises CDI misconfiguration, the crun-only ``keep-groups``
    rejection, and legacy NVIDIA hook failures (typically the rootless
    ``no-cgroups`` requirement); does nothing when the error matches
    none of them.  Defensively
    handles both ``bytes`` and ``str`` stderr so callers that ran
    subprocess with ``text=True`` are not punished with an
    ``AttributeError`` on ``.decode``.
    """
    stderr_raw = exc.stderr or b""
    stderr = (
        stderr_raw.decode(errors="replace") if isinstance(stderr_raw, bytes) else str(stderr_raw)
    )
    hint = None
    if any(pat in stderr for pat in _KEEP_GROUPS_ERROR_PATTERNS):
        hint = _KEEP_GROUPS_HINT
    elif any(pat in stderr for pat in _NVIDIA_HOOK_ERROR_PATTERNS):
        hint = _NVIDIA_HOOK_HINT
    elif any(pat in stderr for pat in _CDI_ERROR_PATTERNS):
        hint = _CDI_HINT
    if hint is not None:
        msg = f"Container launch failed (GPU misconfiguration):\n{stderr.strip()}\n\n{hint}"
        raise GpuConfigError(msg, hint=hint) from exc