Skip to content

Kernel Keyring Exhaustion

Symptom

After starting and stopping many containers, podman run fails with:

crun: create keyring `<container>`: Disk quota exceeded: OCI runtime error

This is not a disk space issue — the misleading message comes from the kernel keyring subsystem returning EDQUOT.

Root cause

The OCI runtime (crun) creates a new session keyring for every container it starts. These keyrings are not reliably cleaned up when the container is removed, so they accumulate until the per-user quota is exhausted.

Linux defaults to 200 keys and 20 000 bytes per non-root user (/proc/sys/kernel/keys/maxkeys and maxbytes). A busy terok host that cycles many agent containers will hit this limit.

Impact on terok containers

The per-container kernel keyring is used by subsystems that terok executor containers never touch:

Subsystem Used by terok agents?
Kerberos ticket cache No
dm-crypt / LUKS No
eCryptfs No
IMA / EVM signing No
Generic keyctl storage No — seccomp blocks keyctl by default

Kernel keyrings are also not namespaced — they are separated by UID only, not by container. Rootless user namespaces and seccomp (both active in terok) provide the real isolation.

Disabling keyring creation has no effect on terok's functionality.

Workaround

Podman does not support disabling keyring creation per container — the setting is global in containers.conf. This means the workaround affects all containers on the host, not just terok's. If you run other workloads that depend on kernel keyrings (Kerberos, dm-crypt, etc.), evaluate the trade-off before applying.

Set keyring = false in the [containers] section of ~/.config/containers/containers.conf (create the file if it does not exist). This tells crun to skip the keyctl(KEYCTL_JOIN_SESSION_KEYRING) call entirely, eliminating the leak:

[containers]
keyring = false

Sickbay detection

terok sickbay warns when keyring creation is not disabled. Run it after installation to verify your setup.

References