Skip to content

supervisor

supervisor

Per-container supervisor — one parent process, one child per service.

The parent (run_supervisor) launches the terok-vault proxy, SSH signer, git gate server, clearance hub, and verdict server each in its own hardened child process (children, spawned via launch_child), so a bug in one service can't reach another's address space. Spawned by the OCI createRuntime hook (the terok_sandbox/resources/hooks/supervisor_hook.py script) through the restart-loop wrapper (terok_sandbox/resources/supervisor_wrapper.py); exits when podman wait returns.

The entry point is run_supervisor — terok-sandbox supervisor <container_id> <sidecar_path> invokes it under asyncio.run with both arguments.

__all__ = ['SidecarConfig', 'SupervisorPaths', 'load_sidecar', 'run_supervisor'] module-attribute

SidecarConfig(container_name, ipc_mode, db_path, runtime_dir, scope_id=None, project_id='', task_id='', tcp_port=None, ssh_signer_port=None, gate_port=None, gate_base_path=None, gate_token=None, dossier_path=None, allow_debugger=False) dataclass

Per-container config the supervisor reads from the sidecar JSON.

Written once at container-creation time by write_sidecar (terok-executor routes through the same writer) and read back by the OCI hook on every podman start. Keyed by container name (<state>/sidecar/<name>.json); the terok.sandbox.sidecar annotation pins the absolute path. The file persists across stop/start cycles — its ports and tokens must keep matching the container's immutable env — and is removed at real teardown by remove_container_state.

container_name instance-attribute

ipc_mode instance-attribute

db_path instance-attribute

runtime_dir instance-attribute

/run/user/<host_uid>/terok/sandbox — pinned by the launch path because the supervisor cannot re-derive it from inside crun's rootless user namespace (its os.getuid() is 0 there, which misroutes generic resolvers to the root-only /run/terok).

scope_id = None class-attribute instance-attribute

project_id = '' class-attribute instance-attribute

task_id = '' class-attribute instance-attribute

tcp_port = None class-attribute instance-attribute

Per-container TCP port for the vault proxy in TCP mode. None in socket mode (the path is derived from the container ID, not carried here).

ssh_signer_port = None class-attribute instance-attribute

Per-container TCP port for the SSH signer in TCP mode. None in socket mode.

gate_port = None class-attribute instance-attribute

Per-container TCP port for the git gate in TCP mode. None in socket mode.

gate_base_path = None class-attribute instance-attribute

Directory holding the shared per-project bare mirrors (<gate_base_path>/<project_id>.git). None when the gate is not wired for this container.

gate_token = None class-attribute instance-attribute

The single token the gate validates. Travels only via the sidecar; None when the gate is not wired.

dossier_path = None class-attribute instance-attribute

allow_debugger = False class-attribute instance-attribute

Debug mode — the children leave themselves ptrace-able instead of clearing the dumpable flag, so a debugger can attach. False (fully hardened) unless the launch path opted the task in.

SupervisorPaths(container_id, container_runtime_dir, vault_socket, ssh_signer_socket, gate_socket, clearance_socket, events_socket, verdict_socket, control_socket, log_path) dataclass

Resolved per-container socket / log / pid locations.

Computed once at supervisor startup from the container ID and the runtime/state dirs the sidecar config doesn't carry directly.

container_id instance-attribute

container_runtime_dir instance-attribute

Per-container directory holding vault.sock and ssh-agent.sock. Keyed on container_name (which the launch path knows before podman run so it can pre-create the dir and bind-mount it as /run/terok/ inside the container). Different containers get different host dirs; the in-container view of these sockets is always /run/terok/.

vault_socket instance-attribute

ssh_signer_socket instance-attribute

gate_socket instance-attribute

Per-container git-gate Unix socket inside container_runtime_dir (= the in-container /run/terok). Used only in socket mode; in TCP mode the gate binds a loopback port instead.

clearance_socket instance-attribute

events_socket instance-attribute

Per-container ingester socket the shield reader and shield up/down push raw line-JSON to. Distinct from clearance_socket (the varlink subscriber socket operator UIs glob): the reader speaks line-JSON, not varlink, so the produce and subscribe roles need separate sockets.

verdict_socket instance-attribute

control_socket instance-attribute

log_path instance-attribute

for_container(container_id, container_name, sidecar_path, runtime_dir) classmethod

Build the per-container path bundle.

Both anchors come from the launch path — neither is re-resolved inside the supervisor:

  • runtime_dir (/run/user/<host_uid>/terok/sandbox) for per-container sockets; carried in the sidecar because the rootless user namespace makes generic is_root-based resolvers (terok_util.namespace_runtime_dir) misroute to /run/terok.
  • sidecar_path's grandparent for the persistent log file; honours whatever paths.root resolved to when the launch path wrote the sidecar.

Sockets carry the 12-char short container ID (podman's display convention) rather than the full UUID — AF_UNIX's sun_path is 108 bytes including the null terminator, and <terok-runtime>/clearance/<64-char-uuid>.sock lands at or past that limit. Twelve characters of hex give 48 bits of entropy, well past the no-collisions-within-one-host bar. Logs keep the full UUID because they live on the filesystem with no AF_UNIX limit and the full UUID is easier to grep.

Clearance / verdict / control sockets live at the cross-package <terok>/ runtime root (parent of the sandbox-namespaced runtime_dir) because they're owned by terok-clearance semantically and consumed by every package that subscribes (terok-shield's NFLOG reader, terok-clearance TUI, …). Sandbox-specific sockets (vault, ssh-agent) live in a per-container runtime_dir/run/<short_id>/ directory the launch path bind-mounts at /run/terok/ inside the container — keeping every container's sockets distinct on the host so concurrent containers don't collide.

Source code in src/terok_sandbox/supervisor/sidecar.py 
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
@classmethod
def for_container(
    cls,
    container_id: str,
    container_name: str,
    sidecar_path: Path,
    runtime_dir: Path,
) -> SupervisorPaths:
    """Build the per-container path bundle.

    Both anchors come from the launch path — neither is re-resolved
    inside the supervisor:

    * *runtime_dir* (``/run/user/<host_uid>/terok/sandbox``) for
      per-container sockets; carried in the sidecar because the
      rootless user namespace makes generic ``is_root``-based
      resolvers (``terok_util.namespace_runtime_dir``) misroute to
      ``/run/terok``.
    * *sidecar_path*'s grandparent for the persistent log file;
      honours whatever ``paths.root`` resolved to when the launch
      path wrote the sidecar.

    Sockets carry the **12-char short container ID** (podman's
    display convention) rather than the full UUID — AF_UNIX's
    ``sun_path`` is 108 bytes including the null terminator, and
    ``<terok-runtime>/clearance/<64-char-uuid>.sock`` lands at or
    past that limit.  Twelve characters of hex give 48 bits of
    entropy, well past the no-collisions-within-one-host bar.
    Logs keep the full UUID because they live on the filesystem
    with no AF_UNIX limit and the full UUID is easier to grep.

    Clearance / verdict / control sockets live at the
    cross-package ``<terok>/`` runtime root (parent of the
    sandbox-namespaced *runtime_dir*) because they're owned by
    terok-clearance semantically and consumed by every package
    that subscribes (terok-shield's NFLOG reader, terok-clearance
    TUI, …).  Sandbox-specific sockets (vault, ssh-agent) live in
    a per-container ``runtime_dir/run/<short_id>/`` directory the
    launch path bind-mounts at ``/run/terok/`` inside the
    container — keeping every container's sockets distinct on
    the host so concurrent containers don't collide.
    """
    short_id = container_id[:12]
    clearance_root = runtime_dir.parent  # <terok>/sandbox/  →  <terok>/
    state_anchor = sidecar_path.parent.parent  # <root>/sidecar/<name>.json → <root>
    # vault + ssh-agent are keyed on container_name (known at
    # launch time, before podman assigns the ID) so the launch
    # path can pre-create the dir and bind-mount it.  Clearance /
    # verdict / control use the 12-char container ID short prefix
    # because they're cross-package (shield's NFLOG reader keys
    # on it too).
    container_runtime_dir = runtime_dir / "run" / container_name
    return cls(
        container_id=container_id,
        container_runtime_dir=container_runtime_dir,
        vault_socket=container_runtime_dir / "vault.sock",
        ssh_signer_socket=container_runtime_dir / "ssh-agent.sock",
        gate_socket=container_runtime_dir / "gate-server.sock",
        clearance_socket=clearance_root / "clearance" / f"{short_id}.sock",
        events_socket=clearance_root / "events" / f"{short_id}.sock",
        verdict_socket=clearance_root / "verdict" / f"{short_id}.sock",
        control_socket=clearance_root / "control" / f"{short_id}.sock",
        log_path=state_anchor / "logs" / f"{container_id}.log",
    )

run_supervisor(container_id, sidecar_path, container_pid=None) async

Launch, monitor, and tear down the per-container child processes.

container_pid is the container's init host-PID, handed down from the createRuntime OCI hook. When present it is the authoritative container-death signal: the supervisor watches it directly (via pidfd), so it tears down the instant the container exits even where nested podman wait is blind (crun handing the hook the container's env). None (older hook, or a runtime that didn't supply it) falls back to the podman wait watch alone.

Lifecycle:

  1. Load the sidecar JSON from sidecar_path; bail with exit code 2 on parse / missing.
  2. Launch one hardened child per service (skipping the git gate when the sidecar didn't wire it). A child that never spawns is logged and skipped; when no child spawns, exit code 3 hands the wrapper its retry.
  3. Install SIGTERM / SIGINT handlers so a host-side terok-sandbox supervisor invocation stops cleanly with Ctrl-C.
  4. Await podman wait <container_id> racing the children's own liveness and the stop signal. On any of them, terminate the children in reverse launch order and return.

The function is the sole parent entry point; the CLI verb terok-sandbox supervisor invokes it via asyncio.run. The per-service work now lives in run_child, one process each.

Source code in src/terok_sandbox/supervisor/main.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
async def run_supervisor(
    container_id: str, sidecar_path: Path, container_pid: int | None = None
) -> int:
    """Launch, monitor, and tear down the per-container child processes.

    *container_pid* is the container's init host-PID, handed down from the
    ``createRuntime`` OCI hook.  When present it is the **authoritative**
    container-death signal: the supervisor watches it directly (via
    ``pidfd``), so it tears down the instant the container exits even where
    nested ``podman wait`` is blind (crun handing the hook the container's
    env).  ``None`` (older hook, or a runtime that didn't supply it) falls
    back to the ``podman wait`` watch alone.

    Lifecycle:

    1. Load the sidecar JSON from *sidecar_path*; bail with exit code 2
       on parse / missing.
    2. Launch one hardened child per service (skipping the git gate when
       the sidecar didn't wire it).  A child that never spawns is logged
       and skipped; when *no* child spawns, exit code 3 hands the wrapper
       its retry.
    3. Install SIGTERM / SIGINT handlers so a host-side
       ``terok-sandbox supervisor`` invocation stops cleanly with Ctrl-C.
    4. Await ``podman wait <container_id>`` racing the children's own
       liveness and the stop signal.  On any of them, terminate the
       children in reverse launch order and return.

    The function is the sole parent entry point; the CLI verb
    ``terok-sandbox supervisor`` invokes it via ``asyncio.run``.  The
    per-service work now lives in
    [`run_child`][terok_sandbox.supervisor.children.run_child], one
    process each.
    """
    cfg = load_sidecar(sidecar_path)
    if cfg is None:
        _logger.error(
            "no usable sidecar at %s — aborting supervisor for %s",
            sidecar_path,
            container_id,
        )
        return 2

    paths = SupervisorPaths.for_container(
        container_id, cfg.container_name, sidecar_path, cfg.runtime_dir
    )
    services = _select_services(cfg, SERVICE_NAMES)
    stop_event = asyncio.Event()
    _install_signal_handlers(stop_event)

    handles: list[ChildHandle] = []
    try:
        handles = await _launch_children(services, container_id, sidecar_path)
        if not handles:
            _logger.error(
                "supervisor: no child could be launched for %s — exiting so the wrapper retries",
                container_id,
            )
            return 3
        await _supervise(container_id, handles, stop_event, container_pid)
        return 0
    finally:
        await _terminate_children(handles)
        # rmtree the per-container dir on every exit path — launch
        # failure included — so a half-bound socket directory can't
        # outlive the supervisor and confuse the next launch.
        shutil.rmtree(paths.container_runtime_dir, ignore_errors=True)

load_sidecar(sidecar_path)

Read and parse the sidecar JSON at sidecar_path.

The OCI hook pinned this exact path via the terok.sandbox.sidecar annotation, so the supervisor never guesses — it opens the named file directly. Returns None on any I/O / schema failure; run_supervisor surfaces that as exit-code 2. The per-field validation lives in helpers that raise _BadSidecar on the first bad value, so this reader stays a flat read → validate → build.

Source code in src/terok_sandbox/supervisor/sidecar.py 
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
def load_sidecar(sidecar_path: Path) -> SidecarConfig | None:
    """Read and parse the sidecar JSON at *sidecar_path*.

    The OCI hook pinned this exact path via the
    ``terok.sandbox.sidecar`` annotation, so the supervisor never
    guesses — it opens the named file directly.  Returns ``None`` on
    any I/O / schema failure; ``run_supervisor`` surfaces that as
    exit-code 2.  The per-field validation lives in helpers that raise
    ``_BadSidecar`` on the first bad value, so this reader stays a flat
    read → validate → build.
    """
    try:
        with sidecar_path.open(encoding="utf-8") as fh:
            raw = json.load(fh)
    except (OSError, ValueError):
        _logger.exception("sidecar parse failure for %s", sidecar_path)
        return None
    if not isinstance(raw, dict):
        _logger.error("sidecar is not a JSON object: %s", sidecar_path)
        return None
    try:
        return _build_config(raw, sidecar_path)
    except _BadSidecar:
        return None
    except (KeyError, TypeError, ValueError):
        _logger.exception("sidecar schema error in %s", sidecar_path)
        return None