supervision

supervision

Post-start supervision check — did the per-container supervisor come up?

A container start deliberately survives a broken supervisor: the OCI hook soft-fails so a spawn failure never blocks the container, and terok-shield's egress firewall is fail-closed on its own hook. What must not happen is that the degradation stays silent — a container whose vault-routed providers and git gate are all dead, behaving normally at the shell, with nothing said (issue #458).

verify_supervision closes that gap on the API launch path. After the container starts, it reads the same sidecar the supervisor reads, and — when the sidecar declares socket-mode services — polls briefly for the sockets the supervisor binds. On timeout it returns a SupervisionStatus naming the container, the unbound socket(s), and the hook diary to read; the caller shouts it but the launch still succeeds (soft-fail preserved), and an orchestrator may escalate on the structured result.

Both transports are covered. Socket-mode wiring binds sockets under /run/terok; TCP-mode wiring binds per-container loopback ports, and /proc/net/tcp says which ports are listening — one read answers for every service at once, and it contacts none of them (an accept-and-close probe on each start would put a stray connection in three service logs to learn what a file already says). The pure-Python file poll never runs a subprocess and never raises.

Every service the sidecar wires is checked, the SSH signer included. A dead signer costs the task its git access, and it is the child most likely to die alone: it and the vault are the only two that must open the credential store, so a passphrase the supervisor cannot resolve takes out exactly those two and leaves the rest of the bundle looking healthy.

outdated_container_warning closes the mirror-image gap. A container's environment is frozen when it is created. A container that outlives a change to the /run/terok layout therefore keeps naming sockets this sandbox no longer binds. The supervisor is healthy and the bridges start; they connect to nothing. The symptom arrives half a minute later as an empty reply, not as a refused connection. The check reads the container's protocol stamp and reports the gap. It never enforces — an operator can keep using the parts of the container that still work, and recreate it when that suits them.

MIN_RUNTIME_PROTOCOL = 3 module-attribute

Lowest TEROK_CONTAINER_PROTOCOL whose socket layout this sandbox still binds.

Each service's socket moved into its own /run/terok subdirectory, which changed TEROK_VAULT_SOCKET, TEROK_SSH_SIGNER_SOCKET and TEROK_GATE_SOCKET. A container stamped below this value predates that move. terok-executor stamps the containers, so this value tracks CONTAINER_PROTOCOL there.

__all__ = ['MIN_RUNTIME_PROTOCOL', 'ServiceEndpoint', 'SupervisionStatus', 'outdated_container_warning', 'verify_supervision', 'warn_unsupervised'] module-attribute

ServiceEndpoint(service, socket=None, port=None) dataclass

One supervisor service and the address its child binds.

Exactly one of socket and port is set — the sidecar's transport decides which — so the poll can test either kind and the warning can name the address the operator will look for.

service instance-attribute

socket = None class-attribute instance-attribute

port = None class-attribute instance-attribute

__str__()

service (address), for a warning line.

Source code in src/terok_sandbox/supervision.py

def __str__(self) -> str:
    """``service (address)``, for a warning line."""
    address = self.socket if self.socket is not None else f"127.0.0.1:{self.port}"
    return f"{self.service} ({address})"

SupervisionStatus(container_name, checked, missing, hook_log, skipped=False, hook_fired=None, supervisor_log=None) dataclass

The result of a post-start supervision check for one container.

missing is the subset of checked endpoints still unbound when the poll gave up — empty on a healthy start. skipped marks the cases with nothing to verify (no sidecar, or a host whose listening ports cannot be read), which is not a failure.

container_name instance-attribute

checked instance-attribute

missing instance-attribute

hook_log instance-attribute

skipped = False class-attribute instance-attribute

hook_fired = None class-attribute instance-attribute

Whether the hook diary has an entry for this container; None when unknown.

supervisor_log = None class-attribute instance-attribute

The per-container supervisor log, when the container id was known.

ok property

True when every required endpoint was bound (or nothing needed checking).

warning()

A loud, multi-line operator warning naming the failure and where to look.

Source code in src/terok_sandbox/supervision.py

def warning(self) -> str:
    """A loud, multi-line operator warning naming the failure and where to look."""
    endpoints = "\n".join(f"warning:     {endpoint}" for endpoint in self.missing)
    return (
        f"warning: container {self.container_name!r} started but these supervisor "
        "services never bound\n"
        f"{endpoints}\n"
        "warning:   what they serve is dead in this container — the vault routes every\n"
        "warning:   provider token, the signer holds the git keys, the gate serves the repo\n"
        f"warning:   {self._where_to_look()}"
    )

verify_supervision(cfg, container_name, *, find_container_id=None, timeout=_DEFAULT_TIMEOUT_S)

Poll for the supervisor's sockets after container_name has started.

Reads <state>/sidecar/<container_name>.json — the same bundle the supervisor reads — and, in socket mode, waits up to timeout seconds for the vault socket (always bound) and the gate socket (when the sidecar wired a gate). Returns a SupervisionStatus; a missing socket means the supervisor is not up. find_container_id is asked for the container's id only when something is missing; with it the status also says whether the hook diary saw this container, so the warning points at the log that has the answer. Never raises and never blocks a healthy start beyond the time the sockets take to appear.

Source code in src/terok_sandbox/supervision.py

def verify_supervision(
    cfg: SandboxConfig,
    container_name: str,
    *,
    find_container_id: Callable[[], str | None] | None = None,
    timeout: float = _DEFAULT_TIMEOUT_S,
) -> SupervisionStatus:
    """Poll for the supervisor's sockets after *container_name* has started.

    Reads ``<state>/sidecar/<container_name>.json`` — the same bundle the
    supervisor reads — and, in socket mode, waits up to *timeout* seconds
    for the vault socket (always bound) and the gate socket (when the
    sidecar wired a gate).  Returns a
    [`SupervisionStatus`][terok_sandbox.supervision.SupervisionStatus]; a
    missing socket means the supervisor is not up.  *find_container_id*
    is asked for the container's id only when something is missing; with
    it the status also says whether the hook diary saw this container, so
    the warning points at the log that has the answer.  Never raises and
    never blocks a healthy start beyond the time the sockets take to
    appear.
    """
    sidecar_path = cfg.state_dir / "sidecar" / f"{container_name}.json"
    # The install-global hook diary the OCI hook appends to (mirrors
    # ``ContainerDiagnostics.hook_log``, which is the host-facing SSOT — but
    # that lives in the surface layer, out of reach from here).
    hook_log = cfg.state_dir / "logs" / "hook.log"
    sidecar = load_sidecar(sidecar_path) if sidecar_path.exists() else None
    if sidecar is None:
        return SupervisionStatus(container_name, (), (), hook_log, skipped=True)

    paths = SupervisorPaths.for_container(
        container_id="",  # the service sockets key on the name, not the id
        container_name=container_name,
        sidecar_path=sidecar_path,
        runtime_dir=sidecar.runtime_dir,
    )
    expected = _expected_endpoints(sidecar, paths)
    if not expected:
        return SupervisionStatus(container_name, (), (), hook_log, skipped=True)

    missing = _poll_until_bound(expected, timeout)
    if missing is None:
        # This host will not say which ports listen, so the TCP-mode
        # answer is unknown rather than bad.  Reporting every service
        # missing would be a false alarm on every start.
        return SupervisionStatus(container_name, expected, (), hook_log, skipped=True)
    container_id = find_container_id() if missing and find_container_id else None
    if container_id is None:
        return SupervisionStatus(container_name, expected, missing, hook_log)
    return SupervisionStatus(
        container_name,
        expected,
        missing,
        hook_log,
        hook_fired=_diary_mentions(hook_log, container_id),
        supervisor_log=cfg.state_dir / "logs" / f"{container_id}.log",
    )

warn_unsupervised(status)

Print the loud warning for a failed check to stderr; no-op when healthy.

Source code in src/terok_sandbox/supervision.py

def warn_unsupervised(status: SupervisionStatus) -> None:
    """Print the loud warning for a failed check to stderr; no-op when healthy."""
    if status.missing:
        print(status.warning(), file=sys.stderr)

outdated_container_warning(container_name, env)

Return the warning for a container older than the current socket layout.

env is the environment recorded on the container at creation — see Container.env. Returns None for a current container, and for one with no usable stamp. Sidecar tool containers are built from a minimal environment that never carried the stamp, so its absence says nothing about age. Pure: no filesystem, no subprocess, never raises.

Source code in src/terok_sandbox/supervision.py

def outdated_container_warning(container_name: str, env: dict[str, str]) -> str | None:
    """Return the warning for a container older than the current socket layout.

    *env* is the environment recorded on the container at creation — see
    [`Container.env`][terok_sandbox.runtime.protocol.Container.env].  Returns
    ``None`` for a current container, and for one with no usable stamp.
    Sidecar tool containers are built from a minimal environment that never
    carried the stamp, so its absence says nothing about age.  Pure: no
    filesystem, no subprocess, never raises.
    """
    try:
        recorded = int(env["TEROK_CONTAINER_PROTOCOL"])
    except (KeyError, ValueError):
        return None
    if recorded >= MIN_RUNTIME_PROTOCOL:
        return None
    return (
        f"warning: container {container_name!r} predates the current /run/terok socket "
        f"layout (protocol {recorded}, this host binds {MIN_RUNTIME_PROTOCOL})\n"
        "warning:   its git gate and vault-routed providers connect nowhere\n"
        "warning:   the bridges still listen, so the symptom is a hang, then an empty reply\n"
        "warning:   recreate the container to pick up the current layout\n"
        "warning:   the rest of the container keeps working until you do"
    )