janitor

janitor

Reconcile stray supervisor process trees against live containers.

The per-container supervisor is meant to die with its container — the poststop hook group-kills the tree, the supervisor self-terminates when its container's init PID dies, and PR_SET_PDEATHSIG takes the service children down with it. Those are the prevention layers. This module is the reconciliation backstop: a periodic sweep that finds supervisor trees whose container is no longer running and kills them, no matter how they were stranded (a poststop that never fired, a host crash that left the tree, or a supervisor built before the prevention layers existed).

Unlike the OCI hook — which crun hands the container's environment, so its nested podman calls are unreliable — the janitor runs from an ordinary host CLI context (doctor, a task launch), where podman ps answers normally. That is what lets it use container liveness as the ground truth the hook cannot.

A tree is identified structurally: the wrapper is a session leader (the OCI hook spawns it with start_new_session=True), and every descendant — supervisor, service children — inherits that process group, so one killpg per group takes the whole tree down. Groups are matched by the container id carried in each process's argv, and each member's identity (PID + start time) is revalidated immediately before signalling so a PGID recycled between the scan and the kill is never touched.

__all__ = ['make_orphan_supervisor_check', 'reap_orphaned_supervisors'] module-attribute

reap_orphaned_supervisors(*, min_age_s=_ORPHAN_GRACE_S)

Kill every supervisor tree whose container is no longer running.

Returns one (container_id, error_or_None) row per tree reaped (None error when it went down cleanly), or an empty list when nothing was stray — the steady state. Returns None when podman was unreachable: liveness is then unknown, so the sweep does nothing and the caller must not read that as "all clean" (distinct from the empty list, which means podman answered and found no strays).

A tree is reaped only when its container is absent from the live set, its age is known and past min_age_s, and — at signal time — an original member still occupies the PGID.

Source code in src/terok_sandbox/supervisor/janitor.py

def reap_orphaned_supervisors(
    *, min_age_s: float = _ORPHAN_GRACE_S
) -> list[tuple[str, str | None]] | None:
    """Kill every supervisor tree whose container is no longer running.

    Returns one ``(container_id, error_or_None)`` row per tree reaped
    (``None`` error when it went down cleanly), or an empty list when
    nothing was stray — the steady state.  Returns **``None``** when podman
    was unreachable: liveness is then unknown, so the sweep does nothing and
    the caller must not read that as "all clean" (distinct from the empty
    list, which means podman answered and found no strays).

    A tree is reaped only when its container is absent from the live set,
    its age is known and past *min_age_s*, and — at signal time — an
    original member still occupies the PGID.
    """
    alive = _live_container_ids()
    if alive is None:
        return None  # podman unreachable — liveness unknown, don't guess
    results: list[tuple[str, str | None]] = []
    for container_id, group in sorted(_scan_supervisor_groups().items()):
        if container_id in alive:
            continue
        if not group.age_known:
            continue  # an unreadable member age → can't rule out a mid-create tree
        if group.youngest_age_s < min_age_s:
            continue  # a just-spawned tree racing its container's registration
        results.append((container_id, _kill_group(group)))
    return results

make_orphan_supervisor_check()

A host-side doctor check that reaps supervisor trees without a live container.

Host-level like the stray-sidecar sweep — one reconciliation per install, not per task — so top-level callers append it rather than it living in the per-container check list. Podman being unreachable is surfaced as a warn (the sweep could not run), never a silent ok.

Source code in src/terok_sandbox/supervisor/janitor.py

def make_orphan_supervisor_check() -> DoctorCheck:
    """A host-side doctor check that reaps supervisor trees without a live container.

    Host-level like the stray-sidecar sweep — one reconciliation per
    install, not per task — so top-level callers append it rather than
    it living in the per-container check list.  Podman being unreachable
    is surfaced as a ``warn`` (the sweep could not run), never a silent ok.
    """
    from ..doctor import CheckVerdict, DoctorCheck

    def _eval(_rc: int, _stdout: str, _stderr: str) -> CheckVerdict:
        reaped = reap_orphaned_supervisors()
        if reaped is None:
            return CheckVerdict(
                "warn", "podman unreachable — could not check for orphaned supervisor trees"
            )
        if not reaped:
            return CheckVerdict("ok", "no orphaned supervisor trees")
        failed = [cid for cid, err in reaped if err is not None]
        detail = f"reaped {len(reaped)} orphaned supervisor tree(s): " + ", ".join(
            cid[:12] for cid, _ in reaped
        )
        if failed:
            return CheckVerdict(
                "warn",
                detail + f" ({len(failed)} would not die: {', '.join(c[:12] for c in failed)})",
            )
        return CheckVerdict("ok", detail)

    return DoctorCheck(
        category="env",
        label="Orphaned supervisor sweep",
        probe_cmd=[],
        evaluate=_eval,
        host_side=True,
    )