The Run Is Stuck at 'requested'

A simulation run was dispatched to a remote GPU node behind a VPN and its status never moved past requested. The worker was not picking up work. The node was online. The tunnel process was alive. And yet nothing flowed.

Alive but Dead — Half-Open

The remote node connects back to the central host over a reverse SSH tunnel — the node exposes its ports to the center, which then pushes work through those ports. The problem was that this tunnel had gone half-open (partly dead).

Rendering diagram…

When the VPN dropped on an idle timeout, the node's sshd left a reverse-forward LISTEN socket behind as an orphan. When autossh tried to bring up a new tunnel it could not bind that port — the zombie was already holding it. Result: the forward was dead but the process was alive. That is why "tunnel is up" was a lie.

Why the Sentinel Was Fooled

The auto-recovery sentinel was also fooled.

⚠️ "Process alive ≠ function alive" — The sentinel declared "recovered" as soon as the autossh process was running. But a live process with a dead forward accomplishes nothing. This false "recovered" status drove the sentinel into an infinite restart loop. Health checks must verify actual function (does a request through the forward get a real response?) — not just process existence.

The decisive diagnostic clue: querying localhost:<port> from the node returned response code 000 (no connection), while the autossh process looked perfectly healthy. That contradiction pointed directly at the half-open zombie.

Recovery — Kill the Zombie First

Rendering diagram…

The order of operations matters — the zombie occupying the port must be killed before restarting autossh. Restarting without clearing the zombie simply fails to bind again. Kill zombie → restart → verify that a real request through the forward succeeds. Running that sequence unblocked all the stuck runs.

💡 Health-check at the leaf — The question is not whether an intermediate process is alive, but whether data actually flows end-to-end. If a run is stuck at requested, look at the leaf of the node's connectivity (is the forward really live?) before debugging the worker, database, or code.

In a nutshell

📌 A run stuck at requested should raise suspicion of a half-open zombie socket in the remote node's reverse tunnel — a VPN idle drop orphans the LISTEN socket, autossh cannot rebind, and the process stays alive while the forward is dead. Recovery: kill the zombie first, restart the tunnel service, then verify with a real request through the forward. Health checks must test actual end-to-end function, not process existence.