Cancelled the Run but the GPU Is Still Pegged

When a run is stuck, deadlocked, or otherwise degraded, the right move is to cancel it immediately. Set the status to cancelled in the database. And yet — GPU utilization does not drop. Something is still pegging the GPU at 90% after the cancellation. What happened?

A Status Flag Is Not a Process Signal

The core issue is simple. The database status is a record, not a signal to a running process.

Rendering diagram…

If the worker has already claimed the run and launched Isaac, that Isaac process is consuming GPU resources independently. Flipping the database flag does not inform that process it has been cancelled — so it becomes a GPU zombie.

Recovery — Kill That Exact Process

Recovery means directly terminating the actual Isaac process. Find its PID from the list of GPU compute processes and send SIGKILL.

# Find the PID of the GPU compute process
nvidia-smi --query-compute-apps=pid,used_memory --format=csv
# Kill that Isaac PID directly
kill -9 <isaac_pid>

⚠️ pkill -f '<run_id>' kills your own session — Using pkill -f to "wipe everything with this run_id" matches every command containing that string — including the ssh session command you issued that contains the run_id. The output cuts out and you have shot yourself in the foot. Kill by specific PID — it is the only safe approach.

💡 Preserve other GPU workloads — The same node may be running other legitimate work (training runs, other simulations). Killing everything to "free the GPU" destroys healthy workloads alongside the zombie. Use nvidia-smi to identify the specific Isaac PID and kill only that.

Why This Happens — and the Lesson

Normal teardown walks through an orderly shutdown: bundle the recording, release resources. Forced cancellation bypasses that path, so the process can be left as an orphan.

💡 Cancellation is two things: record update + actual termination — In a distributed system, "cancel" means recording the intent and reclaiming the resource — as separate operations. Both must happen for cancellation to be real. Update only the record and the GPU keeps burning. This is the inverse of alive ≠ working — cancelled ≠ stopped.

In a nutshell

📌 DB status='cancelled' is only a record — the Isaac process the worker launched keeps running as a GPU zombie. Recovery: use nvidia-smi to find the PID and kill -9 it directly. Do not use pkill -f run_id (it kills your own ssh session), and leave other workloads on the same node untouched. "Cancel" means both the record update and the actual process termination.