Mounting s3fs Over a Directory Makes Your Data 'Disappear'

One day, every video under /training-clips was gone. Dozens of clips that had been there the night before: zero. Data loss is the most gut-wrenching moment in infrastructure work. And yet — the files were still there.

Not Deleted — Shadowed

The truth was a simple matter of Unix mount semantics. When you mount another filesystem over a non-empty directory, the original contents are shadowed and become invisible — they are not deleted.

Rendering diagram…

When s3fs (object storage mounted as a filesystem) was placed over an existing directory, everything underneath was invisible for as long as the mount was active. Unmount it and the originals come straight back.

💡 First response when data "disappears" — Before touching anything, check the mount state. If that path shows up in mount output, the files are almost certainly shadowed, not deleted. A single unmount ends the panic.

Recovery — Write-Through from Backup to Mount Path

Fortunately there was a local backup taken before the mount was applied. Recovery was simply copying from that backup into the s3fs mount path — which write-through uploads everything back to the bucket so every node sees the same files again. All the clips came back.

The Real Trap with Permanent Mounts — One Missing Line

There was a follow-up problem. After a reboot the mount was gone again. Every attempt to make it permanent via a systemd service failed, and the culprit was in an unexpected place.

⚠️ allow_other requires user_allow_other in /etc/fuse.conf — The allow_other option that lets other users read a FUSE mount fails silently unless user_allow_other appears in the system-level config file. That single missing line was the root cause of the mount quietly not attaching at boot. The answer to "why is the mount gone after a reboot" was not an option flag but a global system setting.

Lessons

1. When data "disappears," check mounts before touching anything — shadowing is not deletion.
2. Back up before mounting — what saved this incident was the local backup taken before the mount was applied.
3. "Alive ≠ working" — the mount process can be running while the mount itself is not attached. Verify at the leaf level (are the files visible?).

In a nutshell

📌 When data vanishes wholesale, suspect s3fs shadowing an existing directory — the originals are hidden, not deleted. Unmount and they reappear; recovery is backup → write-through copy to the mount path. If a permanent mount fails to attach at boot, check for user_allow_other in /etc/fuse.conf.