Delivering Assets to Isolated Remote GPU Nodes

Simulation needs to read large assets — USD environments, robots, textures. When GPU nodes are co-located in the same rack, NFS shared storage is straightforward. But when GPUs are scattered across cloud, remote sites, or air-gapped networks, NFS is not an option. How do you give every node a consistent view of the same assets?

Object Storage as a Filesystem — s3fs

The solution is to keep object storage (MinIO or any S3-compatible store) in one place and have each GPU node mount that bucket as a filesystem. The tool that does the mounting is s3fs, built on FUSE.

Rendering diagram…

When any node reads from /assets/..., s3fs transparently issues an S3 GET to the bucket. The node sees "just files" and the USD loader behaves exactly as it would with local storage — no NFS required.

The Trap — Choose the Wrong Write Path and Views Diverge

Reads are straightforward. Writes are where things break.

⚠️ SDK upload ≠ mount reflects the change — When you update an asset by uploading directly through the object-storage SDK, the object reaches the bucket but s3fs's local cache is not invalidated. The result: "new file on Node A, old file on Node B" — views diverge permanently. Assets must always be written directly to the s3fs mount path (write-through) so every node sees the same thing.

Rendering diagram…

💡 "Why is the file visible on this node but not that one?" — When that question comes up, the write path is almost always the cause. Clear the cache directory and consolidate all writes through the mount path and it resolves immediately. Asset consistency in a multi-node remote environment is entirely a matter of where you write.

Why Go Through All This

NFS is fine when it works. But the moment GPU nodes expand into cloud or remote locations, network isolation and firewalls block NFS. s3fs is HTTP(S)-based and punches through from anywhere, and object storage scales essentially without limit — "attach any GPU node and it sees the same assets" is the platform-level scalability foundation.

In a nutshell

📌 Remote and isolated GPU nodes get asset access via object storage (MinIO) mounted with s3fs (FUSE) instead of NFS, making USD assets appear as local files. The critical constraint: always write directly to the mount path (write-through) — SDK uploads do not invalidate the s3fs cache, causing node views to diverge.