The throughput change that helps NVMe/TCP is delaying checkpoint reads on NFS.
Fast block-path throughput protects training capacity, while stable shared-file latency keeps GPU workers productive.
Who you’d be doing this for
“My workers stop at the checkpoint barrier, and I’m watching expensive GPUs wait for storage.”
Ivan Kaminski · ML Engineer
Runs distributed training jobs using the shared file system for checkpoints.
What is at stake
NFS checkpoint reads are pushing GPU workers into visible pauses during a training run. You must weigh a reversible path change against the throughput gain it was meant to preserve.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- NFS tail latency vs NVMe/TCP throughput
- GPU productivity vs fleet-wide rollback
- fast diagnosis vs qualification evidence
- shared-file reliability vs fabric stability
Why Crusoe
Shared file storage feeds training clusters whose GPU workers depend on checkpoint data arriving without stalls.
Written with these in mind
Not your kind of problem? 76 more at Crusoe, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.