The leaf policy that protects lossless traffic is also stalling shared-file reads.
Fast recovery for training jobs pulls against preserving the RDMA behavior neighboring storage traffic depends on.
Who you’d be doing this for
“My workers sit there waiting on reads, then the rerun looks fine and I lose the trail.”
Ximena Monteiro · ML Engineer
Runs distributed training jobs whose workers read checkpoints from the shared file system.
What is at stake
Checkpoint reads are freezing for seconds on one Kubernetes cluster while PFC pauses build on its leaf pair. You must weigh a fast leaf-policy correction against lossless RDMA behavior for nearby storage traffic.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- shared-file latency vs lossless RDMA behavior
- targeted switch policy vs SDN rerouting
- rapid recovery vs validated rollback
- GPU productivity vs network blast radius
Why Crusoe
Kubernetes training clusters depend on the dedicated RDMA storage network to keep GPU workers supplied with shared data.
Written with these in mind
Not your kind of problem? 80 more at Crusoe, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.