Shared query performance is pulling against the evidence a large GPU cluster needs to operate.
Other tenants need fast incident access while the largest training cluster needs detailed per-pod telemetry.
Who you’d be doing this for
“My job was sitting half idle, and the only labels that explain it vanished from the query.”
Ximena Rojas · ML Platform Engineer
Operates a large Kubernetes training cluster and depends on per-pod GPU telemetry to investigate utilization loss.
What is at stake
Unrelated Kubernetes customers are waiting on queries while one training cluster floods the shared telemetry path. You have to weigh fast recovery for everyone against losing the labels needed to investigate scarce GPU capacity.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- shared query speed vs diagnostic fidelity
- storage headroom vs telemetry retention
- fast containment vs durable policy
- tenant fairness vs large-cluster flexibility
Why Crusoe
Kubernetes-based GPU workloads depend on the Cloud Monitoring Service for both fleet-wide performance and workload-level diagnosis.
Written with these in mind
Not your kind of problem? 76 more at Crusoe, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.