Long prompts consume the capacity that short decode requests need.
The batch window protects customer delivery while the routing change protects steady GPU utilization.
Who you’d be doing this for
“The batch is still running when our analysts log in, and they keep asking where the results are.”
Adetola Mensah · ML Platform Engineer
Runs the enterprise team’s production document-extraction service on GPU compute.
What is at stake
Long-context requests push P95 time to first token to 4.6 seconds during the nightly batch. You have to weigh a reversible routing change against the risk of disrupting a live inference window.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- long-context usefulness vs decode responsiveness
- GPU utilization vs tail latency
- fast canary learning vs batch-window stability
Why Crusoe
vLLM serving on GPU compute is judged by whether enterprise inference workloads stay fast under their real traffic mix.
Written with these in mind
Not your kind of problem? 76 more at Crusoe, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.