KV-cache contention is slowing the first visible part of an answer.
GPU memory efficiency protects capacity while fast first tokens protect a researcher’s flow.
Who you’d be doing this for
“I refreshed the answer twice because the sources were taking so long to show up.”
Raquel Diaz · Equity Research Associate
A repeat consumer researcher using source-cited answers to prepare an investment note before market open.
What is at stake
p99 time to first token rose from 2.4 to 3.3 seconds after a model rollout. You must weigh a fast scheduler change against preserving completion and answer behavior.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- first-token speed vs. answer completion
- GPU utilization vs. memory fragmentation
- fast mitigation vs. root-cause repair
Why Perplexity
Perplexity Search serves source-cited answers through an inference engine with tight latency and cost budgets.
Written with these in mind
Not your kind of problem? 31 more at Perplexity, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.