Canonical duplicates are consuming the capacity needed for fresh pages.
Crawler-side prevention protects ingestion capacity while Search-side filtering preserves broader recall.
Who you’d be doing this for
“I searched for a paper announced this morning, and the answer kept citing last week’s coverage.”
Eduardo Correia · Graduate student
An information seeker using cited answers to track fast-moving research and policy updates.
What is at stake
Duplicate documents are up 19%, and affected domains are arriving 14 minutes late. You must weigh crawler-side prevention against a Search-side filter when only one maintenance window remains.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- freshness recovery vs. source recall
- crawler prevention vs. retrieval filtering
- queue relief vs. false exclusions
Why Perplexity
Search answers depend on a web-scale ingestion pipeline that keeps current pages available to retrieval.
Written with these in mind
Not your kind of problem? 31 more at Perplexity, or browse every organization.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.