Prevent duplicate task writes after retry
Recovery mechanisms can preserve availability while quietly weakening data confidence.
Fast retries and exactly-once outcomes need to hold together under partial failure.
“The run says success, but now we’re checking whether yesterday got counted twice.”
Huong Siripong · Analytics Engineering Lead
Owns scheduled transformations that publish daily revenue and inventory tables.
What pulls against what
- rapid recovery vs. duplicate execution
- durable evidence vs. low-latency scheduling
- platform-level prevention vs. downstream reconciliation
What is at stake
The service recovers quickly, but some successful runs leave duplicated data behind. The right design protects both availability and trust in downstream tables
Why Databricks
At Databricks, it can matter because scheduled outputs often feed operational and reporting decisions.
Written for
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.