Govern intermediate artifact retention defaults and migration
Retention defaults often sit between lower exposure and the evidence teams need when something goes wrong.
Reducing stored data can conflict with recovery confidence, audit expectations, and established operating habits.
“I need less retained data, but I can’t be the person who makes recovery impossible later.”
Olumide Toure · Platform Data Engineering Manager
Leads production pipeline standards for a regulated insurer with strict retention commitments.
What pulls against what
- retention reduction vs. recoverability
- uniform defaults vs. legitimate exceptions
- automation speed vs. verified impact
- security exposure vs. audit evidence
What is at stake
Excess retention increases exposure and delays regulated expansion. A wrong default can remove evidence teams need during recovery or audit
Why Databricks
At Databricks, this can matter where managed data operations must support both governed scale and practical recoverability.
Written for
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.