The setting
's puts metrics, logs, traces and alerts in one place so teams can see how their software systems are running. It also includes security tools for cloud, code and data.
A fast retry decision can hide the condition that made a deployment fail.
The pipeline needs quick recovery from flaky failures while production changes need evidence that a real fault is not being repeated.
Who you’d be doing this for
“I reran it because the logs looked noisy, then the same deploy failed after the release window moved.”
Sofie Bakken · DevOps Engineer · a character in this brief
Reviews failed deployment runs for the platform pipeline and decides whether the next action is safe.
What is at stake
Alerting engineers keep rerunning failed deploys while real Kubernetes conditions remain unresolved. You must decide whether the AI only recommends a retry or also drafts the proposed fix for review.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- fast retries vs recurring infrastructure conditions
- concise diagnosis vs inspectable evidence
- pipeline speed vs shared platform stability
- known rules vs cross-system failure patterns
Why Datadog
Datadog's Kubernetes platform and multi-cloud deployment systems depend on fast, reliable validation of production Go changes.
Written with these in mind
Not your kind of problem? 57 more at Datadog, or browse every organization.
Where the organization context comes fromThe postings this brief was written from, and the reporting read before it.
The organization context comes from public information. The situation and characters are written for the brief.
Job postings it was written from
Public descriptions of the work at Datadog. They are listed as sources, not as openings. The lines under a posting are its own words, where it meets this brief.
- Senior Software Engineer - Alerting Platform Automation · no longer posted
- Build automation systems that create and manage Alerting environments, with a focus on fast, realistic test environments that shorten the feedback loop for engineers and AI-assisted development workflows.
- Work with SRE, Core Infrastructure and the Alerting engineering teams to understand their needs and provide solutions.
- Staff Engineer - Cloud Networks · no longer posted
- Design, build, and operate cloud network infrastructure across AWS, GCP, Azure, and Neoclouds in a multi-region environment.
- Datadog’s Cloud Networks team designs, builds, and maintains the production network infrastructure that powers everything built on top of our platform across AWS, GCP, Azure, and beyond.
- Staff Engineer, Compute · no longer posted
- Develop production software in Go to improve Kubernetes platform capabilities, automation, and operational efficiency.
- As a Staff Engineer on Datadog's Compute – Disruption and Workload Placement team, you'll help define how our Kubernetes fleet scales to meet the demands of rapidly growing AI and cloud-native workloads.
- Developer Advocate - Service Management EMEA (opens in a new tab)
- Create content in one or more mediums to build Datadog's reputation as a leader in DevOps, Monitoring, Observability and Security e.g. building demos, public speaking, blogging, documentation, webinars, open source, research reports and more
- With a focus on service management, helping teams respond to incidents, run on-call, and automate their operations, you will work with practitioners and leaders across the industry and broaden your impact to the SRE, Engineer, DevOps, and Operations community at large.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.
Where similar work appears
Other live briefs connected to this one, and what connects them.
Similar problems at other organizations
Like this one, each of these is about the everyday tools people rely on silently breaking.
Known Tekton failures are consuming release windows while engineers reconstruct the same evidence.
You have to decide whether a retry of a failed run counts as a diagnostic step the AI may take alone or as the first production-changing step an engineer must approve.
Recurring deployment failures leave builders retrying publishes while on-call engineers reconstruct familiar context.
You must decide at which point in the runbook the system stops running diagnostic steps and hands the production change to an engineer.
Cloud incidents are taking longer to diagnose because build, route, and edge evidence must be reconstructed by hand.
You must decide what AI can rank safely and what an engineer must verify before a fix moves forward.
The same setting, from another role
Each is set around Datadog’s Datadog Observability and Security Platform, written for a different role. They’re separate briefs, not the same scenario.
Responders are reaching different severity calls because each reconstructs alert context separately.
You must decide where the route stops gathering Cloud SIEM, Logs and APM evidence and where a responder takes over to name the severity and pick the owner.
Renewal risk is being discovered only after enterprise customers raise usage and support concerns.
You have to decide at which point in a renewal review the ranking stops assembling signals and the CSM takes over, before any outreach to the customer.
Strategic discovery context is being reduced to untraceable notes, leaving sellers to reconstruct the next step.
You have to decide how far the extraction goes between a discovery call and the account record, and at which point a seller must step in before a field or a buyer-facing promise stands.
Other problems with a similar purpose
In each of these, as in this one, the aim is to turn fragmented, conflicting operational signals into one clear, actionable call.
Checking customers retry account actions when the mobile app and backend services tell an incomplete story.
You decide where the AI's trace of an incident stops and where the on-call engineer's confirmation of the cause begins.
Conflicting updates are letting carrier-integration risks reach readiness reviews too late.
You must decide which dependency changes the workflow sends straight to a Program Manager as uncertain and which it records as flags for the readiness record.
Regional production can stop while aggregate workspace activity still looks healthy.
You must decide where the ranking stops and the CSM starts: which signals set an account's place in the order, and which the CSM reads for themselves before choosing what to say.
