The setting
's puts metrics, logs, traces and alerts in one place so teams can see how their software systems are running. It also includes security tools for cloud, code and data.
Scheduled test capacity competes with the compute that keeps live workloads stable.
Alerting needs realistic environments before release windows, while Core Infrastructure must preserve headroom for production demand.
Who you’d be doing this for
“My release window is booked, but I still don’t know if we’ll get an environment in time.”
Tariq Moradi · DevOps Engineer · a character in this brief
Runs release validation for an enterprise team that depends on realistic test environments before production changes.
What is at stake
Alerting releases wait on uncertain Kubernetes capacity during scheduled validation windows. You must decide how much capacity a reservation can hold before production demand needs it.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- scheduled validation vs production headroom
- self-service access vs shared capacity control
- realistic environments vs cloud efficiency
- regional predictability vs placement flexibility
Why Datadog
Kubernetes capacity across AWS and Google Cloud underpins the Alerting platform's ability to validate changes without destabilising customer monitoring.
Written with these in mind
Not your kind of problem? 57 more at Datadog, or browse every organization.
Where the organization context comes fromThe postings this brief was written from, and the reporting read before it.
The organization context comes from public information. The situation and characters are written for the brief.
Job postings it was written from
Public descriptions of the work at Datadog. They are listed as sources, not as openings. The lines under a posting are its own words, where it meets this brief.
- Senior Software Engineer - Alerting Platform Automation · no longer posted
- Work with SRE, Core Infrastructure and the Alerting engineering teams to understand their needs and provide solutions.
- Build automation systems that create and manage Alerting environments, with a focus on fast, realistic test environments that shorten the feedback loop for engineers and AI-assisted development workflows.
- Staff Engineer - Cloud Networks · no longer posted
- Design, build, and operate cloud network infrastructure across AWS, GCP, Azure, and Neoclouds in a multi-region environment.
- You’ll partner closely with internal teams and Cloud Service Providers to troubleshoot complex connectivity issues, integrate new networking capabilities, and improve the foundations our engineers and customers rely on.
- Staff Engineer, Compute · no longer posted
- You'll work on the systems that ensure engineering teams have the right compute capacity, in the right region, at the right time across AWS, Google Cloud, and Azure.
- As a Staff Engineer on Datadog's Compute – Disruption and Workload Placement team, you'll help define how our Kubernetes fleet scales to meet the demands of rapidly growing AI and cloud-native workloads.
- Developer Advocate - Service Management EMEA (opens in a new tab)
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.
Where similar work appears
Other live briefs connected to this one, and what connects them.
Similar problems at other organizations
Like this one, each of these turns on moving faster inside a system others depend on.
Validation engineers are rebuilding cloud setup details before comparable closed-loop campaigns can begin.
You have to weigh fast self-service against the controls that make reruns defensible.
Pilot customers cannot tell which queued training job should receive scarce GPUs first.
You must weigh clearer workload priority against consistent behavior across Kubernetes, Slurm, and reservation commitments.
Agent-authored changes are passing CI while dependency failures emerge only after rollout.
You must weigh realistic pre-merge validation against the cost and control of disposable environments.
The same setting, from another role
Each is set around Datadog’s Datadog Observability and Security Platform, written for a different role. They’re separate briefs, not the same scenario.
Responders are reaching different severity calls because each reconstructs alert context separately.
You must decide where the route stops gathering Cloud SIEM, Logs and APM evidence and where a responder takes over to name the severity and pick the owner.
Renewal risk is being discovered only after enterprise customers raise usage and support concerns.
You have to decide at which point in a renewal review the ranking stops assembling signals and the CSM takes over, before any outreach to the customer.
Strategic discovery context is being reduced to untraceable notes, leaving sellers to reconstruct the next step.
You have to decide how far the extraction goes between a discovery call and the account record, and at which point a seller must step in before a field or a buyer-facing promise stands.
Other problems with a similar purpose
In each of these, as in this one, the aim is to build high-assurance systems where a consequential change's correctness can be demonstrated, not merely hoped for.
Rehearsals show the routing works, but recall of sources varies by 0.9% and provenance records are incomplete.
You have to weigh the contractual launch against how much you verify before each matter moves.
The old encrypted store retires on a fixed date, and 6.9% of historical artifacts still lack a source record and retention decision.
You have to weigh speed against proof that reports stay reproducible.
The residency commitment you signed forces a one-way split of event data, and 9% of derived datasets still lean on labels a machine guessed.
With six weeks left, you weigh completeness by region against keeping analytics intact.