The setting
's puts metrics, logs, traces and alerts in one place so teams can see how their software systems are running. It also includes security tools for cloud, code and data.
Regional headroom protects alert latency, while cross-region placement changes the failure domain.
Keeping alert evaluations local preserves network simplicity, while moving them preserves capacity for the next demand spike.
Who you’d be doing this for
“Our alert showed up after the rollback call, and everyone asked why the page was late.”
Sunho Xie · DevOps Engineer · a character in this brief
Runs on-call for an enterprise platform team that depends on alerts arriving during multi-service incidents.
What is at stake
Alert evaluations are queuing during regional AI workload bursts. You must weigh protected headroom against the network risk of moving alerting workloads.
Why it isn’t already fixed
Every obvious fix costs something else. That’s the part you’d have to decide.
- regional headroom vs local alert latency
- network familiarity vs workload rebalancing
- cloud commitment vs future flexibility
- fast relief vs verified failure behavior
Why Datadog
Kubernetes placement and multi-cloud network paths determine whether alert evaluations remain timely when regional demand rises.
Written with these in mind
Not your kind of problem? 57 more at Datadog, or browse every organization.
Where the organization context comes fromThe postings this brief was written from, and the reporting read before it.
The organization context comes from public information. The situation and characters are written for the brief.
Job postings it was written from
Public descriptions of the work at Datadog. They are listed as sources, not as openings. The lines under a posting are its own words, where it meets this brief.
- Senior Software Engineer - Alerting Platform Automation · no longer posted
- At Datadog, we process millions of monitors every minute, alerting our customers when configured thresholds are crossed.
- Work with SRE, Core Infrastructure and the Alerting engineering teams to understand their needs and provide solutions.
- Staff Engineer - Cloud Networks · no longer posted
- Lead deep investigations into latency, packet loss, and connectivity failures – from pcap and path analysis through to escalations with cloud providers that may originate from customer support
- Own connectivity between clouds, customers, and developers—ensuring scalable, secure, and reliable network paths.
- Staff Engineer, Compute · no longer posted
- Use data and capacity signals to influence infrastructure decisions, forecast growth, and improve workload placement strategies.
- Lead the technical direction of capacity management and workload placement for Datadog's Kubernetes platform spanning 100,000+ virtual machines across multiple cloud providers.
- Developer Advocate - Service Management EMEA (opens in a new tab)
- This is a unique opportunity to use both your engineering and creative storytelling skills to shape the landscape in cloud observability, incident response and service management.
This is the setup. The work is inside.
Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.
Where similar work appears
Other live briefs connected to this one, and what connects them.
Similar problems at other organizations
Like this one, each of these turns on moving faster inside a system others depend on.
Active agent sessions are failing when traffic shifts between regions during the cutover.
You must weigh new capacity against the risk of committing to a routing state that cannot be cheaply reversed.
Bank callers face longer pauses while the current AWS capacity expires in nine days.
You must choose where the cutover evidence is strong enough to move customer traffic.
Launch-rehearsal users saw failed LLM responses, and the customer wants production traffic moved this week.
You must weigh a fast dedicated cutover against the evidence and capacity commitments needed to make it safe.
The same setting, from another role
Each is set around Datadog’s Datadog Observability and Security Platform, written for a different role. They’re separate briefs, not the same scenario.
Responders are reaching different severity calls because each reconstructs alert context separately.
You must decide where the route stops gathering Cloud SIEM, Logs and APM evidence and where a responder takes over to name the severity and pick the owner.
Renewal risk is being discovered only after enterprise customers raise usage and support concerns.
You have to decide at which point in a renewal review the ranking stops assembling signals and the CSM takes over, before any outreach to the customer.
Strategic discovery context is being reduced to untraceable notes, leaving sellers to reconstruct the next step.
You have to decide how far the extraction goes between a discovery call and the account record, and at which point a seller must step in before a field or a buyer-facing promise stands.
Other problems with a similar purpose
In each of these, as in this one, the aim is to turn fragmented, conflicting operational signals into one clear, actionable call.
Checking customers retry account actions when the mobile app and backend services tell an incomplete story.
You decide where the AI's trace of an incident stops and where the on-call engineer's confirmation of the cause begins.
Conflicting updates are letting carrier-integration risks reach readiness reviews too late.
You must decide which dependency changes the workflow sends straight to a Program Manager as uncertain and which it records as flags for the readiness record.
Regional production can stop while aggregate workspace activity still looks healthy.
You must decide where the ranking stops and the CSM starts: which signals set an account's place in the order, and which the CSM reads for themselves before choosing what to say.
