Bringing AI in at Superhuman

Distinguish flaky tests from defects when a retry passes

Published Nov 7, 2026

You’re a forward deployed engineer working with Superhuman’s DevOps team. The team is in the room.

The setting

makes , a collaborative document application with tables, for teams that start by building a tracker in a doc and end up running a real application.

A passing retry does not settle whether the first timeout exposed a defect.

Release candidates need a next step, while test failures need an explanation that survives another run.

Who you’d be doing this for

“Our tracker owners asked when the update would land again today. I don't have anything new to tell them.”

Filip Zielinski · Enterprise Document Administrator · a character in this brief

Maintains shared operational trackers whose users depend on safe releases and timely improvements.

What is at stake

Engineers retry timed-out tests, but a passing run may hide a defect. You must decide what evidence lets an AI diagnosis reach the engineer's rerun decision, and when it must withhold a recommendation.

Why it isn’t already fixed

Every obvious fix costs something else. That’s the part you’d have to decide.

  • retry confidence vs exposure to intermittent defects
  • diagnostic coverage vs predictable behavior
  • richer execution evidence vs collection overhead
  • local failure context vs shared paved paths

Why Superhuman

Bazel test results and Kubernetes execution records must explain failures across a codebase that mixes a monolith with microservices and receives hundreds of pull requests each week.

Written with these in mind

Forward deployed engineersSolutions engineers with CI/CD depthEngineers building operational AI systems

Not your kind of problem? 49 more at Superhuman, or browse every organization.

Where the organization context comes fromThe postings this brief was written from, and the reporting read before it.

The organization context comes from public information. The situation and characters are written for the brief.

Job postings it was written from

Public descriptions of the work at Superhuman. They are listed as sources, not as openings. The lines under a posting are its own words, where it meets this brief.

  • Software Engineer, Developer Experience (Full-Stack) (opens in a new tab)
    • Work hands-on with build systems (such as Bazel or equivalents) and containerized, cloud-native infrastructure (Kubernetes) to make builds and pipelines faster and more reliable.
    • Is genuinely motivated by improving how other engineers work, rather than by shipping product features—and enjoys continuously learning across a broad and evolving technology stack.
  • Engineering Manager, Release Engineering (opens in a new tab)
    • Reduce friction for product engineers: cut build and test times, eliminate flaky tests and pipelines, and offer self-service paths so teams can ship without waiting on us.
    • The team builds and runs the CI/CD platform, release pipelines, and rollout tooling that let product teams deploy many times a day with confidence; it enables engineers to be productive, use new technologies to optimize the SLDC, and push on green many times a day.

This is the setup. The work is inside.

Running it puts you in the room: the full situation and its constraints, stakeholders who push back in their own words, and the decisions that are yours to make. What you produce becomes a Day One Plan — work you can show someone instead of describing.

Other live briefs connected to this one, and what connects them.

Similar problems at other organizations

Like this one, each of these is about the everyday tools people rely on silently breaking.

The same setting, from another role

Each is set around Superhuman’s Superhuman Docs, written for a different role. They’re separate briefs, not the same scenario.

Other problems with a similar purpose

In each of these, as in this one, the aim is to surface the one failure mode that truly matters early enough to change the system's future, not just document it afterward.