Design a policy for flaky tests in CI

Your team's pipeline runs 800 tests on every pull request. Current state:

The suite fails on roughly 30% of runs.
Investigating a failure takes 20–40 minutes and about 90% turn out to be flakes.
Developers have learned to press "Re-run failed jobs" and merge on green.
Two real defects reached production last quarter; both had been caught by a test that was re-run until it passed.

Leadership has asked you to "fix the flaky tests." You have one engineer and a quarter.

Your task: propose the policy. Include what you would measure, what you would automate, and what you would ask other teams to change. Then answer: "Would you allow automatic retries?"

Design a policy for flaky tests in CI

Your team's pipeline runs 800 tests on every pull request. Current state:

The suite fails on roughly 30% of runs.
Investigating a failure takes 20–40 minutes and about 90% turn out to be flakes.
Developers have learned to press "Re-run failed jobs" and merge on green.
Two real defects reached production last quarter; both had been caught by a test that was re-run until it passed.

Leadership has asked you to "fix the flaky tests." You have one engineer and a quarter.

Your task: propose the policy. Include what you would measure, what you would automate, and what you would ask other teams to change. Then answer: "Would you allow automatic retries?"