Ideally, engineers write only deterministic tests.
Not only is that unlikely to happen, it’s sometimes not the best use of their time.
The objective is for test results to be reliable: a passing test should indicate system correctness, and a failing test should represent a real issue rather than infrastructure-related noise.
Bazel returns a special FLAKY status when a test has a mix of fail and pass.
Managing flaky tests
There are two reasonable approaches for continuous integration:
- Use
--flaky_test_attempts=[number]
This approach is commonly used with a value like 2 or 3.
This tells Bazel to run the test 1-3 additional times if it fails. This works particularly well because you don’t have to tell Bazel which tests are flaky ahead of time.
Also, if you only perform retries on CI, you still see the failure locally, which is a good reminder to fix the problem.
However, the downside is that it increases the time to report an actual failing test to 2-3x the test’s runtime.
- Tag tests as
flaky = True
This approach allows a single test failure to fail the build initially, but Bazel will retry a flaky test up to two additional times.
The downside is that the version control system becomes the “database” of which tests are flaky, and the database needs to be maintained manually.
We recommend giving the BuildCop a one-click way to mark a test as flaky (or remove it)
by making a bot commit to the repository that uses Buildozer to edit the BUILD file.
Aspect plans to build a GitHub bot which does exactly this.
Determining if flakiness is fixed
When fixing a flaky test, it can be hard to know whether a flaky test has been fully resolved. A few passing runs in a row do not guarantee reliability, since the test may pass or fail unpredictably.
To ensure a flaky test has been fixed, run it multiple times locally using --runs_per_test=[number]. This helps confirm that the test consistently passes. 
