Diagnose a test that passes locally and fails in CI

This test passes on every developer machine and fails in CI roughly one run in four:

java

@Test
void searchReturnsMatchingProducts() {
    driver.get(baseUrl + "/catalog");
    driver.findElement(By.id("search")).sendKeys("desk");
    Thread.sleep(2000);
    List<WebElement> cards = driver.findElements(By.cssSelector("[data-testid='product-card']"));
    assertFalse(cards.isEmpty(), "expected search results");
    assertTrue(cards.get(0).getText().toLowerCase().contains("desk"));
}

CI facts: containers are limited to 0.5 CPU, and the catalog API can take up to 1.5 seconds under the slow-api-response condition. Locally the API responds in about 40ms.

Your task: explain the mechanism of the failure, rewrite the test, and answer the question that follows in every interview: "Someone suggests wrapping it in a retry. What do you say?"

Diagnose a test that passes locally and fails in CI

Diagnose a test that passes locally and fails in CI

Community Discussions