How to Exclude `@Flaky` Tests in TestNG

Problem Statement

In large-scale test automation, some tests may be flaky—they fail intermittently due to environmental issues, race conditions, or external dependencies. Running these tests in a CI/CD pipeline (e.g., Jenkins) can lead to unreliable build results.

To avoid unnecessary failures, we need a way to mark such tests as flaky and ensure they are automatically excluded from test execution in the pipeline.

Solution Approach

1. Define a Custom `@Flaky` Annotation

We create a custom Java annotation @Flaky that marks a test as flaky.

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface Flaky {
}

2. Implement a TestNG `IMethodInterceptor`

TestNG provides an IMethodInterceptor interface that allows dynamic filtering of test methods. We implement an interceptor that checks if a test method is annotated with @Flaky and excludes it from execution.

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;
import java.lang.reflect.Method;
import java.util.List;
import java.util.stream.Collectors;

public class FlakyTestInterceptor implements IMethodInterceptor {
    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
        return methods.stream()
                .filter(method -> {
                    Method testMethod = method.getMethod().getConstructorOrMethod().getMethod();
                    return !testMethod.isAnnotationPresent(Flaky.class);
                })
                .collect(Collectors.toList());
    }
}

3. Apply the Interceptor in TestNG Configuration

Modify the testng.xml file to register the interceptor as a listener.

<suite name="TestSuite">
    <listeners>
        <listener class-name="com.example.FlakyTestInterceptor"/>
    </listeners>
    <test name="RegressionTests">
        <classes>
            <class name="com.example.SampleTest"/>
        </classes>
    </test>
</suite>

4. Annotate Tests with `@Flaky`

Now, mark the flaky tests using the @Flaky annotation. These tests will be excluded automatically.

import org.testng.annotations.Test;

public class SampleTest {

    @Test
    public void stableTest() {
        System.out.println("This test should always run.");
    }

    @Test
    @Flaky
    public void flakyTest() {
        System.out.println("This test should be excluded.");
    }
}

5. Running Tests in Jenkins

Since FlakyTestInterceptor dynamically excludes flaky tests, no additional configuration is needed in Jenkins. The standard Maven or Gradle commands will work as expected.

pipeline {
    agent any
    stages {
        stage('Build and Test') {
            steps {
                script {
                    sh 'mvn clean test'  // If using Maven
                    sh 'gradle test'     // If using Gradle
                }
            }
        }
    }
}

Conclusion

✅ The IMethodInterceptor dynamically filters out @Flaky tests at runtime. ✅ No need to modify testng.xml for each test—just annotate the test with @Flaky. ✅ Jenkins runs only stable tests, preventing flaky failures from breaking the build.

This ensures a reliable, maintainable, and automated approach to handling flaky tests in TestNG! 🚀

Problem Statement​

Solution Approach​

1. Define a Custom @Flaky Annotation​

2. Implement a TestNG IMethodInterceptor​

3. Apply the Interceptor in TestNG Configuration​

4. Annotate Tests with @Flaky​

5. Running Tests in Jenkins​

Conclusion​