How to only output one Pass/Fail result when test is rerun?

Question

I'm using the IRetryAnalyzer class in TestNG in order to relaunch tests that fail (up to 5 attempts). Right now if I run a test and it fails three times before passing, it says:

Tests run: 4, Failures: 3, Errors 0, Skipped 0.

What I want the results to show is:

Tests run: 1, Failures: 0, Errors 0, Skipped 0.

Basically, I don't want it to count as a failure unless it fails all 5 reruns. How do I do this?

Answer 1

since nobody replied, i give it a try.

just random thoughts i had when reading your post, while i was searching for a solution for my problem...

you can get the result of your test in your @AfterTest method via ITestResult result

@AfterMethod
public void AfterMethod_CheckForFailure(ITestResult result, Method m)

an @AfterMethod is called when a method with @Test annotation has finished. able to read the current iteration from a static class with a static member and if its <5 then use

result.setStatus(ITestResult.SUCCESS); // or what ever status you want
setCurrentTestResult(result);

dunno if that works, but looks to me it could :) Good luck

Answer 2

I have just done this and how I did it was I considered everything before the last time a success. I did this to allow 3 reruns, here is the code I used:

import org.testng.IRetryAnalyzer;
import org.testng.ITestResult;
import org.testng.Reporter;

public class RetryAnalyzer implements IRetryAnalyzer {
  private int count = 0;
  private int maxCount = 2;

  @Override
  public boolean retry(ITestResult result) {
    if (!result.isSuccess()) {
      if (count < maxCount) {
        count++;
        result.setStatus(ITestResult.SUCCESS);
        String message = Thread.currentThread().getName() + ": Error in " + result.getName() + " Retrying "
            + (maxCount + 1 - count) + " more times";
        System.out.println(message);
        Reporter.log(message);
        return true;
      } else {
        result.setStatus(ITestResult.FAILURE);
      }
    }
    return false;
  }
}

The key here was when I am still retrying, I set the result to success

result.setStatus(ITestResult.SUCCESS)

and when the attempt has been made and it still fails, then set it to a fail

result.setStatus(ITestResult.FAILURE);