API testing with Karate on Azure Pipelines

Question

i have coded my API testing on Karate and now I'd like to run them on Azure Pipelines.

There are no errors when I'm running them on Karate but when I'm implementing them on Azure, I always get an error back. I created a repository on Github with my pom.xml, .feature and .java files and selected this repository for my pipeline, butv every time I'm running the Test I get error

This is my pipeline. do I have to add the class or my feature file somewhere?

steps:
- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'package'

Would love to get some help, since there aren't a lot article which can help me on that matter

Answer 1

Just one comment, if you are using Karate 1.X

• the JUnit (XML) reports need to be enabled by calling outputJunitXml(true) on the Runner: https://github.com/intuit/karate/wiki/1.0-upgrade-guide#java-projects
• and the files end up in karate-reports (not surefire-reports) and have a different naming convention

Answer 2

Here is how I do it:

  - task: Maven@3
    inputs:
      publishJUnitResults: false
      javaHomeOption: JDKVersion
      jdkVersionOption: default
      jdkArchitectureOption: x64
      mavenPomFile: pom.xml
      mavenVersionOption: Default
      mavenAuthenticateFeed: false
      effectivePomSkip: false
      sonarQubeRunAnalysis: false
      goals: "clean test -Dkarate.env=$(environment) -Pintegration-test"
      mavenOptions: '-Xmx2048m -Dmaven.repo.local=$(Pipeline.Workspace)/.m2/repository'
    displayName: "Run Karate Integration Tests"

  - task: PublishTestResults@2
    name: PublishKarateTestResults
    inputs:
      testRunTitle: "Karate Integration Tests"
      testResultsFormat: "JUnit"
      testResultsFiles: "**/karate-reports/xml/*.xml"
      failTaskOnFailedTests: true
      mergeTestResults: true

The important thing here is the very last line mergeTestResults: true because I want results from integration tests to be merged with the JUnit test results.

And my test is defined like so:

@Test
public void runAPITests() {
    Results results = Runner.path("classpath:features")
            .tags("regression")
            .reportDir("target/karate-reports/xml")
            .outputJunitXml(true)
            .tags("~@ignore")
            .parallel(2);
    Assertions.assertTrue(results.getFeaturesTotal() > 0,
            "Did not find any cucumber tests to execute.");

    generateReport(results.getReportDir());

    Assertions.assertEquals(0, results.getFailCount(), 
      "Had at least one test failure.");
}

static void generateReport(String karateOutputPath) {
    Collection<File> jsonFiles = FileUtils.listFiles(new File(karateOutputPath), 
      new String[] {"json"}, true);
    List<String> jsonPaths =
      jsonFiles.stream().map(File::getAbsolutePath)
      .collect(Collectors.toList());
    Configuration config = new Configuration(new File("target"), 
      REPORT_TITLE);
    ReportBuilder reportBuilder = new ReportBuilder(jsonPaths, config);

    reportBuilder.generateReports();
}