What is the JUnit XML format specification that Hudson supports

Question

I have Hudson as continuous integration server and I want to use option  Publish JUnit test result report   But I don t use xUnit tools for testing  instead of that i have shell scripts which run tests and return results in simple format  I am thinking to make a script which transforms these results to the JUnit format  So i m interesting how the JUnit file must look

User · Answer

Basic Structure Here is an example of a JUnit output file, showing a skip and failed result, as well as a single passed result.

<?xml version="1.0" encoding="UTF-8"?>
<testsuites>
   <testsuite name="JUnitXmlReporter" errors="0" tests="0" failures="0" time="0" timestamp="2013-05-24T10:23:58" />
   <testsuite name="JUnitXmlReporter.constructor" errors="0" skipped="1" tests="3" failures="1" time="0.006" timestamp="2013-05-24T10:23:58">
      <properties>
         <property name="java.vendor" value="Sun Microsystems Inc." />
         <property name="compiler.debug" value="on" />
         <property name="project.jdk.classpath" value="jdk.classpath.1.6" />
      </properties>
      <testcase classname="JUnitXmlReporter.constructor" name="should default path to an empty string" time="0.006">
         <failure message="test failure">Assertion failed</failure>
      </testcase>
      <testcase classname="JUnitXmlReporter.constructor" name="should default consolidate to true" time="0">
         <skipped />
      </testcase>
      <testcase classname="JUnitXmlReporter.constructor" name="should default useDotNotation to true" time="0" />
   </testsuite>
</testsuites>

Below is the documented structure of a typical JUnit XML report. Notice that a report can contain 1 or more test suite. Each test suite has a set of properties (recording environment information). Each test suite also contains 1 or more test case and each test case will either contain a skipped, failure or error node if the test did not pass. If the test case has passed, then it will not contain any nodes. For more details of which attributes are valid for each node please consult the following section "Schema".

<testsuites>        => the aggregated result of all junit testfiles
  <testsuite>       => the output from a single TestSuite
    <properties>    => the defined properties at test execution
      <property>    => name/value pair for a single property
      ...
    </properties>
    <error></error> => optional information, in place of a test case - normally if the tests in the suite could not be found etc.
    <testcase>      => the results from executing a test method
      <system-out>  => data written to System.out during the test run
      <system-err>  => data written to System.err during the test run
      <skipped/>    => test was skipped
      <failure>     => test failed
      <error>       => test encountered an error
    </testcase>
    ...
  </testsuite>
  ...
</testsuites>

User · Answer

There are multiple schemas for  JUnit  and  xUnit  results    XSD for Apache Ant s JUnit output can be found at   https   github com windyroad JUnit-Schema  credit goes to this answer  https   stackoverflow com a 4926073 1733117  XSD from Jenkins xunit-plugin can be found at   https   github com jenkinsci xunit-plugin tree master src main resources org jenkinsci plugins xunit types  under model xsd    Please note that there are several versions of the schema in use by the Jenkins xunit-plugin  the current latest version is junit-10 xsd which adds support for Erlang OTP Junit format    Some testing frameworks as well as  xUnit -style reporting plugins also use their own secret sauce to generate  xUnit -style reports  those may not use a particular schema  please read  they try to but the tools may not validate against any one schema   Python unittests in Jenkins  gives a quick comparison of several of these libraries and slight differences between the xml reports generated

User · Answer

Good answers here on using python   there are many ways to do it   Python unittests in Jenkins   IMHO the best way is write python unittest tests and install pytest  something like  yum install pytest   to get py test installed   Then run tests like this   py test --junitxml results xml test py   You can run any unittest python script and get jUnit xml results   https   docs python org 2 7 library unittest html  In jenkins build configuration Post-build actions Add a  Publish JUnit test result report  action with result xml and any more test result files you produce

User · Answer

I just grabbed the junit-4 xsd that others have linked to and used a tool named XMLSpear to convert the schema to a blank XML file with the options shown below  This is the  slightly cleaned up  result    lt  xml version  1 0  encoding  UTF-8   gt   lt testsuites disabled    errors    failures    name    tests    time    gt       lt testsuite disabled    errors    failures    hostname    id                   name    package    skipped    tests    time    timestamp    gt           lt properties gt               lt property name    value     gt           lt  properties gt           lt testcase assertions    classname    name    status    time    gt               lt skipped  gt               lt error message    type     gt               lt failure message    type     gt               lt system-out  gt               lt system-err  gt           lt  testcase gt           lt system-out  gt           lt system-err  gt       lt  testsuite gt   lt  testsuites gt    Some of these items can occur multiple times    There can only be one testsuites element  since that   s how XML works  but there can be multiple testsuite elements within the testsuites element  Each properties element can have multiple property children  Each testsuite element can have multiple testcase children  Each testcase element can have multiple error  failure  system-out  or system-err children

User · Answer

The top answer of the question Anders Lindahl refers to an xsd file   Personally I found this xsd file also very useful  I don t remember how I found that one   It looks a bit less intimidating  and as far as I used it  all the elements and attributes seem to be recognized by Jenkins  v1 451   One thing though  when adding multiple  lt failure     elements  only one was retained in Jenkins  When creating the xml file  I now concatenate all the failures in one     Update 2016-11 The link is broken now  A better alternative is this page from cubic org  JUnit XML reporting file format  where a nice effort has been taken to provide a sensible documented example  Example and xsd are copied below  but their page looks waay nicer     sample JUnit XML file   lt  xml version  1 0  encoding  UTF-8   gt   lt  -- a description of the JUnit XML format and how Jenkins parses it  See also junit xsd -- gt    lt  -- if only a single testsuite element is present  the testsuites      element can be omitted  All attributes are optional  -- gt   lt testsuites disabled     lt  -- total number of disabled tests from all testsuites  -- gt              errors       lt  -- total number of tests with error result from all testsuites  -- gt              failures     lt  -- total number of failed tests from all testsuites  -- gt              name                tests        lt  -- total number of successful tests from all testsuites  -- gt              time         lt  -- time in seconds to execute all test suites  -- gt           gt      lt  -- testsuite can appear multiple times  if contained in a testsuites element         It can also be the root element  -- gt     lt testsuite name          lt  -- Full  class  name of the test for non-aggregated testsuite documents                                 Class name without the package for aggregated testsuites documents  Required -- gt           tests         lt  -- The total number of tests in the suite  required  -- gt           disabled      lt  -- the total number of disabled tests in the suite  optional -- gt               errors        lt  -- The total number of tests in the suite that errored  An errored test is one that had an unanticipated problem                                 for example an unchecked throwable  or a problem with the implementation of the test  optional -- gt               failures      lt  -- The total number of tests in the suite that failed  A failure is a test which the code has explicitly failed                                by using the mechanisms for that purpose  e g   via an assertEquals  optional -- gt               hostname      lt  -- Host on which the tests were executed   localhost  should be used if the hostname cannot be determined  optional -- gt           id            lt  -- Starts at 0 for the first testsuite and is incremented by 1 for each following testsuite -- gt           package       lt  -- Derived from testsuite  name in the non-aggregated documents  optional -- gt           skipped       lt  -- The total number of skipped tests  optional -- gt           time          lt  -- Time taken  in seconds  to execute the tests in the suite  optional -- gt           timestamp     lt  -- when the test was executed in ISO 8601 format  2014-01-21T16 17 18   Timezone may not be specified  optional -- gt            gt        lt  -- Properties  e g   environment settings  set during test      execution  The properties element can appear 0 or once  -- gt       lt properties gt         lt  -- property can appear multiple times  The name and value attributres are required  -- gt         lt property name    value     gt       lt  properties gt        lt  -- testcase can appear multiple times  see  testsuites testsuite tests -- gt       lt testcase name           lt  -- Name of the test method  required  -- gt            assertions     lt  -- number of assertions in the test case  optional -- gt            classname      lt  -- Full class name for the class the test method is in  required -- gt            status              time           lt  -- Time taken  in seconds  to execute the test  optional -- gt             gt          lt  -- If the test was not executed or failed  you can specify one            the skipped  error or failure elements  -- gt          lt  -- skipped can appear 0 or once  optional -- gt         lt skipped  gt          lt  -- Indicates that the test errored  An errored test is one            that had an unanticipated problem  For example an unchecked            throwable or a problem with the implementation of the            test  Contains as a text node relevant data for the error             for example a stack trace  optional -- gt         lt error message     lt  -- The error message  e g   if a java exception is thrown  the return value of getMessage   -- gt           type        lt  -- The type of error that occured  e g   if a java execption is thrown the full class name of the exception  -- gt            gt  lt  error gt          lt  -- Indicates that the test failed  A failure is a test which        the code has explicitly failed by using the mechanisms for        that purpose  For example via an assertEquals  Contains as        a text node relevant data for the failure  e g   a stack        trace  optional -- gt         lt failure message     lt  -- The message specified in the assert  -- gt             type        lt  -- The type of the assert  -- gt              gt  lt  failure gt          lt  -- Data that was written to standard out while the test was executed  optional -- gt         lt system-out gt  lt  system-out gt          lt  -- Data that was written to standard error while the test was executed  optional -- gt         lt system-err gt  lt  system-err gt       lt  testcase gt        lt  -- Data that was written to standard out while the test suite was executed  optional -- gt       lt system-out gt  lt  system-out gt       lt  -- Data that was written to standard error while the test suite was executed  optional -- gt       lt system-err gt  lt  system-err gt     lt  testsuite gt   lt  testsuites gt      JUnit XSD file   lt  xml version  1 0  encoding  UTF-8    gt   lt  -- from https   svn jenkins-ci org trunk hudson dtkit dtkit-format dtkit-junit-model src main resources com thalesgroup dtkit junit model xsd junit-4 xsd -- gt   lt xs schema xmlns xs  http   www w3 org 2001 XMLSchema  gt        lt xs element name  failure  gt           lt xs complexType mixed  true  gt               lt xs attribute name  type  type  xs string  use  optional   gt               lt xs attribute name  message  type  xs string  use  optional   gt           lt  xs complexType gt       lt  xs element gt        lt xs element name  error  gt           lt xs complexType mixed  true  gt               lt xs attribute name  type  type  xs string  use  optional   gt               lt xs attribute name  message  type  xs string  use  optional   gt           lt  xs complexType gt       lt  xs element gt        lt xs element name  properties  gt           lt xs complexType gt               lt xs sequence gt                   lt xs element ref  property  maxOccurs  unbounded   gt               lt  xs sequence gt           lt  xs complexType gt       lt  xs element gt        lt xs element name  property  gt           lt xs complexType gt               lt xs attribute name  name  type  xs string  use  required   gt               lt xs attribute name  value  type  xs string  use  required   gt           lt  xs complexType gt       lt  xs element gt        lt xs element name  skipped  type  xs string   gt       lt xs element name  system-err  type  xs string   gt       lt xs element name  system-out  type  xs string   gt        lt xs element name  testcase  gt           lt xs complexType gt               lt xs sequence gt                   lt xs element ref  skipped  minOccurs  0  maxOccurs  1   gt                   lt xs element ref  error  minOccurs  0  maxOccurs  unbounded   gt                   lt xs element ref  failure  minOccurs  0  maxOccurs  unbounded   gt                   lt xs element ref  system-out  minOccurs  0  maxOccurs  unbounded   gt                   lt xs element ref  system-err  minOccurs  0  maxOccurs  unbounded   gt               lt  xs sequence gt               lt xs attribute name  name  type  xs string  use  required   gt               lt xs attribute name  assertions  type  xs string  use  optional   gt               lt xs attribute name  time  type  xs string  use  optional   gt               lt xs attribute name  classname  type  xs string  use  optional   gt               lt xs attribute name  status  type  xs string  use  optional   gt           lt  xs complexType gt       lt  xs element gt        lt xs element name  testsuite  gt           lt xs complexType gt               lt xs sequence gt                   lt xs element ref  properties  minOccurs  0  maxOccurs  1   gt                   lt xs element ref  testcase  minOccurs  0  maxOccurs  unbounded   gt                   lt xs element ref  system-out  minOccurs  0  maxOccurs  1   gt                   lt xs element ref  system-err  minOccurs  0  maxOccurs  1   gt               lt  xs sequence gt               lt xs attribute name  name  type  xs string  use  required   gt               lt xs attribute name  tests  type  xs string  use  required   gt               lt xs attribute name  failures  type  xs string  use  optional   gt               lt xs attribute name  errors  type  xs string  use  optional   gt               lt xs attribute name  time  type  xs string  use  optional   gt               lt xs attribute name  disabled  type  xs string  use  optional   gt               lt xs attribute name  skipped  type  xs string  use  optional   gt               lt xs attribute name  timestamp  type  xs string  use  optional   gt               lt xs attribute name  hostname  type  xs string  use  optional   gt               lt xs attribute name  id  type  xs string  use  optional   gt               lt xs attribute name  package  type  xs string  use  optional   gt           lt  xs complexType gt       lt  xs element gt        lt xs element name  testsuites  gt           lt xs complexType gt               lt xs sequence gt                   lt xs element ref  testsuite  minOccurs  0  maxOccurs  unbounded   gt               lt  xs sequence gt               lt xs attribute name  name  type  xs string  use  optional   gt               lt xs attribute name  time  type  xs string  use  optional   gt               lt xs attribute name  tests  type  xs string  use  optional   gt               lt xs attribute name  failures  type  xs string  use  optional   gt               lt xs attribute name  disabled  type  xs string  use  optional   gt               lt xs attribute name  errors  type  xs string  use  optional   gt           lt  xs complexType gt       lt  xs element gt    lt  xs schema gt

User · Answer

I ve decided to post a new answer  because some existing answers are outdated or incomplete   First of all  there is nothing like JUnit XML Format Specification  simply because JUnit doesn t produce any kind of XML or HTML report   The XML report generation itself comes from the Ant JUnit task  Maven Surefire Plugin  Gradle  whichever you use for running your tests   The XML report format was first introduced by Ant and later adapted by Maven  and Gradle    If somebody just needs an official XML format then    There exists a schema for a maven surefire-generated XML report and it can be found here  surefire-test-report xsd  For an ant-generated XML there is a 3rd party schema available here  but it might be slightly outdated     Hope it will help somebody

User · Answer

I couldn t find any good information on this  so I did some trial and error   The following attributes and fields  and only these  are recognized by Jenkins  v1 585     lt  xml version  1 0  encoding  UTF-8   gt   lt testsuite gt      lt  -- if your classname does not include a dot  the package defaults to   root   -- gt     lt testcase name  my testcase  classname  my package my classname  time  29  gt        lt  -- If the test didn t pass  specify ONE of the following 3 cases -- gt        lt  -- option 1 -- gt   lt skipped   gt       lt  -- option 2 -- gt   lt failure message  my failure message  gt my stack trace lt  failure gt       lt  -- option 3 -- gt   lt error message  my error message  gt my crash report lt  error gt        lt system-out gt my STDOUT dump lt  system-out gt        lt system-err gt my STDERR dump lt  system-err gt      lt  testcase gt    lt  testsuite gt     I started with this sample XML document and worked backwards from there

User · Answer

I did a similar thing a few months ago  and it turned out this simple format was enough for Hudson to accept it as a test protocol    lt testsuite tests  3  gt       lt testcase classname  foo1  name  ASuccessfulTest   gt       lt testcase classname  foo2  name  AnotherSuccessfulTest   gt       lt testcase classname  foo3  name  AFailingTest  gt           lt failure type  NotEnoughFoo  gt  details about failure  lt  failure gt       lt  testcase gt   lt  testsuite gt    This question has answers with more details  Spec  for JUnit XML Output

[jenkins] What is the JUnit XML format specification that Hudson supports?

Examples related to jenkins

Examples related to junit

Examples related to hudson

Examples related to xunit