[java] How to run test methods in specific order in JUnit4?

I want to execute test methods which are annotated by @Test in specific order.

For example:

public class MyTest {
    @Test public void test1(){}
    @Test public void test2(){}
}

I want to ensure to run test1() before test2() each time I run MyTest, but I couldn't find annotation like @Test(order=xx).

I think it's quite important feature for JUnit, if author of JUnit doesn't want the order feature, why?

Here is an extension to JUnit that can produce the desired behavior: https://github.com/aafuks/aaf-junit

I know that this is against the authors of JUnit philosophy, but when using JUnit in environments that are not strict unit testing (as practiced in Java) this can be very helpful.

I've read a few answers and agree its not best practice, but the easiest way to order your tests - and the way that JUnit runs tests by default is by alphabetic name ascending.

So just name your tests in the alphabetic order that you want. Also note the test name must begin with the word test. Just watch out for numbers

test12 will run before test2

so:

testA_MyFirstTest testC_ThirdTest testB_ATestThatRunsSecond

Please check out this one: https://github.com/TransparentMarket/junit. It runs the test in the order they are specified (defined within the compiled class file). Also it features a AllTests suite to run tests defined by sub package first. Using the AllTests implementation one can extend the solution in also filtering for properties (we used to use @Fast annotations but those were not published yet).

Not sure I agree, If I want to test 'File Upload' and then test 'Data Inserted by File Upload' why would I not want these to be independent from each other? Perfectly reasonable I think to be able to run them separately rather than having both in a Goliath test case.

You can use one of these piece of codes: @FixMethodOrder(MethodSorters.JVM) OR @FixMethodOrder(MethodSorters.DEFAULT) OR @FixMethodOrder(MethodSorters.NAME_ASCENDING)

Before your test class like this:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class BookTest {...}

See my solution here: "Junit and java 7."

In this article I describe how to run junit tests in order - "just as in your source code". Tests will be run, in order as your test methods appears in class file.

http://intellijava.blogspot.com/2012/05/junit-and-java-7.html

But as Pascal Thivent said, this is not a good practise.

Its one of the main issue which I faced when I worked on Junit and I came up with following solution which works fine for me:

import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.FrameworkMethod;
import org.junit.runners.model.InitializationError;

public class OrderedRunner extends BlockJUnit4ClassRunner {

    public OrderedRunner(Class<?> clazz) throws InitializationError {
        super(clazz);
    }

    @Override
    protected List<FrameworkMethod> computeTestMethods() {
        List<FrameworkMethod> list = super.computeTestMethods();
        List<FrameworkMethod> copy = new ArrayList<FrameworkMethod>(list);
        Collections.sort(copy, new Comparator<FrameworkMethod>() {

            @Override
            public int compare(FrameworkMethod f1, FrameworkMethod f2) {
                Order o1 = f1.getAnnotation(Order.class);
                Order o2 = f2.getAnnotation(Order.class);

                if (o1 == null || o2 == null) {
                    return -1;
                }

                return o1.order() - o2.order();
            }
        });
        return copy;
    }
}

also create a interface like below:

 @Retention(RetentionPolicy.RUNTIME)


@Target({ ElementType.METHOD})

public @interface Order {
public int order();
}

Now suppose you have class A where you have written several test cases like below:

(@runWith=OrderRunner.class)
Class A{
@Test
@Order(order = 1)

void method(){

//do something

}

}

So execution will start from method named "method()". Thanks!

With JUnit 5.4, you can specify the order :

@Test
@Order(2)
public void sendEmailTestGmail() throws MessagingException {

you just need to annotate your class

@TestMethodOrder(OrderAnnotation.class)

https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order

i'm using this in my project and it works very well !

If the order is important, you should make the order yourself.

@Test public void test1() { ... }
@Test public void test2() { test1(); ... }

In particular, you should list some or all possible order permutations to test, if necessary.

For example,

void test1(); 
void test2(); 
void test3(); 


@Test
public void testOrder1() { test1(); test3(); }

@Test(expected = Exception.class)
public void testOrder2() { test2(); test3(); test1(); }

@Test(expected = NullPointerException.class)
public void testOrder3() { test3(); test1(); test2(); }

Or, a full test of all permutations:

@Test
public void testAllOrders() {
    for (Object[] sample: permute(1, 2, 3)) {
        for (Object index: sample) {
            switch (((Integer) index).intValue()) {
                case 1: test1(); break; 
                case 2: test2(); break; 
                case 3: test3(); break; 
            }
        }
    }
}

Here, permute() is a simple function which iterates all possible permuations into a Collection of array.

If you want to run test methods in a specific order in JUnit 5, you can use the below code.

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class MyClassTest { 

    @Test
    @Order(1)
    public void test1() {}

    @Test
    @Order(2)
    public void test2() {}

}

I ended up here thinking that my tests weren't run in order, but the truth is that the mess was in my async jobs. When working with concurrency you need to perform concurrency checks between your tests as well. In my case, jobs and tests share a semaphore, so next tests hang until the running job releases the lock.

I know this is not fully related to this question, but maybe could help targeting the correct issue

Migration to TestNG seems the best way, but I see no clear solution here for jUnit. Here is most readable solution / formatting I found for jUnit:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class SampleTest {
    @Test
    void stage1_prepareAndTest(){};

    @Test
    void stage2_checkSomething(){};

    @Test
    void stage2_checkSomethingElse(){};

    @Test
    void stage3_thisDependsOnStage2(){};

    @Test
    void callTimeDoesntMatter(){}
}

This ensures stage2 methods are called after stage1 ones and before stage3 ones.

The (as yet unreleased) change https://github.com/junit-team/junit/pull/386 introduces a @SortMethodsWith. https://github.com/junit-team/junit/pull/293 at least made the order predictable without that (in Java 7 it can be quite random).

JUnit 5 update (and my opinion)

I think it's quite important feature for JUnit, if author of JUnit doesn't want the order feature, why?

By default, unit testing libraries don't try to execute tests in the order that occurs in the source file.
JUnit 5 as JUnit 4 work in that way. Why ? Because if the order matters it means that some tests are coupled between them and that is undesirable for unit tests.
So the @Nested feature introduced by JUnit 5 follows the same default approach.

But for integration tests, the order of the test method may matter since a test method may change the state of the application in a way expected by another test method. For example when you write an integration test for a e-shop checkout processing, the first test method to be executed is registering a client, the second is adding items in the basket and the last one is doing the checkout. If the test runner doesn't respect that order, the test scenario is flawed and will fail.
So in JUnit 5 (from the 5.4 version) you have all the same the possibility to set the execution order by annotating the test class with @TestMethodOrder(OrderAnnotation.class) and by specifying the order with @Order(numericOrderValue) for the methods which the order matters.

For example :

@TestMethodOrder(OrderAnnotation.class) 
class FooTest {

    @Order(3)
    @Test
    void checkoutOrder() {
        System.out.println("checkoutOrder");
    }

    @Order(2)
    @Test
    void addItemsInBasket() {
        System.out.println("addItemsInBasket");
    }

    @Order(1)
    @Test
    void createUserAndLogin() {
        System.out.println("createUserAndLogin");
    }
}

Output :

createUserAndLogin

addItemsInBasket

checkoutOrder

By the way, specifying @TestMethodOrder(OrderAnnotation.class) looks like not needed (at least in the 5.4.0 version I tested).

Side note
About the question : is JUnit 5 the best choice to write integration tests ? I don't think that it should be the first tool to consider (Cucumber and co may often bring more specific value and features for integration tests) but in some integration test cases, the JUnit framework is enough. So that is a good news that the feature exists.

If you get rid of your existing instance of Junit, and download JUnit 4.11 or greater in the build path, the following code will execute the test methods in the order of their names, sorted in ascending order:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class SampleTest {

    @Test
    public void testAcreate() {
        System.out.println("first");
    }
    @Test
    public void testBupdate() {
        System.out.println("second");
    }
    @Test
    public void testCdelete() {
        System.out.println("third");
    }
}

JUnit since 5.5 allows @TestMethodOrder(OrderAnnotation.class) on class and @Order(1) on test-methods.

JUnit old versions allow test methods run ordering using class annotations:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
@FixMethodOrder(MethodSorters.JVM)
@FixMethodOrder(MethodSorters.DEFAULT)

By default test methods are run in alphabetical order. So, to set specific methods order you can name them like:

a_TestWorkUnit_WithCertainState_ShouldDoSomething b_TestWorkUnit_WithCertainState_ShouldDoSomething c_TestWorkUnit_WithCertainState_ShouldDoSomething

Or

_1_TestWorkUnit_WithCertainState_ShouldDoSomething _2_TestWorkUnit_WithCertainState_ShouldDoSomething _3_TestWorkUnit_WithCertainState_ShouldDoSomething

You can find examples here.

Look at a JUnit report. JUnit is already organized by package. Each package has (or can have) TestSuite classes, each of which in turn run multiple TestCases. Each TestCase can have multiple test methods of the form public void test*(), each of which will actually become an instance of the TestCase class to which they belong. Each test method (TestCase instance) has a name and a pass/fail criteria.

What my management requires is the concept of individual TestStep items, each of which reports their own pass/fail criteria. Failure of any test step must not prevent the execution of subsequent test steps.

In the past, test developers in my position organized TestCase classes into packages that correspond to the part(s) of the product under test, created a TestCase class for each test, and made each test method a separate "step" in the test, complete with its own pass/fail criteria in the JUnit output. Each TestCase is a standalone "test", but the individual methods, or test "steps" within the TestCase, must occur in a specific order.

The TestCase methods were the steps of the TestCase, and test designers got a separate pass/fail criterion per test step. Now the test steps are jumbled, and the tests (of course) fail.

For example:

Class testStateChanges extends TestCase

public void testCreateObjectPlacesTheObjectInStateA()
public void testTransitionToStateBAndValidateStateB()
public void testTransitionToStateCAndValidateStateC()
public void testTryToDeleteObjectinStateCAndValidateObjectStillExists()
public void testTransitionToStateAAndValidateStateA()
public void testDeleteObjectInStateAAndObjectDoesNotExist()
public void cleanupIfAnythingWentWrong()

Each test method asserts and reports its own separate pass/fail criteria. Collapsing this into "one big test method" for the sake of ordering loses the pass/fail criteria granularity of each "step" in the JUnit summary report. ...and that upsets my managers. They are currently demanding another alternative.

Can anyone explain how a JUnit with scrambled test method ordering would support separate pass/fail criteria of each sequential test step, as exemplified above and required by my management?

Regardless of the documentation, I see this as a serious regression in the JUnit framework that is making life difficult for lots of test developers.

What you want is perfectly reasonable when test cases are being run as a suite.

Unfortunately no time to give a complete solution right now, but have a look at class:

org.junit.runners.Suite

Which allows you to call test cases (from any test class) in a specific order.

These might be used to create functional, integration or system tests.

This leaves your unit tests as they are without specific order (as recommended), whether you run them like that or not, and then re-use the tests as part of a bigger picture.

We re-use/inherit the same code for unit, integration and system tests, sometimes data driven, sometimes commit driven, and sometimes run as a suite.