Controlling execution order of unit tests in Visual Studio

Question

Okay  I m done searching for good information on this  I have a series of Unit Tests that call a static class which  once initialized  sets properties that cannot  or I don t wish to  change   My problem is I cannot enforce a set order for the tests to run  If I could  I could run them in such a way as the static properties would be set in a reliable way  and I could Assert on them  but unfortunately the Microsoft VisualStudio TestTools UnitTesting framework just runs them in a seemingly random order   So  I found this http   msdn microsoft com en-us library microsoft visualstudio testtools unittesting priorityattribute aspx which says in the Remarks section  This attribute is not used by the test system  It is provided to the user for custom purposes   Huh  What good is it then  Do they expect me to write my own testing wrapper to take advantage of this fabulous attribute  of which I could easily write myself if I wanted to go to that level of effort      So  enough of the rant  Bottom line  is there a way to control the order my unit tests run     TestMethod   Priority 0     etc  does NOT seem to work  which makes sense  since Microsoft says it won t   Also  please no comments about  violating isolation   The TestClass isolates what I am testing  not the individual TestMethods  Regardless  each test can be run independently just fine  they just can t be run together in a random order as there is no way to tear down the static class   Oh  I also know about  Ordered Test

User · Answer

I ll not address the order of tests  sorry  Others already did it  Also  if you know about  ordered tests  - well  this is MS VS s response to the problem  I know that those ordered-tests are no fun  But they thought it will be  it  and there s really nothing more in MSTest about that   I write about one of your assumptions      as there is no way to tear down the static class    Unless your static class represents some process-wide external state external to your code  like ie  the state of an unmanaged native DLL library thats P Invoked by the rest of your code   your assumption that there is no way is not true   If your static class refers to this  then sorry  you are perfectly right  the rest of this anwer is irrelevant  Still  as you didn t say that  I assume your code is  managed    Think and check the AppDomain thingy  Rarely it is needed  but this is exactly the case when you d probably like to use them   You can create a new AppDomain  and instantiate the test there  and run the test method there  Static data used by managed code will isolated there and upon completion  you will be able to unload the AppDomain and all the data  statics included  will evaporate  Then  next test would initialize another appdomain  and so on   This will work unless you have external state that you must track  AppDomains only isolate the managed memory  Any native DLL will still be load per-process and their state will be shared by all AppDomains   Also  creating tearing down the appdomains will  well  slow down the tests  Also  you may have problems with assembly resolution in the child appdomain  but they are solvable with reasonable amount of reusable code   Also  you may have small problems with passing test data to - and back from - the child AppDomain  Objects passed will either have to be serializable in some way  or be MarshalByRef or etc  Talking cross-domain is almost like IPC   However  take care here  it will be 100  managed talking  If you take some extra care and add a little work to the AppDomain setup  you will be able to even pass delegates and run them in the target domain  Then  instead of making some hairy cross-domain setup  you can wrap your tests with to something like   void testmethod         TestAppDomainHelper Run       gt                   your test code             or even   IsolatedAppDomain  void testmethod            your test code     if your test framework supports creating such wrappers extensions  After some initial research and work  using them is almost trivial

User · Answer

Merge your tests into one giant test will work  To make the test method more readable  you can do something like   TestMethod  public void MyIntegratonTestLikeUnitTest         AssertScenarioA         AssertScenarioB                 private void AssertScenarioA             Assert    private void AssertScenarioB             Assert     Actually the issue you have suggests you probably should improve the testability of the implementation

User · Answer

You can Use Playlist   Right click on the test method -  Add to playlist -  New playlist  the execution order will be as you add them to the playlist but if you want to change it you have the file

User · Answer

As you should know by now  purists say it s forbiden to run ordered tests  That might be true for unit tests  MSTest and other Unit Test frameworks are used to run pure unit test but also UI tests  full integration tests  you name it  Maybe we shouldn t call them Unit Test frameworks  or maybe we should and use them according to our needs  That s what most people do anyway   I m running VS2015 and I MUST run tests in a given order because I m running UI tests  Selenium    Priority - Doesn t do anything at all This attribute is not used by the test system  It is provided to the user for custom purposes   orderedtest - it works but I don t recommend it because    An orderedtest a text file that lists your tests in the order they should be executed  If you change a method name  you must fix the file  The test execution order is respected inside a class  You can t order which class executes its tests first  An orderedtest file is bound to a configuration  either Debug or Release You can have several orderedtest files but a given method can not be repeated in different orderedtest files  So you can t have one orderedtest file for Debug and another for Release    Other suggestions in this thread are interesting but you loose the ability to follow the test progress on Test Explorer   You are left with the solution that purist will advise against  but in fact is the solution that works  sort by declaration order   The MSTest executor uses an interop that manages to get the declaration order and this trick will work until Microsoft changes the test executor code   This means the test method that is declared in the first place executes before the one that is declared in second place  etc   To make your life easier  the declaration order should match the alphabetical order that is is shown in the Test Explorer    A010 FirstTest A020 SecondTest etc A100 TenthTest   I strongly suggest some old and tested rules    use a step of 10 because you will need to insert a test method later on avoid the need to renumber your tests by using a generous step between test numbers use 3 digits to number your tests if you are running more than 10 tests use 4 digits to number your tests if you are running more than 100 tests   VERY IMPORTANT  In order to execute the tests by the declaration order  you must use Run All in the Test Explorer   Say you have 3 test classes  in my case tests for Chrome  Firefox and Edge   If you select a given class and right click Run Selected Tests it usually starts by executing the method declared in the last place   Again  as I said before  declared order and listed order should match or else you ll in big trouble in no time

User · Answer

I dont see anyone mentioning the ClassInitialize attribute method   The attributes are pretty straight forward   Create methods that are marked with either the  ClassInitialize    or  TestInitialize    attribute to prepare aspects of the environment in which your unit test will run  The purpose of this is to establish a known state for running your unit test  For example  you may use the  ClassInitialize    or the  TestInitialize    method to copy  alter  or create certain data files that your test will use   Create methods that are marked with either the  ClassCleanup    or  TestCleanUp    attribute to return the environment to a known state after a test has run  This might mean the deletion of files in folders or the return of a database to a known state  An example of this is to reset an inventory database to an initial state after testing a method that is used in an order-entry application     ClassInitialize    Use ClassInitialize to run code before you run the first test in the class   ClassCleanUp    Use ClassCleanup to run code after all tests in a class have run   TestInitialize    Use TestInitialize to run code before you run each test   TestCleanUp    Use TestCleanup to run code after each test has run

User · Answer

they just can t be run together in a random order as there is no way to tear down the static class   You can name namespaces and classes in alphabetical order  eg     MyApp Test Stage01 Setup Step01 BuildDB MyApp Test Stage01 Setup Step02 UpgradeDB MyApp Test Stage02 Domain Step01 TestMyStaff MyApp Test Stage03 Integration Step01 TestMyStaff   where MyApp Test Stage01 Setup is a namespace and Step01 BuildDB is a class name

User · Answer

I see that this topic is almost 6 years old  and we now have new version of Visual studio but I will reply anyway  I had that order problem in Visual Studio 19 and I figured it out by adding capital letter  you can also add small letter  in front of your method name and in alphabetical order like this    TestMethod          public void AName1               TestMethod          public void BName2                And so on  I know that this doesn t look appealing  but it looks like Visual is sorting your tests in test explorer in alphabetical order  doesn t matter how you write it in your code  Playlist didn t work for me in this case   Hope that this will help

User · Answer

Here is a class that can be used to setup and run ordered tests independent of MS Ordered Tests framework for whatever reason--like not have to adjust mstest exe arguments on a build machine  or mixing ordered with non-ordered in a class     The original testing framework only sees the list of ordered tests as a single test so any init cleanup like  TestInitalize    Init   is only called before and after the entire set   Usage            TestMethod     place only on the list--not the individuals         public void OrderedStepsTest                         OrderedTest Run TestContext  new List lt OrderedTest gt                                new OrderedTest   T10 Reset Database  false                    new OrderedTest   T20 LoginUser1  false                    new OrderedTest   T30 DoLoginUser1Task1  true       continue on failure                 new OrderedTest   T40 DoLoginUser1Task2  true       continue on failure                                                                    Implementation   using Microsoft VisualStudio TestTools UnitTesting  using System  using System Collections Generic  using System Diagnostics  using System Linq   namespace UnitTests Utility                lt summary gt          Define and Run a list of ordered tests           2016 08 25  Posted to SO by crokusek           lt  summary gt          public class OrderedTest                    lt summary gt Test Method to run lt  summary gt          public Action TestMethod   get  private set                  lt summary gt Flag indicating whether testing should continue with the next test if the current one fails lt  summary gt          public bool ContinueOnFailure   get  private set                  lt summary gt Any Exception thrown by the test lt  summary gt          public Exception ExceptionResult                lt summary gt              Constructor              lt  summary gt               lt param name  testMethod  gt  lt  param gt               lt param name  continueOnFailure  gt True to continue with the next test if this test fails lt  param gt          public OrderedTest Action testMethod  bool continueOnFailure   false                        TestMethod   testMethod              ContinueOnFailure   continueOnFailure                          lt summary gt              Run the test saving any exception within ExceptionResult             Throw to the caller only if ContinueOnFailure    false              lt  summary gt               lt param name  testContextOpt  gt  lt  param gt          public void Run                         try                               TestMethod                              catch  Exception ex                                ExceptionResult   ex                  throw                                        lt summary gt              Run a list of OrderedTest s              lt  summary gt          static public void Run TestContext testContext  List lt OrderedTest gt  tests                        Stopwatch overallStopWatch   new Stopwatch                overallStopWatch Start                 List lt Exception gt  exceptions   new List lt Exception gt                  int testsAttempted   0              for  int i   0  i  lt  tests Count  i                                  OrderedTest test   tests i                    Stopwatch stopWatch   new Stopwatch                    stopWatch Start                     testContext WriteLine  Starting ordered test step   0  of  1     2   at  3     n                       i   1                      tests Count                      test TestMethod Method                      DateTime Now ToString  G                      try                                       testsAttempted                        test Run                                      catch                                       if   test ContinueOnFailure                          break                                    finally                                       Exception testEx   test ExceptionResult                       if  testEx    null      capture any  continue on fail  exception                         exceptions Add testEx                        testContext WriteLine   n 0  ordered test step  1  of  2    3   in  4  at  5  6  n                           testEx    null    Error   Failed     Successfully completed                           i   1                          tests Count                          test TestMethod Method                          stopWatch ElapsedMilliseconds  gt  1000                                stopWatch ElapsedMilliseconds    001     s                                stopWatch ElapsedMilliseconds    ms                           DateTime Now ToString  G                            testEx    null                                 nException       testEx Message                                     nStackTrace       testEx StackTrace                                     nContinueOnFailure       test ContinueOnFailure                                                                                 testContext WriteLine  Completed running  0  of  1  ordered tests with a total of  2  error s  at  3  in  4                    testsAttempted                  tests Count                  exceptions Count                  DateTime Now ToString  G                    overallStopWatch ElapsedMilliseconds  gt  1000                        overallStopWatch ElapsedMilliseconds    001     s                        overallStopWatch ElapsedMilliseconds    ms                 if  exceptions Any                                     Test Explorer prints better msgs with this hierarchy rather than using 1 AggregateException                    throw new Exception String Join       exceptions Select e   gt  e Message   new AggregateException exceptions

User · Answer

Since you ve already mentioned the Ordered Test functionality that the Visual Studio testing framework supplies  I ll ignore that  You also seem to be aware that what you re trying to accomplish in order to test this Static Class is a  bad idea   so I ll ignore that to   Instead  lets focus on how you might actually be able to guarantee that your tests are executed in the order you want  One option  as supplied by  gaog  is  one test method  many test functions   calling your test functions in the order that you want from within a single function marked with the TestMethod attribute  This is the simplest way  and the only disadvantage is that the first test function to fail will prevent any of the remaining test functions from executing    With your description of the situation  this is the solution I would suggest you use   If the bolded part is a problem for you  you can accomplish an ordered execution of isolated tests by leveraging the in built data driven test functionality  Its more complicated and feels a bit dirty  but it gets the job done   In short  you define a data source  like a CSV file  or a database table  that controls the order in which you need to run your tests  and names of the functions that actually contain the test functionality  You then hook that data source into a data driven test  use the sequential read option  and execute your functions  in the order you want  as individual tests    TestClass  public class OrderedTests       public TestContext TestContext   get  set         private const string  OrderedTestFilename    TestList csv         TestMethod       DeploymentItem  OrderedTestFilename        DataSource  Microsoft VisualStudio TestTools DataSource CSV    OrderedTestFilename   OrderedTestFilename  DataAccessMethod Sequential       public void OrderedTests                 var methodName    string TestContext DataRow 0           var method   GetType   GetMethod methodName           method Invoke this  new object                    public void Method 01                 Assert IsTrue true              public void Method 02                 Assert IsTrue false              public void Method 03                 Assert IsTrue true             In my example  I have a supporting file called TestList csv  which gets copied to output  It looks like this   TestName Method 01 Method 02 Method 03   Your tests will be executed in the order that you specified  and in normal test isolation  i e  if one fails  the rest still get executed  but sharing static classes     The above is really only the basic idea  if I were to use it in production I would generate the test function names and their order dynamically before the test is run  Perhaps by leveraging PriorityAttribute you found and some simple reflection code to extract the test methods in the class and order them appropriately  then write that order to the data source

[c#] Controlling execution order of unit tests in Visual Studio

Examples related to c#

Examples related to unit-testing

Examples related to visual-studio-2012