How should I unit test multithreaded code

Question

I have thus far avoided the nightmare that is testing multi-threaded code since it just seems like too much of a minefield   I d like to ask how people have gone about testing code that relies on threads for successful execution  or just how people have gone about testing those kinds of issues that only show up when two threads interact in a given manner   This seems like a really key problem for programmers today  it would be useful to pool our knowledge on this one imho

User · Answer

Concurrency is a complex interplay between the memory model, hardware, caches and our code. In the case of Java at least such tests have been partly addressed mainly by jcstress. The creators of that library are known to be authors of many JVM, GC and Java concurrency features.

But even this library needs good knowledge of the Java Memory Model specification so that we know exactly what we are testing. But I think the focus of this effort is mircobenchmarks. Not huge business applications.

User · Answer

For J2E code  I ve used SilkPerformer  LoadRunner and JMeter for concurrency testing of threads   They all do the same thing   Basically  they give you a relatively simple interface for administrating their version of the proxy server  required  in order to analyze the TCP IP data stream  and simulate multiple users making simultaneous requests to your app server   The proxy server can give you the ability to do things like analyze the requests made  by presenting the whole page and URL sent to the server  as well as the response from the server  after processing the request   You can find some bugs in insecure http mode  where you can at least analyze the form data that is being sent  and systematically alter that for each user   But the true tests are when you run in https  Secured Socket Layers    Then  you also have to contend with systematically altering the session and cookie data  which can be a little more convoluted   The best bug I ever found  while testing concurrency  was when I discovered that the developer had relied upon Java garbage collection to close the connection request that was established at login  to the LDAP server  when logging in   This resulted in users being exposed to other users  sessions and very confusing results  when trying to analyze what happened when the server was brought to it s knees  barely able to complete one transaction  every few seconds   In the end  you or someone will probably have to buckle down and analyze the code for blunders like the one I just mentioned   And an open discussion across departments  like the one that occurred  when we unfolded the problem described above  are most useful   But these tools are the best solution to testing multi-threaded code   JMeter is open source   SilkPerformer and LoadRunner are proprietary   If you really want to know whether your app is thread safe  that s how the big boys do it   I ve done this for very large companies professionally  so I m not guessing   I m speaking from personal experience   A word of caution   it does take some time to understand these tools   It will not be a matter of simply installing the software and firing up the GUI  unless you ve already had some exposure to multi-threaded programming   I ve tried to identify the 3 critical categories of areas to understand  forms  session and cookie data   with the hope that at least starting with understanding these topics will help you focus on quick results  as opposed to having to read through the entire documentation

User · Answer

Assuming under  multi-threaded  code was meant something that is    stateful and mutable AND accessed modified by multiple threads concurrently   In other words we are talking about testing custom stateful thread-safe class method unit - which should be a very rare beast nowadays   Because this beast is rare  first of all we need to make sure that there are all valid excuses to write it   Step 1  Consider modifying state in same synchronization context   Today it is easy to write compose-able concurrent and asynchronous code where IO or other slow operations offloaded to background but shared state is updated and queried in one synchronization context  e g  async await tasks and Rx in  NET etc  - they are all testable by design   real  Tasks and schedulers can be substituted to make testing deterministic  however this is out of scope of the question     It may sound very constrained but this approach works surprisingly well  It is possible to write whole apps in this style without need to make any state thread-safe  I do    Step 2  If manipulating of shared state on single synchronization context is absolutely not possible    Make sure the wheel is not being reinvented   there s definitely no standard alternative that can be adapted for the job  It should be likely that code is very cohesive and contained within one unit e g  with a good chance it is a special case of some standard thread-safe data structure like hash map or collection or whatever   Note  if code is large   spans across multiple classes AND needs multi-thread state manipulation then there s a very high chance that design is not good  reconsider Step 1   Step 3  If this step is reached then we need to test our own custom stateful thread-safe class method unit   I ll be dead honest   I never had to write proper tests for such code  Most of the time I get away at Step 1  sometimes at Step 2  Last time I had to write custom thread-safe code was so many years ago that it was before I adopted unit testing   probably I wouldn t have to write it with the current knowledge anyway   If I really had to test such code  finally  actual answer  then I would try couple of things below   Non-deterministic stress testing  e g  run 100 threads simultaneously and check that end result is consistent  This is more typical for higher level   integration testing of multiple users scenarios but also can be used at the unit level  Expose some test  hooks  where test can inject some code to help make deterministic scenarios where one thread must perform operation before the other  As ugly as it is  I can t think of anything better  Delay-driven testing to make threads run and perform operations in particular order  Strictly speaking such tests are non-deterministic too  there s a chance of system freeze   stop-the-world GC collection which can distort otherwise orchestrated delays   also it is ugly but allows to avoid hooks

User · Answer

It s been a while when this question was posted  but it s still not answered      kleolb02 s answer is a good one  I ll try going into more details   There is a way  which I practice for C  code  For unit tests you should be able to program reproducible tests  which is the biggest challenge in multithreaded code  So my answer aims toward forcing asynchronous code into a test harness  which works synchronously   It s an idea from Gerard Meszardos s book  xUnit Test Patterns  and is called  Humble Object   p  695   You have to separate core logic code and anything which smells like asynchronous code from each other  This would result to a class for the core logic  which works synchronously   This puts you into the position to test the core logic code in a synchronous way  You have absolute control over the timing of the calls you are doing on the core logic and thus can make reproducible tests  And this is your gain from separating core logic and asynchronous logic   This core logic needs be wrapped around by another class  which is responsible for receiving calls to the core logic asynchronously and delegates these calls to the core logic  Production code will only access the core logic via that class  Because this class should only delegate calls  it s a very  dumb  class without much logic  So you can keep your unit tests for this asychronous working class at a minimum   Anything above that  testing interaction between classes  are component tests  Also in this case  you should be able to have absolute control over timing  if you stick to the  Humble Object  pattern

User · Answer

I like to write two or more test methods to execute on parallel threads  and each of them make calls into the object under test  I ve been using Sleep   calls to coordinate the order of the calls from the different threads  but that s not really reliable  It s also a lot slower because you have to sleep long enough that the timing usually works   I found the Multithreaded TC Java library from the same group that wrote FindBugs  It lets you specify the order of events without using Sleep    and it s reliable  I haven t tried it yet   The biggest limitation to this approach is that it only lets you test the scenarios you suspect will cause trouble  As others have said  you really need to isolate your multithreaded code into a small number of simple classes to have any hope of thoroughly testing them   Once you ve carefully tested the scenarios you expect to cause trouble  an unscientific test that throws a bunch of simultaneous requests at the class for a while is a good way to look for unexpected trouble   Update  I ve played a bit with the Multithreaded TC Java library  and it works well  I ve also ported some of its features to a  NET version I call TickingTest

User · Answer

It s not perfect  but I wrote this helper for my tests in C   using System  using System Collections Generic  using System Threading  using System Threading Tasks   namespace Proto Promises Tests Threading       public class ThreadHelper               public static readonly int multiThreadCount   Environment ProcessorCount   100          private static readonly int   offsets   new int     0  10  100  1000             private readonly Stack lt Task gt   executingTasks   new Stack lt Task gt  multiThreadCount           private readonly Barrier  barrier   new Barrier 1           private int  currentParticipants   0          private readonly TimeSpan  timeout           public ThreadHelper     this TimeSpan FromSeconds 10          10 second timeout should be enough for most cases           public ThreadHelper TimeSpan timeout                         timeout   timeout                          lt summary gt              Execute the action multiple times in parallel threads               lt  summary gt          public void ExecuteMultiActionParallel Action action                        for  int i   0  i  lt  multiThreadCount    i                                AddParallelAction action                             ExecutePendingParallelActions                            lt summary gt              Execute the action once in a separate thread               lt  summary gt          public void ExecuteSingleAction Action action                        AddParallelAction action               ExecutePendingParallelActions                            lt summary gt              Add an action to be run in parallel               lt  summary gt          public void AddParallelAction Action action                        var taskSource   new TaskCompletionSource lt bool gt                 lock   executingTasks                                   currentParticipants                   barrier AddParticipant                     executingTasks Push taskSource Task                             new Thread      gt                                try                                        barrier SignalAndWait       Try to make actions run in lock-step to increase likelihood of breaking race conditions                      action Invoke                        taskSource SetResult true                                     catch  Exception e                                        taskSource SetException e                                    Start                            lt summary gt              Runs the pending actions in parallel  attempting to run them in lock-step               lt  summary gt          public void ExecutePendingParallelActions                         Task   tasks              lock   executingTasks                                 barrier SignalAndWait                     barrier RemoveParticipants  currentParticipants                    currentParticipants   0                  tasks    executingTasks ToArray                     executingTasks Clear                              try                               if   Task WaitAll tasks   timeout                                         throw new TimeoutException   quot Action s  timed out after   timeout   there may be a deadlock  quot                                                catch  AggregateException e                                   Only throw one exception instead of aggregate to try to avoid overloading the test error output                  throw e Flatten   InnerException                                        lt summary gt              Run each action in parallel multiple times with differing offsets for each run               lt para  gt The number of runs is 4 actions Length  so be careful if you don t want the test to run too long               lt  summary gt               lt param name  quot expandToProcessorCount quot  gt If true  copies each action on additional threads up to the processor count  This can help test more without increasing the time it takes to complete               lt para  gt Example  2 actions with 6 processors  runs each action 3 times in parallel  lt  param gt               lt param name  quot setup quot  gt The action to run before each parallel run  lt  param gt               lt param name  quot teardown quot  gt The action to run after each parallel run  lt  param gt               lt param name  quot actions quot  gt The actions to run in parallel  lt  param gt          public void ExecuteParallelActionsWithOffsets bool expandToProcessorCount  Action setup  Action teardown  params Action   actions                        setup         gt                   teardown         gt                   int actionCount   actions Length              int expandCount   expandToProcessorCount   Math Max Environment ProcessorCount   actionCount  1    1              foreach  var combo in GenerateCombinations offsets  actionCount                                 setup Invoke                    for  int k   0  k  lt  expandCount    k                                        for  int i   0  i  lt  actionCount    i                                                int offset   combo i                           Action action   actions i                           AddParallelAction      gt                                                        for  int j   offset  j  gt  0  --j         Just spin in a loop for the offset                              action Invoke                                                                                        ExecutePendingParallelActions                    teardown Invoke                                        Input   1  2  3   3            Ouput                         1  1  1                        2  1  1                        3  1  1                        1  2  1                        2  2  1                        3  2  1                        1  3  1                        2  3  1                        3  3  1                        1  1  2                        2  1  2                        3  1  2                        1  2  2                        2  2  2                        3  2  2                        1  3  2                        2  3  2                        3  3  2                        1  1  3                        2  1  3                        3  1  3                        1  2  3                        2  2  3                        3  2  3                        1  3  3                        2  3  3                        3  3  3                              private static IEnumerable lt int   gt  GenerateCombinations int   options  int count                        int   indexTracker   new int count               int   combo   new int count               for  int i   0  i  lt  count    i                                combo i    options 0                                Same algorithm as picking a combination lock              int rollovers   0              while  rollovers  lt  count                                yield return combo     No need to duplicate the array since we re just reading it                  for  int i   0  i  lt  count    i                                        int index     indexTracker i                       if  index    options Length                                                indexTracker i    0                          combo i    options 0                           if  i    rollovers                                                          rollovers                                                                      else                                               combo i    options index                           break                                                                           Example usage   Test  public void DeferredMayBeBeResolvedAndPromiseAwaitedConcurrently void0         Promise Deferred deferred   default Promise Deferred       Promise promise   default Promise        int invokedCount   0       var threadHelper   new ThreadHelper        threadHelper ExecuteParallelActionsWithOffsets false             Setup              gt                        invokedCount   0              deferred   Promise NewDeferred                promise   deferred Promise                        Teardown              gt  Assert AreEqual 1  invokedCount              Parallel Actions              gt  deferred Resolve                 gt  promise Then      gt    Interlocked Increment ref invokedCount      Forget

User · Answer

I have had the unfortunate task of testing threaded code and they are definitely the hardest tests I have ever written   When writing my tests  I used a combination of delegates and events   Basically it is all about using PropertyNotifyChanged events with a WaitCallback or some kind of ConditionalWaiter that polls   I am not sure if this was the best approach  but it has worked out for me

User · Answer

Pete Goodliffe has a series on the unit testing of threaded code   It s hard  I take the easier way out and try to keep the threading code abstracted from the actual test  Pete does mention that the way I do it is wrong but I ve either got the separation right or I ve just been lucky

User · Answer

Testing MT code for correctness is  as already stated  quite a hard problem  In the end it boils down to ensuring that there are no incorrectly synchronised data races in your code  The problem with this is that there are infinitely many possibilities of thread execution  interleavings  over which you do not have much control  be sure to read this article  though   In simple scenarios it might be possible to actually prove correctness by reasoning but this is usually not the case  Especially if you want to avoid minimize synchronization and not go for the most obvious easiest synchronization option   An approach that I follow is to write highly concurrent test code in order to make potentially undetected data races likely to occur  And then I run those tests for some time    I once stumbled upon a talk where some computer scientist where showing off a tool that kind of does this  randomly devising test from specs and then running them wildly  concurrently  checking for the defined invariants to be broken    By the way  I think this aspect of testing MT code has not been mentioned here  identify invariants of the code that you can check for randomly  Unfortunately  finding those invariants is quite a hard problem  too  Also they might not hold all the time during execution  so you have to find enforce executions points where you can expect them to be true  Bringing the code execution to such a state is also a hard problem  and might itself incur concurrency issues  Whew  it s damn hard   Some interesting links to read    Deterministic interleaving  A framework that allows to force certain thread interleavings and then check for invariants jMock Blitzer    Stress test synchronization assertConcurrent   JUnit version of stress testing synronization Testing concurrent code   Short overview of the two primary methods of brute force  stress test  or deterministic  going for the invariants

User · Answer

I also had serious problems testing multi- threaded code  Then I found a really cool solution in  xUnit Test Patterns  by Gerard Meszaros  The pattern he describes is called Humble object    Basically it describes how you can extract the logic into a separate  easy-to-test component that is decoupled from its environment  After you tested this logic  you can test the complicated behaviour  multi- threading  asynchronous execution  etc

User · Answer

I have faced this issue several times in recent years when writing thread handling code for several projects   I m providing a late answer because most of the other answers  while providing alternatives  do not actually answer the question about testing   My answer is addressed to the cases where there is no alternative to multithreaded code  I do cover code design issues for completeness  but also discuss unit testing   Writing testable multithreaded code  The first thing to do is to separate your production thread handling code from all the code that does actual data processing   That way  the data processing can be tested as singly threaded code  and the only thing the multithreaded code does is to coordinate threads   The second thing to remember is that bugs in multithreaded code are probabilistic  the bugs that manifest themselves least frequently are the bugs that will sneak through into production  will be difficult to reproduce even in production  and will thus cause the biggest problems   For this reason  the standard coding approach of writing the code quickly and then debugging it until it works is a bad idea for multithreaded code  it will result in code where the easy bugs are fixed and the dangerous bugs are still there   Instead  when writing multithreaded code  you must write the code with the attitude that you are going to avoid writing the bugs in the first place   If you have properly removed the data processing code  the thread handling code should be small enough - preferably a few lines  at worst a few dozen lines - that you have a chance of writing it without writing a bug  and certainly without writing many bugs  if you understand threading  take your time  and are careful   Writing unit tests for multithreaded code  Once the multithreaded code is written as carefully as possible  it is still worthwhile writing tests for that code   The primary purpose of the tests is not so much to test for highly timing dependent race condition bugs - it s impossible to test for such race conditions repeatably - but rather to test that your locking strategy for preventing such bugs allows for multiple threads to interact as intended   To properly test correct locking behavior  a test must start multiple threads   To make the test repeatable  we want the interactions between the threads to happen in a predictable order   We don t want to externally synchronize the threads in the test  because that will mask bugs that could happen in production where the threads are not externally synchronized   That leaves the use of timing delays for thread synchronization  which is the technique that I have used successfully whenever I ve had to write tests of multithreaded code   If the delays are too short  then the test becomes fragile  because minor timing differences - say between different machines on which the tests may be run - may cause the timing to be off and the test to fail   What I ve typically done is start with delays that cause test failures  increase the delays so that the test passes reliably on my development machine  and then double the delays beyond that so the test has a good chance of passing on other machines   This does mean that the test will take a macroscopic amount of time  though in my experience  careful test design can limit that time to no more than a dozen seconds   Since you shouldn t have very many places requiring thread coordination code in your application  that should be acceptable for your test suite   Finally  keep track of the number of bugs caught by your test   If your test has 80  code coverage  it can be expected to catch about 80  of your bugs   If your test is well designed but finds no bugs  there s a reasonable chance that you don t have additional bugs that will only show up in production   If the test catches one or two bugs  you might still get lucky   Beyond that  and you may want to consider a careful review of or even a complete rewrite of your thread handling code  since it is likely that code still contains hidden bugs that will be very difficult to find until the code is in production  and very difficult to fix then

User · Answer

I spent most of last week at a university library studying debugging of concurrent code  The central problem is concurrent code is non-deterministic  Typically  academic debugging has fallen into one of three camps here    Event-trace replay  This requires an event monitor and then reviewing the events that were sent  In a UT framework  this would involve manually sending the events as part of a test  and then doing post-mortem reviews  Scriptable  This is where you interact with the running code with a set of triggers   On x   foo  baz     This could be interpreted into a UT framework where you have a run-time system triggering a given test on a certain condition  Interactive  This obviously won t work in an automatic testing situation       Now  as above commentators have noticed  you can design your concurrent system into a more deterministic state  However  if you don t do that properly  you re just back to designing a sequential system again    My suggestion would be to focus on having a very strict design protocol about what gets threaded and what doesn t get threaded  If you constrain your interface so that there is minimal dependancies between elements  it is much easier   Good luck  and keep working on the problem

User · Answer

Awaitility can also be useful to help you write deterministic unit tests  It allows you to wait until some state somewhere in your system is updated  For example   await   untilCall  to myService  myMethod    greaterThan 3       or  await   atMost 5 SECONDS  until fieldIn myObject  ofType int class   equalTo 1      It also has Scala and Groovy support    await until   something    gt  4      Scala example

User · Answer

For Java  check out chapter 12 of JCIP   There are some concrete examples of writing deterministic  multi-threaded unit tests to at least test the correctness and invariants of concurrent code    Proving  thread-safety with unit tests is much dicier   My belief is that this is better served by automated integration testing on a variety of platforms configurations

User · Answer

Look  there s no easy way to do this   I m working on a project that is inherently multithreaded   Events come in from the operating system and I have to process them concurrently  The simplest way to deal with testing complex  multithreaded application code is this   If it s too complex to test  you re doing it wrong   If you have a single instance that has multiple threads acting upon it  and you can t test situations where these threads step all over each other  then your design needs to be redone   It s both as simple and as complex as this  There are many ways to program for multithreading that avoids threads running through instances at the same time   The simplest is to make all your objects immutable   Of course  that s not usually possible   So you have to identify those places in your design where threads interact with the same instance and reduce the number of those places   By doing this  you isolate a few classes where multithreading actually occurs  reducing the overall complexity of testing your system  But you have to realize that even by doing this  you still can t test every situation where two threads step on each other   To do that  you d have to run two threads concurrently in the same test  then control exactly what lines they are executing at any given moment   The best you can do is simulate this situation   But this might require you to code specifically for testing  and that s at best a half step towards a true solution  Probably the best way to test code for threading issues is through static analysis of the code   If your threaded code doesn t follow a finite set of thread safe patterns  then you might have a problem   I believe Code Analysis in VS does contain some knowledge of threading  but probably not much  Look  as things stand currently  and probably will stand for a good time to come   the best way to test multithreaded apps is to reduce the complexity of threaded code as much as possible   Minimize areas where threads interact  test as best as possible  and use code analysis to identify danger areas

User · Answer

I handle unit tests of threaded components the same way I handle any unit test  that is  with inversion of control and isolation frameworks  I develop in the  Net-arena and  out of the box  the threading  among other things  is very hard  I d say nearly impossible  to fully isolate  Therefore  I ve written wrappers that looks something like this  simplified   public interface IThread       void Start               public class ThreadWrapper   IThread       private readonly Thread  thread            public ThreadWrapper ThreadStart threadStart                 thread   new Thread threadStart              public Start                  thread Start                 public interface IThreadingManager       IThread CreateThread ThreadStart threadStart      public class ThreadingManager   IThreadingManager       public IThread CreateThread ThreadStart threadStart                 return new ThreadWrapper threadStart           From there  I can easily inject the IThreadingManager into my components and use my isolation framework of choice to make the thread behave as I expect during the test  That has so far worked great for me  and I use the same approach for the thread pool  things in System Environment  Sleep etc  etc

User · Answer

Have a look at my related answer at  Designing a Test class for a custom Barrier  It s biased towards Java but has a reasonable summary of the options    In summary though  IMO  its not the use of some fancy framework that will ensure correctness but how you go about designing you multithreaded code  Splitting the concerns  concurrency and functionality  goes a huge way towards raising confidence  Growing Object Orientated Software Guided By Tests explains some options better than I can   Static analysis and formal methods  see  Concurrency  State Models and Java Programs  is an option but I ve found them to be of limited use in commercial development   Don t forget that any load soak style tests are rarely guaranteed to highlight problems   Good luck

User · Answer

Another way to  kinda  test threaded code  and very complex systems in general is through Fuzz Testing  It s not great  and it won t find everything  but its likely to be useful and its simple to do   Quote      Fuzz testing or fuzzing is a software testing technique that provides random data  fuzz   to the inputs of a program  If the program fails  for example  by crashing  or by failing built-in code assertions   the defects can be noted  The great advantage of fuzz testing is that the test design is extremely simple  and free of preconceptions about system behavior                 Fuzz testing is often used in large software development projects that employ black box testing  These projects usually have a budget to develop test tools  and fuzz testing is one of the techniques which offers a high benefit to cost ratio                 However  fuzz testing is not a substitute for exhaustive testing or formal methods  it can only provide a random sample of the system s behavior  and in many cases passing a fuzz test may only demonstrate that a piece of software handles exceptions without crashing  rather than behaving correctly  Thus  fuzz testing can only be regarded as a bug-finding tool rather than an assurance of quality

User · Answer

If you are testing simple new Thread runnable  run   You can mock Thread to run the runnable sequentially  For instance  if the code of the tested object invokes a new thread like this  Class TestedClass       public void doAsychOp            new Thread new myRunnable    start              Then mocking new Threads and run the runnable argument sequentially can help   Mock private Thread threadMock    Test public void myTest   throws Exception       PowerMockito mockStatic Thread class         when new thread is created execute runnable immediately      PowerMockito whenNew Thread class  withAnyArguments   then new Answer lt Thread gt               Override         public Thread answer InvocationOnMock invocation  throws Throwable                  immediately run the runnable             Runnable runnable   invocation getArgumentAt 0  Runnable class               if runnable    null                    runnable run                              return threadMock   return a mock so Thread start   will do nothing                                 TestedClass testcls   new TestedClass       testcls doAsychOp      will invoke myRunnable run in current thread            check expected

User · Answer

if possible  don t use threads  use actors   active objects  Easy to test

User · Answer

Tough one indeed   In my  C    unit tests  I ve broken this down into several categories along the lines of the concurrency pattern used    Unit tests for classes that operate in a single thread and aren t thread aware -- easy  test as usual  Unit tests for Monitor objects  those that execute synchronized methods in the callers  thread of control  that expose a synchronized public API -- instantiate multiple mock threads that exercise the API   Construct scenarios that exercise internal conditions of the passive object   Include one longer running test that basically beats the heck out of it from multiple threads for a long period of time   This is unscientific I know but it does build confidence  Unit tests for Active objects  those that encapsulate their own thread or threads of control  -- similar to  2 above with variations depending on the class design   Public API may be blocking or non-blocking  callers may obtain futures  data may arrive at queues or need to be dequeued   There are many combinations possible here  white box away   Still requires multiple mock threads to make calls to the object under test    As an aside   In internal developer training that I do  I teach the Pillars of Concurrency and these two patterns as the primary framework for thinking about and decomposing concurrency problems   There s obviously more advanced concepts out there but I ve found that this set of basics helps keep engineers out of the soup   It also leads to code that is more unit testable  as described above

User · Answer

The following article suggests 2 solutions  Wrapping a semaphore  CountDownLatch  and adds functionality like  externalize data from internal thread  Another way of achieving this purpose is to use Thread Pool  see Points of Interest    Sprinkler - Advanced synchronization object

User · Answer

There are a few tools around that are quite good  Here is a summary of some of the Java ones   Some good static analysis tools include FindBugs  gives some useful hints   JLint  Java Pathfinder  JPF  amp  JPF2   and Bogor   MultithreadedTC is quite a good dynamic analysis tool  integrated into JUnit  where you have to set up your own test cases   ConTest from IBM Research is interesting  It instruments your code by inserting all kinds of thread modifying behaviours  e g  sleep  amp  yield  to try to uncover bugs randomly   SPIN is a really cool tool for modelling your Java  and other  components  but you need to have some useful framework  It is hard to use as is  but extremely powerful if you know how to use it  Quite a few tools use SPIN underneath the hood   MultithreadedTC is probably the most mainstream  but some of the static analysis tools listed above are definitely worth looking at

User · Answer

I ve done a lot of this  and yes it sucks   Some tips    GroboUtils for running multiple test threads alphaWorks ConTest to instrument classes to cause interleavings to vary between iterations Create a throwable field and check it in tearDown  see Listing 1   If you catch a bad exception in another thread  just assign it to throwable  I created the utils class in Listing 2 and have found it invaluable  especially waitForVerify and waitForCondition  which will greatly increase the performance of your tests  Make good use of AtomicBoolean in your tests   It is thread safe  and you ll often need a final reference type to store values from callback classes and suchlike   See example in Listing 3  Make sure to always give your test a timeout  e g    Test timeout 60 1000    as concurrency tests can sometimes hang forever when they re broken    Listing 1    After public void tearDown         if   throwable    null           throw throwable      Listing 2   import static org junit Assert fail  import java io File  import java lang reflect InvocationHandler  import java lang reflect Proxy  import java util Random  import org apache commons collections Closure  import org apache commons collections Predicate  import org apache commons lang time StopWatch  import org easymock EasyMock  import org easymock classextension internal ClassExtensionHelper  import static org easymock classextension EasyMock     import ca digitalrapids io DRFileUtils          Various utilities for testing     public abstract class DRTestUtils       static private Random random   new Random         Calls   link  waitForCondition Integer  Integer  Predicate  String   with    default max wait and check period values      static public void waitForCondition Predicate predicate  String errorMessage       throws Throwable       waitForCondition null  null  predicate  errorMessage          Blocks until a condition is true  throwing an   link AssertionError  if    it does not become true during a given max time      param maxWait ms max time to wait for true condition  Optional  defaults    to 30   1000 ms  30 seconds       param checkPeriod ms period at which to try the condition  Optional  defaults    to 100 ms      param predicate the condition     param errorMessage message use in the   link AssertionError      throws Throwable on   link AssertionError  or any other exception error     static public void waitForCondition Integer maxWait ms  Integer checkPeriod ms       Predicate predicate  String errorMessage  throws Throwable        waitForCondition maxWait ms  checkPeriod ms  predicate  new Closure             public void execute Object errorMessage                        fail  String errorMessage                    errorMessage          Blocks until a condition is true  running a closure if    it does not become true during a given max time      param maxWait ms max time to wait for true condition  Optional  defaults    to 30   1000 ms  30 seconds       param checkPeriod ms period at which to try the condition  Optional  defaults    to 100 ms      param predicate the condition     param closure closure to run     param argument argument for closure     throws Throwable on   link AssertionError  or any other exception error     static public void waitForCondition Integer maxWait ms  Integer checkPeriod ms       Predicate predicate  Closure closure  Object argument  throws Throwable        if   maxWait ms    null           maxWait ms   30   1000      if   checkPeriod ms    null           checkPeriod ms   100      StopWatch stopWatch   new StopWatch        stopWatch start        while    predicate evaluate null              Thread sleep checkPeriod ms           if   stopWatch getTime    gt  maxWait ms                 closure execute argument                          Calls   link  waitForVerify Integer  Object   with  lt code gt null lt  code gt     for   code maxWait ms      static public void waitForVerify Object easyMockProxy      throws Throwable       waitForVerify null  easyMockProxy          Repeatedly calls   link EasyMock verify Object     until it succeeds  or a    max wait time has elapsed      param maxWait ms Max wait time   lt code gt null lt  code gt  defaults to 30s      param easyMockProxy Proxy to call verify on     throws Throwable     static public void waitForVerify Integer maxWait ms  Object easyMockProxy      throws Throwable       if   maxWait ms    null           maxWait ms   30   1000      StopWatch stopWatch   new StopWatch        stopWatch start        for               try                       verify easyMockProxy               break                    catch  AssertionError e                        if   stopWatch getTime    gt  maxWait ms                   throw e              Thread sleep 100                          Returns a path to a directory in the temp dir with the name of the given    class  This is useful for temporary test files      param aClass test class for which to create dir     return the path     static public String getTestDirPathForTestClass Object object          String filename   object instanceof Class              Class object  getName             object getClass   getName        return DRFileUtils getTempDir     File separator            filename     static public byte   createRandomByteArray int bytesLength        byte   sourceBytes   new byte bytesLength       random nextBytes sourceBytes       return sourceBytes         Returns  lt code gt true lt  code gt  if the given object is an EasyMock mock object      static public boolean isEasyMockMock Object object        try           InvocationHandler invocationHandler   Proxy                  getInvocationHandler object           return invocationHandler getClass   getName   contains  easymock          catch  IllegalArgumentException e            return false              Listing 3    Test public void testSomething         final AtomicBoolean called   new AtomicBoolean false       subject setCallback new SomeCallback             public void callback Object arg                   check arg here             called set true                         subject run        assertTrue called get

User · Answer

You may use EasyMock makeThreadSafe to make testing instance threadsafe

User · Answer

I just recently discovered  for Java  a tool called Threadsafe  It is a static analysis tool much like findbugs but specifically to spot multi-threading issues  It is not a replacement for testing but I can recommend it as part of writing reliable multi-threaded Java   It even catches some very subtle potential issues around things like class subsumption  accessing unsafe objects through concurrent classes and spotting missing volatile modifiers when using the double checked locking paradigm   If you write multithreaded Java give it a shot

User · Answer

There is an article on the topic  using Rust as the language in the example code   https   medium com  polyglot factotum rust-concurrency-five-easy-pieces-871f1c62906a  In summary  the trick is to write your concurrent logic so that it is robust to the non-determinism involved with multiple threads of execution  using tools like channels and condvars    Then  if that is how you ve structured your  components   the easiest way to test them is by using channels to send messages to them  and then block on other channels to assert that the component sends certain expected messages    The linked-to article is fully written using unit-tests

[multithreading] How should I unit test multithreaded code?

Examples related to multithreading

Examples related to unit-testing