How do you generate dynamic parameterized unit tests in Python

Question

I have some kind of test data and want to create a unit test for each item  My first idea was to do it like this  import unittest  l      quot foo quot    quot a quot    quot a quot       quot bar quot    quot a quot    quot b quot      quot lee quot    quot b quot    quot b quot     class TestSequence unittest TestCase       def testsample self           for name  a b in l              print  quot test quot   name             self assertEqual a b   if   name         main         unittest main    The downside of this is that it handles all data in one test  I would like to generate one test for each item on the fly  Any suggestions

User · Answer

I d been having trouble with a very particular style of parameterized tests   All our Selenium tests can run locally  but they also should be able to be run remotely against several platforms on SauceLabs   Basically  I wanted to take a large amount of already-written test cases and parameterize them with the fewest changes to code possible   Furthermore  I needed to be able to pass the parameters into the setUp method  something which I haven t seen any solutions for elsewhere   Here s what I ve come up with   import inspect import types  test platforms           browserName    internet explorer    platform    Windows 7    version    10 0          browserName    internet explorer    platform    Windows 7    version    11 0          browserName    firefox    platform    Linux    version    43 0        def sauce labs        def wrapper cls           return test on platforms cls      return wrapper   def test on platforms base class       for name  function in inspect getmembers base class  inspect isfunction           if name startswith  test                 for platform in test platforms                  new name       join list  name     join platform  browserName   title   split     platform  version                      new function   types FunctionType function   code    function   globals    new name                                                    function   defaults    function   closure                    setattr new function   platform   platform                  setattr base class  new name  new function              delattr base class  name       return base class   With this  all I had to do was add a simple decorator  sauce labs   to each regular old TestCase  and now when running them  they re wrapped up and rewritten  so that all the test methods are parameterized and renamed   LoginTests test login self  runs as LoginTests test login internet explorer 10 0 self   LoginTests test login internet explorer 11 0 self   and LoginTests test login firefox 43 0 self   and each one has the parameter self platform to decide what browser platform to run against  even in LoginTests setUp  which is crucial for my task since that s where the connection to SauceLabs is initialized   Anyway  I hope this might be of help to someone looking to do a similar  global  parameterization of their tests

User · Answer

Use the ddt library  It adds simple decorators for the test methods   import unittest from ddt import ddt  data from mycode import larger than two   ddt class FooTestCase unittest TestCase         data 3  4  12  23      def test larger than two self  value           self assertTrue larger than two value         data 1  -3  2  0      def test not larger than two self  value           self assertFalse larger than two value     This library can be installed with pip  It doesn t require nose  and works excellent with the standard library unittest module

User · Answer

import unittest  def generator test class  a  b c d name       def test self           print  Testexecution   name          print  a   a          print  b   b          print  c   c          print  d   d       return test  def add test methods test class       test list     3 3 5 6   one     5 5 8 9   two     0 0 5 6   three    0 0 2 3  Four        for case in test list          print  case   case 0   case 1  case 2  case 3  case 4           test   generator test class  case 0   case 1  case 2  case 3  case 4           setattr test class   quot test  s quot    case 4   test    class TestAuto unittest TestCase       def setUp self           print   Setup           pass      def tearDown self           print   TearDown           pass  add test methods TestAuto   if   name         main         unittest main verbosity 1

User · Answer

Besides using setattr  we can use load tests with Python  3 2 and later  Please refer to blog post blog livreuro com en coding python how-to-generate-discoverable-unit-tests-in-python-dynamically  class Test unittest TestCase       pass  def  test self  file name       open file name   r   as f          self assertEqual  test result  f read     def  generate test file name       def test self            test self  file name      return test  def  generate tests        for file in files          file name   os path splitext os path basename file   0          setattr Test   test  s    file name   generate test file    test cases    Test    def load tests loader  tests  pattern        generate tests       suite   TestSuite       for test class in test cases          tests   loader loadTestsFromTestCase test class          suite addTests tests      return suite  if   name         main          generate tests       unittest main

User · Answer

It can be done by using pytest  Just write the file test me py with content  import pytest   pytest mark parametrize  name  left  right      foo    a    a                                                     bar    a    b                                                     baz    b    b     def test me name  left  right       assert left    right  name  And run your test with command py test --tb short test me py  Then the output will look like                              test session starts                              platform darwin -- Python 2 7 6 -- py-1 4 23 -- pytest-2 6 1 collected 3 items  test me py  F                                     FAILURES                                                                 test me bar-a-b                                test me py 8  in test me     assert left    right  name E   AssertionError  bar                      1 failed  2 passed in 0 01 seconds                       It is simple  Also pytest has more features like fixtures  mark  assert  etc

User · Answer

This solution works with unittest and nose for Python 2 and Python 3      usr bin env python import unittest  def make function description  a  b       def ghost self           self assertEqual a  b  description      print description      ghost   name      test  0   format description      return ghost   class TestsContainer unittest TestCase       pass  testsmap          foo    1  1        bar    1  2        baz    5  5    def generator        for name  params in testsmap iteritems            test func   make function name  params 0   params 1           setattr TestsContainer   test  0   format name   test func   generator    if   name         main         unittest main

User · Answer

I have found that this works well for my purposes  especially if I need to generate tests that do slightly difference processes on a collection of data   import unittest  def rename newName       def renamingFunc func           func   name      newName         return func     return renamingFunc  class TestGenerator unittest TestCase        TEST DATA            classmethod     def generateTests cls           for dataName  dataValue in TestGenerator TEST DATA              for func in cls getTests dataName  dataValue                   setattr cls   test   s    s   format func   name    dataName   func        classmethod     def getTests cls           raise NotImplementedError  This must be implemented     class TestCluster TestGenerator        TEST CASES            staticmethod     def getTests dataName  dataValue            def makeTest case                 rename    s   format case  name                 def test self                     Do things with self  case  data                 pass              return test          return  makeTest c  for c in TestCluster TEST CASES   TestCluster generateTests     The TestGenerator class can be used to spawn different sets of test cases like TestCluster   TestCluster can be thought of as an implementation of the TestGenerator interface

User · Answer

I use metaclasses and decorators for generate tests  You can check my implementation python wrap cases  This library doesn t require any test frameworks  Your example  import unittest from python wrap cases import wrap case    wrap case class TestSequence unittest TestCase         wrap case  quot foo quot    quot a quot    quot a quot        wrap case  quot bar quot    quot a quot    quot b quot        wrap case  quot lee quot    quot b quot    quot b quot       def testsample self  name  a  b           print  quot test quot   name         self assertEqual a  b   Console output  testsample u bar  u a  u b   tests example test stackoverflow TestSequence      test bar FAIL testsample u foo  u a  u a   tests example test stackoverflow TestSequence      test foo ok testsample u lee  u b  u b   tests example test stackoverflow TestSequence      test lee ok  Also you may use generators  For example this code generate all possible combinations of tests with arguments a  list and b  list import unittest from python wrap cases import wrap case    wrap case class TestSequence unittest TestCase         wrap case a  list   quot a quot    quot b quot    b  list   quot a quot    quot b quot        def testsample self  a  b           self assertEqual a  b   Console output  testsample a u a   b u a    tests example test stackoverflow TestSequence      ok testsample a u a   b u b    tests example test stackoverflow TestSequence      FAIL testsample a u b   b u a    tests example test stackoverflow TestSequence      FAIL testsample a u b   b u b    tests example test stackoverflow TestSequence      ok

User · Answer

You can use TestSuite and custom TestCase classes    import unittest  class CustomTest unittest TestCase       def   init   self  name  a  b           super     init             self name   name         self a   a         self b   b      def runTest self           print  test   self name          self assertEqual self a  self b   if   name         main         suite   unittest TestSuite       suite addTest CustomTest  Foo   1337  1337       suite addTest CustomTest  Bar   0xDEAD  0xC0DE       unittest TextTestRunner   run suite

User · Answer

Following is my solution  I find this useful when   Should work for unittest Testcase and unittest discover  Have a set of tests to be run for different parameter settings   Very simple and no dependency on other packages  import unittest   class BaseClass unittest TestCase        def setUp self            self param   2          self base   2       def test me self            self assertGreaterEqual 5  self param self base        def test me too self            self assertLessEqual 3  self param self base      class Child One BaseClass        def setUp self            BaseClass setUp self           self param   4     class Child Two BaseClass        def setUp self            BaseClass setUp self           self param   1

User · Answer

load tests is a little known mechanism introduced in 2 7 to dynamically create a TestSuite  With it  you can easily create parametrized tests   For example   import unittest  class GeneralTestCase unittest TestCase       def   init   self  methodName  param1 None  param2 None           super GeneralTestCase  self    init   methodName           self param1   param1         self param2   param2      def runTest self           pass    Test that depends on param 1 and 2    def load tests loader  tests  pattern       test cases   unittest TestSuite       for p1  p2 in   1  2    3  4            test cases addTest GeneralTestCase  runTest   p1  p2       return test cases   That code will run all the TestCases in the TestSuite returned by load tests  No other tests are automatically run by the discovery mechanism   Alternatively  you can also use inheritance as shown in this ticket  http   bugs python org msg151444

User · Answer

This is effectively the same as parameterized as mentioned in a previous answer  but specific to unittest  def sub test param list        quot  quot  quot Decorates a test case to run it as a set of subtests  quot  quot  quot       def decorator f             functools wraps f          def wrapped self               for param in param list                  with self subTest   param                       f self    param           return wrapped      return decorator  Example usage  class TestStuff unittest TestCase        sub test           dict arg1  a   arg2  b            dict arg1  x   arg2  y               def test stuff self  a  b

User · Answer

The metaclass-based answers still work in Python  3  but instead of the   metaclass   attribute  one has to use the metaclass parameter  as in  class ExampleTestCase TestCase metaclass DocTestMeta       pass

User · Answer

As of Python 3 4  subtests have been introduced to unittest for this purpose  See the documentation for details  TestCase subTest is a context manager which allows one to isolate asserts in a test so that a failure will be reported with parameter information  but it does not stop the test execution  Here s the example from the documentation  class NumbersTest unittest TestCase    def test even self        quot  quot  quot      Test that numbers between 0 and 5 are all even       quot  quot  quot      for i in range 0  6           with self subTest i i               self assertEqual i   2  0   The output of a test run would be                                                                         FAIL  test even    main   NumbersTest   i 1  ---------------------------------------------------------------------- Traceback  most recent call last     File  quot subtests py quot   line 32  in test even     self assertEqual i   2  0  AssertionError  1    0                                                                         FAIL  test even    main   NumbersTest   i 3  ---------------------------------------------------------------------- Traceback  most recent call last     File  quot subtests py quot   line 32  in test even     self assertEqual i   2  0  AssertionError  1    0                                                                         FAIL  test even    main   NumbersTest   i 5  ---------------------------------------------------------------------- Traceback  most recent call last     File  quot subtests py quot   line 32  in test even     self assertEqual i   2  0  AssertionError  1    0  This is also part of unittest2  so it is available for earlier versions of Python

User · Answer

You would benefit from trying the TestScenarios library      testscenarios provides clean dependency injection for python unittest style tests  This can be used for interface testing  testing many implementations via a single test suite  or for classic dependency injection  provide tests with dependencies externally to the test code itself  allowing easy testing in different situations

User · Answer

I had trouble making these work for setUpClass  Here s a version of Javier s answer that gives setUpClass access to dynamically allocated attributes  import unittest   class GeneralTestCase unittest TestCase        classmethod     def setUpClass cls           print            print cls p1         print cls p2      def runTest1 self           self assertTrue  self p2 - self p1     1       def runTest2 self           self assertFalse  self p2 - self p1     2    def load tests loader  tests  pattern       test cases   unittest TestSuite       for p1  p2 in   1  2    3  4            clsname    TestCase        format p1  p2          dct                  p1   p1               p2   p2                    cls   type clsname   GeneralTestCase    dct          test cases addTest cls  runTest1            test cases addTest cls  runTest2        return test cases  Outputs 1 2    3 4    ---------------------------------------------------------------------- Ran 4 tests in 0 000s  OK

User · Answer

import unittest  def generator test class  a  b       def test self           self assertEqual a  b      return test  def add test methods test class         The first element of list is variable  quot a quot   then variable  quot b quot   then name of test case that will be used as suffix      test list     2 3   one     5 5   two     0 0   three        for case in test list          test   generator test class  case 0   case 1           setattr test class   quot test  s quot    case 2   test    class TestAuto unittest TestCase       def setUp self           print  Setup          pass      def tearDown self           print  TearDown          pass   add test methods TestAuto     It s better to start with underscore so it is not detected as a test itself  if   name         main         unittest main verbosity 1   RESULT   gt  gt  gt  Setup FTearDown Setup TearDown  Setup TearDown                                                                          FAIL  test one    main   TestAuto  ---------------------------------------------------------------------- Traceback  most recent call last     File  quot D  inchowar Desktop PyTrash test auto 3 py quot   line 5  in test     self assertEqual a  b  AssertionError  2    3  ---------------------------------------------------------------------- Ran 3 tests in 0 019s  FAILED  failures 1

User · Answer

This can be solved elegantly using Metaclasses   import unittest  l      foo    a    a       bar    a    b      lee    b    b     class TestSequenceMeta type       def   new   mcs  name  bases  dict            def gen test a  b               def test self                   self assertEqual a  b              return test          for tname  a  b in l              test name    test  s    tname             dict test name    gen test a b          return type   new   mcs  name  bases  dict   class TestSequence unittest TestCase         metaclass     TestSequenceMeta  if   name         main         unittest main

User · Answer

Just use metaclasses  as seen here   class DocTestMeta type               Test functions are generated in metaclass due to the way some     test loaders work  For example  setupClass   won t get called     unless there are other existing test methods  and will also     prevent unit test loader logic being called before the test     methods have been defined              def   init   self  name  bases  attrs           super DocTestMeta  self    init   name  bases  attrs       def   new   cls  name  bases  attrs           def func self                  Inner test method goes here                self assertTrue 1           func   name      test sample          attrs func   name      func         return super DocTestMeta  cls    new   cls  name  bases  attrs   class ExampleTestCase TestCase          Our example test case  with no methods defined          metaclass     DocTestMeta   Output   test sample  ExampleTestCase      OK

User · Answer

Using unittest  since 3 4  Since Python 3 4  the standard library unittest package has the subTest context manager  See the documentation   26 4 7  Distinguishing test iterations using subtests subTest  Example  from unittest import TestCase  param list      a    a      a    b      b    b     class TestDemonstrateSubtest TestCase       def test works as expected self           for p1  p2 in param list              with self subTest                    self assertEqual p1  p2   You can also specify a custom message and parameter values to subTest    with self subTest msg  quot Checking if p1 equals p2 quot   p1 p1  p2 p2    Using nose The nose testing framework supports this  Example  the code below is the entire contents of the file containing the test   param list      a    a      a    b      b    b     def test generator        for params in param list          yield check em  params 0   params 1   def check em a  b       assert a    b  The output of the nosetests command   gt  nosetests -v testgen test generator  a    a       ok testgen test generator  a    b       FAIL testgen test generator  b    b       ok                                                                         FAIL  testgen test generator  a    b   ---------------------------------------------------------------------- Traceback  most recent call last     File  quot  usr lib python2 5 site-packages nose-0 10 1-py2 5 egg nose case py quot   line 203  in runTest     self test  self arg    File  quot testgen py quot   line 7  in check em     assert a    b AssertionError  ---------------------------------------------------------------------- Ran 3 tests in 0 006s  FAILED  failures 1

User · Answer

There s also Hypothesis which adds fuzz or property based testing  This is a very powerful testing method

User · Answer

This is called  quot parametrization quot   There are several tools that support this approach  E g    pytest s decorator parameterized  The resulting code looks like this  from parameterized import parameterized  class TestSequence unittest TestCase        parameterized expand             quot foo quot    quot a quot    quot a quot               quot bar quot    quot a quot    quot b quot              quot lee quot    quot b quot    quot b quot               def test sequence self  name  a  b           self assertEqual a b   Which will generate the tests  test sequence 0 foo    main   TestSequence      ok test sequence 1 bar    main   TestSequence      FAIL test sequence 2 lee    main   TestSequence      ok                                                                         FAIL  test sequence 1 bar    main   TestSequence  ---------------------------------------------------------------------- Traceback  most recent call last     File  quot  usr local lib python2 7 site-packages parameterized parameterized py quot   line 233  in  lt lambda gt      standalone func   lambda  a  func   a   p args     p kwargs    File  quot x py quot   line 12  in test sequence     self assertEqual a b  AssertionError   a      b   For historical reasons I ll leave the original answer circa 2008   I use something like this  import unittest  l      quot foo quot    quot a quot    quot a quot       quot bar quot    quot a quot    quot b quot      quot lee quot    quot b quot    quot b quot     class TestSequense unittest TestCase       pass  def test generator a  b       def test self           self assertEqual a b      return test  if   name         main         for t in l          test name    test  s    t 0          test   test generator t 1   t 2           setattr TestSequense  test name  test      unittest main

User · Answer

Meta-programming is fun  but it can get in the way  Most solutions here make it difficult to   selectively launch a test point back to the code given the test s name  So  my first suggestion is to follow the simple explicit path  works with any test runner   import unittest  class TestSequence unittest TestCase        def  test complex property self  a  b           self assertEqual a b       def test foo self           self  test complex property  quot a quot    quot a quot       def test bar self           self  test complex property  quot a quot    quot b quot       def test lee self           self  test complex property  quot b quot    quot b quot    if   name         main         unittest main    Since we shouldn t repeat ourselves  my second suggestion builds on Javier s answer  embrace property based testing  Hypothesis library   is  quot more relentlessly devious about test case generation than us mere humans quot   will provide simple count-examples  works with any test runner  has many more interesting features  statistics  additional test output       class TestSequence unittest TestCase      given st text    st text      def test complex property self  a  b         self assertEqual a b     To test your specific examples  just add       example  quot a quot    quot a quot        example  quot a quot    quot b quot        example  quot b quot    quot b quot    To run only one particular example  you can comment out the other examples  provided example will be run first   You may want to use  given st nothing     Another option is to replace the whole block by       given st just  quot a quot    st just  quot b quot     OK  you don t have distinct test names  But maybe you just need   a descriptive name of the property under test  which input leads to failure  falsifying example    Funnier example

User · Answer

I came across ParamUnittest the other day when looking at the source code for radon  example usage on the GitHub repository   It should work with other frameworks that extend TestCase  like Nose   Here is an example  import unittest import paramunittest    paramunittest parametrized        1    2          4  3       lt ---- Uncomment to have a failing test       2    3           4        b    5                a   5   b   6          a   5   b   6     class TestBar TestCase       def setParameters self  a  b           self a   a         self b   b      def testLess self           self assertLess self a  self b

User · Answer

You can use the nose-ittr plugin  pip install nose-ittr   It s very easy to integrate with existing tests  and minimal changes  if any  are required  It also supports the nose multiprocessing plugin  Note that you can also have a customize setup function per test   ittr number  1  2  3  4   def test even self       assert equal self number   2  0   It is also possible to pass nosetest parameters like with their built-in plugin attrib  This way you can run only a specific test with specific parameter  nosetest -a number 2

[python] How do you generate dynamic (parameterized) unit tests in Python?

Examples related to python

Examples related to unit-testing

Examples related to parameterized-unit-test