How to speedup python unittest on muticore machines? [duplicate]

This question already has answers here: Can Python's unittest test in parallel, like nose can? (7 answers) Closed 6 years ago.

I'm using python unittest in order to test some other external application but it takes too much time to run the test one by one.

I would like to know how can I speedup this process by using the power of multi-cores. Can I tweak unittest to execute tests in parallel? How?

This question is not able python GIL limitation because in fact not the python code takes time but the external application that I execute, currently 开发者_Python百科via os.system().

---

If your tests are not too involved, you may be able to run them using py.test which has support for distributed testing. If you are not running on Windows, then nose might also work for you.

---

The testtools package is an extension of unittest which supports running tests concurrently. It can be used with your old test classes that inherit unittest.TestCase.

For example:

import unittest
import testtools

class MyTester(unittest.TestCase):
    # Tests...

suite = unittest.TestLoader().loadTestsFromTestCase(MyTester)
concurrent_suite = testtools.ConcurrentStreamTestSuite(lambda: ((case, None) for case in suite))
concurrent_suite.run(testtools.StreamResult())

---

Maybe you can run each test on a different process using the multiprocessing library. This implies that each unit test (or group of unit tests) should be independent and doesn't need to share the state. It will open other processes, and will make use of other cores.

Check specifically the 'Using a pool of workers' on this page ( http://docs.python.org/library/multiprocessing.html#using-a-pool-of-workers)

EDIT: This module is included since version 2.6

---

As the @vinay-sajip suggested, a few non-core python packages like py.test and nose provided parallel execution of unit tests via multiprocessing lib right out of the box.

However, one thing to consider is that if you are testing a web app with database backend and majority of your test cases are relying on connecting to the same test database, then your unit test execution speed is bottlenecked on the DB not I/O per se. And using multiprocess won't speed it up.

Given that each unit test case requires an independent setup of the database schema + data, you cannot scale out the execution speed only on CPU but restricted with a single test database connection to a single test database server (otherwise the state of the data may interfere with other other while parallel executing each test case so on and so forth).