OpenStack

OpenStack logo

Gerrit

Hack OpenStack

Tests

Tools

See also

nose usage and pbr (github).

testr: isolate bug

https://rbtcollins.wordpress.com/2015/12/02/diagnosing-flaky-tests/

Load a subunit file downloaded from OpenStack gates:

(py27) haypo@smithers$ wget http://logs.openstack.org/45/275645/1/check/gate-glance-python27/e3fffca/testrepository.subunit.gz

(py27) haypo@smithers$ gunzip testrepository.subunit.gz (py27) haypo@smithers$ testr load testrepository.subunit

testr

Run tests:

  • run –no-parallel: run all tests in a single process
  • testr run –until-failure: run forever, until a test fails

Analyze latest run:

  • testr last –subunit|subunit2ls: list tests of the previous run
  • testr run –analyze-isolation: try to isolate the failing test, find the minimum tests to reproduce the fail

subunit tools:

  • subunit-filter
  • subunit-ls

Bisection: diagnosing flaky tests.

notes

Run unit tests:

. .tox/py27/bin/activate
testr run

Shell commands to run unit tests:

set -e && \
        TEMP_REZ=`mktemp -t` && \
        python setup.py testr --slowest --testr-args='--subunit  ' \
                | tee $$TEMP_REZ | subunit2pyunit || true ; \
        cat $$TEMP_REZ | subunit-filter -s --no-passthrough | subunit-stats ; \
        rm -f $$TEMP_REZ ;
  • --slowest shows the statistics at the end of the test run. Nothing fancy.
  • --testr-args='--subunit tells testr to output a subunit2 format for its unit tests. subunit2 format is a BINARY format, which you shouldn’t output to the screen.
  • subunit2pyunit will convert that to a nicer output
  • tee $$TEMP_REZ .. cat $$TEMP_REZ | subunit-filter -s --no-passthrough | subunit-stats shows the nice statistics about the test run (eg: how many tests in total, how many skips, how many failed, how many success)

testr: list skipped tests

testr last --subunit|subunit-filter -s|subunit-ls >A
testr last --subunit|subunit-filter -s --no-skip|subunit-ls >B
diff -u A B

tox/testr: “db type could not be determined” error

testr uses a database to store test results. If the database is created by Python 2, Python 3 cannot read it and then you get the error “db type could not be determined”.

Workaround: remove .testrepository directory and rerun tox again.

tox/testr: “gdbm is missing”

If you run tox -e py34 and then tox -e py27, the second commands may fail because Python 2.7 does not have the gdbm module.

On Ubuntu, type:

sudo apt-get install -y python-gdbm

testr: “local variable ‘run_subunit_content’ referenced before assignment” error

See Error message opaque when .testrepository files are unreadable.

Re-run a single failing test

testtools

Re-run a single test with testtools:

$ tox -e py33
...
FAIL: tests.test_swiftclient.TestPutObject.test_unicode_ok
...
$ . .tox/py33/bin/activate
$ python -m testtools.run tests.test_swiftclient.TestPutObject.test_unicode_ok
 Tests running...
 ======================================================================
 FAIL: tests.test_swiftclient.TestPutObject.test_unicode_ok
 ----------------------------------------------------------------------
 ...
 Ran 1 test in 0.002s

tox

Re-run a single test with tox+testr:

$ tox -e py33
...
FAIL: tests.test_swiftclient.TestPutObject.test_unicode_ok
...
$ tox -e py33 -- --isolated tests.test_swiftclient.TestPutObject.test_unicode_ok
...
FAIL: tests.test_swiftclient.TestPutObject.test_unicode_ok
...

Note

Enter the virtualenv and type testr run tests.test_swiftclient.TestPutObject.test_unicode_ok should work, but it doesn’t in the Python 3.3 virtual environment of python-sphinxclient?!

nose

Re-run a single test with nose:

$ nosetests
...
======================================================================
FAIL: tests.test_command_helpers.TestStatHelpers.test_stat_account_human
----------------------------------------------------------------------
...

$ nosetests tests.test_command_helpers:TestStatHelpers.test_stat_account_human