Testing Python – Part 1

Definitions

TDD: Test Driven Development

BDD: Behavior Driven Development

smoke test: show that different components of an application are working together

unit test: test most basic feature

integration test: test a larger bit of code (ex: several classes)

system test: check the all system. Something as close to the end-user environment as possible.

functional test: check a single bit of functionality

How to structure the test folder

This one should be roughly the structure of the main code such as shown here.

PyTest

to run PyTest, simply type py.test (or pytest) at the top level of your project

$ py.test

PyTest will run all the files with names ending in _test.py.

In order to only see the various tests (failing, skipped…etc), use the flag -r as shown here

$ py.test -r f
  • f: failed – test did not run successfully
  • E: error – exception happened outside of the test function
  • s: skipped – test was skipped (see decorators @pytest.mark.skip() and @pytest.mark.skipif())
  • x: failed – test was not supposed to pass, ran and failed (good) (see decorators @pytest.mark.xfail())
  • X: passed – test was not supposed to pass, ran and passed (not good)

Verbose Mode

$ py.test -v

With verbose, each row has 1 test

Without verbose, each row has 1 file

-k flag

$ pytest -k "test_yes_is"

This will look for a test with the test_yes_is string and then run it

Coverage

$ pip install pytest-cov
$ py.test --cov. app/  test/

app/ is the application folder

test/ is the test folder

It’s possible to customized the coverage, for example listing files that do not need to be covered or checked for coverage

.coveragec

[run]
omit = *__init__.py

Configuration file

3 configuration files are possible

  • setup.cfg
  • pytest.ini
  • tox.ini

Using Options

PyTest comes with a lot of options ($ pytest –help)

-k ExpressionOnly run tests/classes which match substring expression
-m MARKEXPRrun tests matching mark expression
–lf, –last-failedrerun only the tests that failed last time
–ff, –failed-firstrun all but rerun failed first
-v, –verboseverbose
-q, –quietdecrease verbosity
–collect-onlycollects tests but do not run them
–maxfail=numstop after num failures
-x, –exitfirststop after first test failing
–tb=styleauto
long
short
line
native
no

Examples

-k

$ pytest -k "asdict or defaults" --collect-only
================ test session starts ==================
collected 6 items
<Module 'tasks/test_four.py'>
  <Function 'test_asdict'>
<Module 'tasks/test_three.py'>
  <Function 'test_defaults'>

-m

import pytest
...
@pytest.mark_run_these_please
def test_member_access():
   ...
$ pytest -v -m run_these_please
================ test session starts ==================
collected 4 items
test_four.py::test_replace PASSED
test_three.py::test_member_access PASSED
============= 2 tests deselected =======================
======= 2 passed, 2 deselected in 0.02 seconds ==========