pytest-matrix 0.7.7

Description:

pytestmatrix 0.7.7

Easy fixture combinations.

Instalition
pip install pytest_matrix

Quickstart:
import pytest
from myproject import my_func

@pytest.mark.matrix(names=['arg_firs'],
combs=[
{
'arg_first': ['val_1', 'val_2'],
},
])
def test_my_fn(self, arg_first, arg_second, result):
assert my_function(arg_first, arg_second) == result


@pytest.fixture
def arg_first_val_1(self):
return 'val_1'

@pytest.fixture
def arg_first_val_2(self):
return 'val_2'

@pytest.fixture
def arg_second(self):
return 'val'

@pytest.fixture
def result(self, arg_first, arg_second):
# prepare expected result base on arg_first and arg_second fixture
# in each test the arg_first parameter will have different value
return ...
OR
import pytest
from pytest_matrix import TestMatrixMixin
from myproject import my_func


class MyTestCase(TestMatrixMixin):
def test_my_fn(self, arg_first, arg_second, result):
assert my_function(arg_first, arg_second) == result

MY_FN_FIXTURES_NAMES = ['arg_first']
MY_FN_FIXTURES = [
{
'arg_first': ['val_1', 'val_2'],
}
]

@pytest.fixture
def arg_first_val_1(self):
return 'val_1'

@pytest.fixture
def arg_first_val_2(self):
return 'val_2'

@pytest.fixture
def arg_second(self):
return 'val'

@pytest.fixture
def result(self, arg_first, arg_second):
# prepare expected result base on arg_first and arg_second fixture
# in each test the arg_first parameter will have different value
return ...



This will generate tests:

MyTestCase::test_my_fn[arg_first_val_1|arg_second|result]
MyTestCase::test_my_fn[arg_first_val_2|arg_second|result]

Both examples are equal. But in class you have better scope controlling and other options.
Please notice result fixture and arg_first fixture. There is no arg_first fixture definition, the arg_first is created by py.test during test generation and you get access to current value of arg_first, same as test function receive.

Test Data
Every test function must be prefixed with ‘test_’
For every test function must be defined two class attributes. If test function is named ‘test_**my_function**’,
there must be defined MY_FUNCTION_FIXTURES_NAMES and MY_FUNCTION_FIXTURES lists.
You must define them in every class (they are not inherited).



MY_FUNCTION_FIXTURES_NAMES:

they are not required, it just could be little bit clearer some times, because you can choose order (the way how test name will be generated)
list of names of fixtures to be combined in test
you can define fixtures, which ARE NOT defined in test as parameter, these fixtures will be
stored in request.param and also it will be accessible by other fixtures



MY_FUNCTION_FIXTURES:

list of fixture combinations

each combination is dict

keys are same as in MY_FUNCTION_FIXTURES_NAMES

values are list of fixture name

fixture name is combination of parameter name and the list item











Fixtures definitions:

For every item in MY_FUNCTION_FIXTURES must exists fixture. It does not have to be in same class.
Fixtures names are defined in MY_FUNCTION_FIXTURES. The name si combination of key and each item in list.

MY_FN_FIXTURES = [
{
'par': ['a', 'b'],
}
]
# will search for fixtures **par_a** and **par_b**
WARNING:
Be aware that every test has his own fixture context. This is useful when you want to access current value
of function parameter by fixture name, but can be easily overlooked.
Example:
class MyTestCase(TestMatrixMixin):
def test_my_fn(self, par, result):
# some test

MY_FN_FIXTURES_NAMES = ['par']
MY_FN_FIXTURES = [
{
'par': ['a', 'b'],
}
]

@pytest.fixture
def par_a(self):
return 'val_a'

@pytest.fixture
def par_b(self):
return 'val_b'

@pytest.fixture
def par(self):
# THIS WILL NEVER BE USED IN GENERATED TESTS
# the context of the generated test inject in every test to par fixture either par_a or par_b

@pytest.fixture
def result(self, par):
# par is either value of par_a or par_b, it depends on test


Simple Fixtures
There are two simple fixtures types: String (with prefix ‘@’) and Integer (with prefix ‘#’)
import pytest
from myproject import my_func

@pytest.mark.matrix(names=['arg_firs'],
combs=[
{
'arg_first': ['#1', '@2'],
},
])
def test_my_fn(self, arg_first):
assert arg_first in (1, '2')
There is no need to define fixtures arg_first_#1 (returning int(1)), arg_first_@2 (returning str(2)). It is impossible have functions (fixture definition) with these names in python anyway :).


Test Generator
The test are generated for cartesian product of defined fixture_names.
class MyTestCase(TestMatrixMixin):
def test_my_fn(self, s, b):
# some test

MY_FN_FIXTURES_NAMES = ['a', 'b']
MY_FN_FIXTURES = [
{
'a': ['x', 'y'],
'b': ['i', 'j'],
},
{
'a': ['x', 'y'],
'b': ['k', 'l'],
}
]


this will generate tests:

test_my_fn[a_x|b_i]
test_my_fn[a_x|b_j]
test_my_fn[a_y|b_i]
test_my_fn[a_y|b_j]
test_my_fn[a_x|b_k]
test_my_fn[a_x|b_l]
test_my_fn[a_y|b_k]
test_my_fn[a_y|b_l]


MIXIN and inheritance



IS_MIXIN
You can define tests in separate class and reuse them in multiple other class. You usually don’t want to collect these tests and run them. So you can add class attribute IS_MIXIN = True and tests in this class
will not be collected by pytest.
If you use some of these mixins you have to define _FIXTURES for each test. It could happen, that you won’t use some of the tests, or you do not want generate from some of the tests.


SKIP_TEST
You can skip tests by writing the test name in SKIP_TESTS class attribute.


NOT_GENERATE_TESTS
Write name of test you don’t want to generate ot NOT_GENEREATE_TESTS attribute. Difference between NOT_GENERATE_TESTS and SKIP_TESTS is that NOT_GENERATE_TESTS will be actually run, but they will not be paramatrize.
Attributes IS_MIXIN, SKIP_TESTS and NOT_GENERATE_TESTS are not inherited from parent class.
Example:
class MyTestMixin(TestMatrixMixin):
IS_MIXIN = True

def test_a(self):
pass

def test_b(self):
pass


class RealTest(MyTestMixin):

SKIP_TESTS = ['test_a']
NOT_GENERATE_TESTS = ['test_b']


class DeeperInheritanceTest(RealTest):
SKIP_TESTS = ['test_b']

A_FIXTURES_NAMES = ['par']
A_FIXTURES = [
{
'par': ['a', 'b'],
}
]

@pytest.fixture
def par_a(self):
return 'val_a'

@pytest.fixture
def par_b(self):
return 'val_b'


This will skip:

RealTest.test_a
DeeperInheritanceTest.test_b



And run these tests:

RealTest.test_b
DeeperInheritanceTest.test_a[par_a]
DeeperInheritanceTest.test_a[par_b]


Combination Tester
Sometimes you want test if you covered all combinations of specific fixtures. You can define the combinations you want to cover in class attribute COMBINATIONS_COVER.



test_combcover_fn_fx_x_y PASSED
class TestCombinations(TestMatrixMixin):
FN_FIXTURES = [
{
'x': ['a', 'b'],
'y': ['c'],
},
{
'x': ['a'],
'y': ['d'],
}
]
FN_FIXTURES_NAMES = ['x', 'y']

FX_FIXTURES = [
{
'x': ['b'],
'y': ['d'],
'z': ['j', 'k']
}
]
FX_FIXTURES_NAMES = ['x', 'y', 'z']

# **COMBINATIONS**
COMBINATIONS_COVER = [
{
"fixture_names": ['x', 'y'],
"fixture_functions": ['fn', 'fx'],
}
]

def test_fx(self):
pass

def test_fn(self):
pass

@pytest.fixture
def x_a(self):
pass

#... rest of class with rest of fixtures (x_b, y_c, y_d, z_j, z_k)
This will generate test test_combcover_fn_fx_x_y. The prefix for combination cover test is test_combcover_ followed by names of functions (test_fx and test_fn) separated by underscore: fn_fx_ and suffix are names of fixtures (their combinations we want to cover) x_y.
This concrete test will find all types of x (‘a’, ‘b’) and y (‘c’, ‘d’) fixtures, combine them ([x_a|y_c], [x_b|y_c], [x_a|y_d], [x_b|y_d]) and compare them with combinations manually defined in _FIXTURES configuration (fn: [x_a|y_c], [x_b|y_c], [x_a|y_d] and fx: [x_b|y_d]). If they are not equal, the test will fail and print all uncovered combinations. But this test will pass.


test_combcover_fn_x_y FAILED
Now we added other test combination.
class TestCombinations(TestMatrixMixin):
FN_FIXTURES = [
{
'x': ['a', 'b'],
'y': ['c'],
},
{
'x': ['a'],
'y': ['d'],
}
]

# other configs

COMBINATIONS_COVER = [
{
"fixture_names": ['x', 'y'],
"fixture_functions": ['fn', 'fx'],
},
{
"fixture_names": ['x', 'y'],
"fixture_functions": ['fn'], # **TEST ONLY ONE TEST'S FIXTURE COMBINATIONS**
},
]

# rest of the class...
This will generate two tests test_combcover_fn_fx_x_y PASSED and test_combcover_fn_x_y FAILED. The second test failed because combination of [x_b|y_d] is not covered in FN_FIXTURES. It will be also shown in test_result.


test_combcover_fx_x_y FAILED OR PASSED according to scope

There are two type of scopes which combcover can use when looking for all types of fixtures.


class scope:

default scope
the combcover will look in ALL _FIXTURES defined in same class





functions scope:

the combcover will look for fixture types only in these _FIXTURES from functions define in combcover config







class TestCombinations(TestMatrixMixin):
FN_FIXTURES = [
{
'x': ['a', 'b'],
'y': ['c'],
},
{
'x': ['a'],
'y': ['d'],
}
]
FN_FIXTURES_NAMES = ['x', 'y']

FX_FIXTURES = [
{
'x': ['b'],
'y': ['d'],
'z': ['j', 'k']
}
]
FX_FIXTURES_NAMES = ['x', 'y', 'z']

# **COMBINATIONS**
COMBINATIONS_COVER = [
{
"fixture_names": ['x', 'y'],
"fixture_functions": ['fx'],
"scope": 'class', # this is not required *class* is default scope
}
]
# rest of the class...
The test will find all types of x (‘a’, ‘b’) and y (‘c’, ‘d’) in ALL fixtures, combine them ([x_a|y_c], [x_b|y_c], [x_a|y_d], [x_b|y_d]) and compare them with combinations manually defined in FX_FIXTURES configuration ([x_b|y_d]). The result of the test will be FAILED and missing combinations will be: [x_a|y_c], [x_b|y_c], [x_a|y_d]
If you remove the scope key from COMBINATIONS_COVER the test will be PASSED, because combcover will be looking for only for fixtures type defined in FX_FIXTURES (x_a and y_d).
class TestCombinations(TestMatrixMixin):
FN_FIXTURES = [
{
'x': ['a', 'b'],
'y': ['c'],
},
{
'x': ['a'],
'y': ['d'],
}
]
FN_FIXTURES_NAMES = ['x', 'y']

FX_FIXTURES = [
{
'x': ['b'],
'y': ['d'],
'z': ['j', 'k']
}
]
FX_FIXTURES_NAMES = ['x', 'y', 'z']

# **COMBINATIONS**
COMBINATIONS_COVER = [
{
"fixture_names": ['x', 'y'],
"fixture_functions": ['fx'],
"scope": 'functions', # this is required
}
]
# rest of the class...
This combocover test will PASS

TODO:
[X] exclude test if test’s cls TestMatrixMixin.is_mixin == True
[X] force to define _FIXTURES and _FIXTURES_NAMES in every class, except mixin class
[X] raise error if _FIXTURES keys are not exactly same as _FIXTURE_NAMES
[ ] edit function to control use of all fixtures combinations
[X] check names of fixtures combinations are same as defined FIXTURES_NAMES
[X] allow skip tests
[X] allow not generate tests
[ ] validate sctructure of SKIP_TESTS, NOT_GENERATE_TESTS, FIXTURE_NAMES and FIXTURES
[ ] check for duplicity in _FIXTURES and COMBINATION_COVER