ptr 22.7.12

Description:

ptr 22.7.12

🏃‍♀️ ptr - Python Test Runner 🏃‍♂️




Python Test Runner (ptr) was born to run tests in an opinionated way, within arbitrary code repositories.
ptr supports many Python projects with unit tests defined in their setup.(cfg|py) files per repository.
ptr allows developers to test multiple projects/modules in one Python environment through the use of a single test virtual environment.

ptr requires >= python 3.7
ptr itself uses ptr to run its tests 👌🏼
ptr is supported and tested on Linux, MacOS + Windows Operating Systems

By adding ptr configuration to your either of your pyproject.toml, setup.cfg or setup.py you can have ptr perform the following,
per test suite, in parallel:

run your test suite
check and enforce coverage requirements (via coverage),
format code (via black)
perform static type analysis (via mypy)

Quickstart

Install ptr into you virtualenv

pip install ptr


Ensure your tests have a base file that can be executed directly

i.e. python3 test.py (possibly using unittest.main())


After adding ptr_params to setup.py (see example below), run:

cd repo
ptr

How does ptr perform this magic? 🎩
I'm glad you ask. Under the covers ptr performs:

Recursively searches for setup.(cfg|py) files from BASE_DIR (defaults to your "current working directory" (CWD))

AST parses out the config for each setup.py test requirements
If a pyproject.toml or setup.cfg exists, load via configparser/tomli and prefer if a [ptr] section exists


Creates a Python Virtual Environment (OPTIONALLY pointed at an internal PyPI mirror)
Runs ATONCE tests suites in parallel (i.e. per setup.(cfg|ptr))
All steps will be run for each suite and ONLY FAILED runs will have output written to stdout

Usage 🤓
To use ptr all you need to do is cd to your project or set the base dir via -b and execute:
$ ptr [-dk] [-b some/path] [--venv /tmp/existing_venv]

For faster runs when testing, it is recommended to reuse a Virtual Environment:

-k - To keep the virtualenv created by ptr.
Use --venv VENV_PATH to reuse to an existing virtualenv created by the user.

Help Output 🙋‍♀️ 🙋‍♂️
usage: ptr.py [-h] [-a ATONCE] [-b BASE_DIR] [-d] [-e] [-k] [-m MIRROR]
[--print-cov] [--print-non-configured]
[--progress-interval PROGRESS_INTERVAL] [--run-disabled]
[--stats-file STATS_FILE] [--system-site-packages] [--venv VENV]
[--venv-timeout VENV_TIMEOUT]

optional arguments:
-h, --help show this help message and exit
-a ATONCE, --atonce ATONCE
How many tests to run at once [Default: 6]
-b BASE_DIR, --base-dir BASE_DIR
Path to recursively look for setup.py files [Default:
/Users/cooper/repos/ptr]
-d, --debug Verbose debug output
-e, --error-on-warnings
Have Python warnings raise DeprecationWarning on tests
run
-k, --keep-venv Do not remove created venv
-m MIRROR, --mirror MIRROR
URL for pip to use for Simple API [Default:
https://pypi.org/simple/]
--print-cov Print modules coverage report
--print-non-configured
Print modules not configured to run ptr
--progress-interval PROGRESS_INTERVAL
Seconds between status update on test running
[Default: Disabled]
--run-disabled Force any disabled tests suites to run
--stats-file STATS_FILE
JSON statistics file [Default: /var/folders/tc/hbwxh76
j1hn6gqjd2n2sjn4j9k1glp/T/ptr_stats_12510]
--system-site-packages
Give the virtual environment access to the system
site-packages dir
--venv VENV Path to venv to reuse
--venv-timeout VENV_TIMEOUT
Timeout in seconds for venv creation + deps install
[Default: 120]

Configuration 🧰
ptr is configured by placing directives in one or more of the following files. .ptrconfig provides
base configuration and default values for all projects in the repository, while each setup.(cfg|py)
overrides the base configuration for the respective packages they define.
.ptrconfig
ptr supports a general config in ini (ConfigParser) format.
A .ptrconfig file can be placed at the root of any repository or in any directory within your repository.
The first .ptrconfig file found via a recursive walk to the root ("/" in POSIX systems) will be used.
Please refer to ptrconfig.sample for the options available.
setup.py
This is per project in your repository. A simple example, based on ptr itself:
# Specific Python Test Runner (ptr) params for Unit Testing Enforcement
ptr_params = {
# Where mypy will run to type check your program
"entry_point_module": "ptr",
# Base Unittest file
"test_suite": "ptr_tests",
"test_suite_timeout": 300,
# Relative path from setup.py to module (e.g. ptr == ptr.py)
"required_coverage": {"ptr.py": 99, "TOTAL": 99},
# Run `black --check` or not
"run_black": False,
# Run mypy or not
"run_mypy": True,
}

pyproject.toml
This is per project in your repository and if exists is preferred over setup.py and setup.cfg.
Please refer to pyproject.toml
for the options available + format.
setup.cfg
This is per project in your repository and if exists is preferred over setup.py.
Please refer to setup.cfg.sample
for the options available + format.
mypy Specifics
When enabled, (in setup.(cfg|py)) mypy can support using a custom mypy.ini for each setup.py (module) defined.
To have ptr run mypy using you config:

create a mypy.ini in the same directory as your setup.py
OR add [mypy] section to your setup.cfg

mypy Configuration Documentation can be found here

An example setup.cfg can be seen here.

Example Output 📝
Here are some example runs.
Successful ptr Run:
Here is what you want to see in your CI logs!
[2019-02-06 21:51:45,442] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:51:59,471] INFO: Successfully created venv @ /var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_24397 to run tests (14s) (ptr.py:547)
[2019-02-06 21:51:59,472] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:52:00,726] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
[2019-02-06 21:52:04,153] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-02-06 21:52:04,368] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:09,915] INFO: Running flake8 for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:10,422] INFO: Running pylint for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-05-03 14:54:14,020] INFO: Running pyre for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
[2019-02-06 21:52:07,733] INFO: /Users/cooper/repos/ptr/setup.py has passed all configured tests (ptr.py:509)
-- Summary (total time 22s):

✅ PASS: 1
❌ FAIL: 0
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

Unsuccessful ptr Run Examples:
Here are some examples of runs failing. Any "step" can fail. All output is predominately the underlying tool.
Unit Test Failure
[2019-02-06 21:53:58,121] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:53:58,143] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:53:59,698] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
-- Summary (total time 5s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'tests_run' step):
...F....................
======================================================================
FAIL: test_config (__main__.TestPtr)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/cooper/repos/ptr/ptr_tests.py", line 125, in test_config
self.assertEqual(len(sc["ptr"]["venv_pkgs"].split()), 4)
AssertionError: 5 != 4

----------------------------------------------------------------------
Ran 24 tests in 3.221s

FAILED (failures=1)

coverage
[2019-02-06 21:55:42,947] INFO: Starting ptr.py (ptr.py:782)
[2019-02-06 21:55:42,969] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:417)
[2019-02-06 21:55:44,920] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:417)
[2019-02-06 21:55:49,628] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:417)
-- Summary (total time 7s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'analyze_coverage' step):
The following files did not meet coverage requirements:
ptr.py: 84 < 99 - Missing: 146-147, 175, 209, 245, 269, 288-291, 334-336, 414-415, 425-446, 466, 497, 506, 541-543, 562, 611-614, 639-688

black
[2019-02-06 22:34:20,029] INFO: Starting ptr.py (ptr.py:804)
[2019-02-06 22:34:20,060] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:430)
[2019-02-06 22:34:21,614] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:430)
[2019-02-06 22:34:25,208] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
[2019-02-06 22:34:25,450] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
[2019-02-06 22:34:26,422] INFO: Running black for /Users/cooper/repos/ptr/setup.py (ptr.py:430)
-- Summary (total time 7s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'black_run' step):
would reformat /Users/cooper/repos/ptr/ptr.py
All done! 💥 💔 💥
1 file would be reformatted, 4 files would be left unchanged.

mypy
[2019-02-06 22:35:39,480] INFO: Starting ptr.py (ptr.py:802)
[2019-02-06 22:35:39,531] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:428)
[2019-02-06 22:35:41,203] INFO: Running /Users/cooper/repos/ptr/ptr_tests.py tests via coverage (ptr.py:428)
[2019-02-06 22:35:45,156] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:428)
[2019-02-06 22:35:45,413] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:428)
-- Summary (total time 6s):

✅ PASS: 0
❌ FAIL: 1
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'mypy_run' step):
/Users/cooper/repos/ptr/ptr.py: note: In function "_write_stats_file":
/Users/cooper/repos/ptr/ptr.py:179: error: Argument 1 to "open" has incompatible type "Path"; expected "Union[str, bytes, int]"
/Users/cooper/repos/ptr/ptr.py: note: In function "run_tests":
/Users/cooper/repos/ptr/ptr.py:700: error: Argument 1 to "_write_stats_file" has incompatible type "str"; expected "Path"

pyre
cooper-mbp1:ptr cooper$ /tmp/tp/bin/ptr --venv /var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_49117
[2019-05-03 14:51:43,623] INFO: Starting /tmp/tp/bin/ptr (ptr.py:1023)
[2019-05-03 14:51:43,657] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:565)
[2019-05-03 14:51:44,840] INFO: Running ptr_tests tests via coverage (ptr.py:565)
[2019-05-03 14:51:47,361] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,559] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,827] INFO: Running black for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:47,996] INFO: Running flake8 for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:48,566] INFO: Running pylint for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:52,301] INFO: Running pyre for /Users/cooper/repos/ptr/setup.py (ptr.py:565)
[2019-05-03 14:51:54,983] INFO: /Users/cooper/repos/ptr/setup.py has passed all configured tests (ptr.py:668)
-- Summary (total time 11s):

✅ PASS: 1
❌ FAIL: 0
⌛️ TIMEOUT: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'pyre_run' step):
2019-05-03 14:54:14,173 INFO No binary specified, looking for `pyre.bin` in PATH
2019-05-03 14:54:14,174 INFO Found: `/var/folders/tc/hbwxh76j1hn6gqjd2n2sjn4j9k1glp/T/ptr_venv_49117/bin/pyre.bin`
... *(truncated)* ...
ptr.py:602:25 Undefined name [18]: Global name `stdout` is not defined, or there is at least one control flow path that doesn't define `stdout`.

usort
If your imports are not making usort happy it would look like this:
[2021-05-29 09:30:56,044] INFO: Starting /tmp/tp/bin/ptr (ptr.py:1129)
[2021-05-29 09:30:56,051] INFO: Installing /Users/cooper/repos/ptr/setup.py + deps (ptr.py:637)
[2021-05-29 09:30:56,587] INFO: Running ptr_tests tests via coverage (ptr.py:637)
[2021-05-29 09:30:58,238] INFO: Analyzing coverage report for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
[2021-05-29 09:30:58,341] INFO: Running mypy for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
[2021-05-29 09:30:58,436] INFO: Running usort for /Users/cooper/repos/ptr/setup.py (ptr.py:637)
-- Summary (total time 2s):

✅ PASS: 0
❌ FAIL: 1
️⌛ TIMEOUT: 0
🔒 DISABLED: 0
💩 TOTAL: 1

-- 1 / 1 (100%) `setup.py`'s have `ptr` tests running

-- Failure Output --

/Users/cooper/repos/ptr/setup.py (failed 'usort_run' step):
Would sort /Users/cooper/repos/ptr/setup.py

FAQ ⁉️
Q. How do I debug? I need output!

ptr developers recommend that if you want output, please cause a test to fail

e.g. raise ZeroDivisionError


Another recommended way is to run your tests with the default setup.py test using a ptr created venv:

cd to/my/code
/tmp/venv/bin/python setup.py test



Q. How do I get specific version of black, coverage, mypy etc.?

Just simply hard set the version in the .ptrconfig in your repo or use requirements.txt to pre-install before running ptr
All pip PEP 440 version specifiers are supported

Q. Why is the venv creation so slow?

ptr attempts to update from a PyPI compatible mirror (PEP 381) or PyPI itself
Running a package cache or local mirror can greatly increase speed. Example software to do this:

bandersnatch: Can do selected or FULL PyPI mirrors. The maintainer is also devilishly good looking.
devpi: Can be ran and used to proxy packages locally when pip goes out to grab your dependencies.


Please ensure you're using the -k or --venv option to no recreate a virtualenv each run when debugging your tests!

Q. Why is ptr not able to run pyre on Windows?

pyre (pyre-check on PyPI) does not ship a Windows wheel with the ocaml pyre.bin

Q. Why do you depend on >= coverage 5.0.1

coverage 5.0 introduced using sqlite and we don't want to have a mix of 4.x and 5.x for ptr
< 5.0 could possibly still work as we now ensure to run each projects tests from setup_py.parent CWD with subprocess

Contact or join the ptr community 💬
To chat in real time, hit us up on IRC. Otherwise, GitHub issues are always welcome!
IRC: #pythontestrunner on FreeNode
See the CONTRIBUTING file for how to help out.
License
ptr is MIT licensed, as found in the LICENSE file.
Terms of Use + Privacy Policy

Terms of Use
Privacy Policy

Copyright © Meta Platforms, Inc. and affiliates
ptr Change History
Each release to PyPI I'm going to give a codename as to where I am or was in the world 🌏.
2022.7.12
Codename: Seattle 🦑
Release named in honoUr of my visit to Seattle last week ...

3.11 Supported - added metadata + made 3.11 CI pass - PR #121
Add pyproject.toml user config support - PR #120
Add basic Hypothesis fuzz testing - PR #118

Learnt during US PyCon 2022


Add pyproject.toml basic PEP517 building support - PR #117
This (again) will be the last 3.7 supported version

I forgot to remove + CI still passes so will leave for now :)



2022.2.2
Codename: Richie 2 for 22 🏏
Release named in honoUr of the Late Great Richie Benaud

This will be the last 3.7 supported version

Why? Want a less buggy asyncio.run for starting asyncio


Reactivate pylint in CI

f-string all ptr strings


Drop python 3.6 CI + support
We're Meta now - Update copyrights due to comapny change name

3.11 CI is failing due to depedencies. Will watch and fix asap.
2021.11.23
Codename: A 🅰️
Release to support 'Three dot Ten' and A == 10 in hexidecimal

Add Official Support and CI for Python 3.10

This will be the last release (if no bugs present themselves) for 3.6


Also start ci_latest running 3.11

2021.5.28
Codename: Memorial Day 🇺
Release on Memorial Day long weekend

Add usort to the family of CI options

https://pypi.org/project/usort/



2021.3.16
Codename: Wildwood 🪵
First release from my South Lake Tahoe residence

Support floats for coverage comparisions in required_coverage - PR #108
Change --print-cov to work when required_coverage does not exist in ptr config - PR #109
Move tests out of install but include in sdist - PR #99 - Thanks @jayvdb
Test with 3.10 alphas via deadsnakes - PR #110
Misc f-string + lint/typing fixes

2020.2.26
Codename: Month after Straya Day 🇦🇺🇺
1 month since @cooperlees was in AU for Australia day!

Enable pyre for Python 3.8 now it's fully supported

2019.12.25
Codename: Vernon Hills, IL 🎅
@cooperlees visiting girlfriend's family in IL

Fix documentation for some new arguments - Thanks @omikader
Windows now defaults to updating pip + setuptools
Rely on setting current working directory with subprocess and remove coverage file environment setting
f-string everything via flynt now we're >= 3.6

2019.12.13
Codename: College Park, MD 🐢
@omikader studied at the University of Maryland, College Park

Add ability to create partial .ptrconfig files - PR: #83
Use native recursive file search for black and flake8 - PR: #81

2019.12.12
Codename: Sapporo, Japan 🇯🇵
@omikader is going skiing there in February

Add support for passing --system-site-packages during venv creation - PR: #80

2019.11.21
Codename: Santa Clara, CA
@thatch lives there!*

Add license file for @thatch - PR: #77 - Thanks @thatch

2019.11.15
Codename: Russian River, CA 🇺🇸
@cooperlees is going to the Russian River tomorrow

Run ptr even when test_suite is not provided - PR: #73 - Thanks @spurav
Remove use of deprecated setuptools test option
Move to GitHub Actions for CI + Releases

2019.11.2
Codename: Mumbai, India 🇮🇳
@spurav is from Mumbai, India

Fix math error dividing 1 / 2 for os.cpu_count - PR: #69 - Thanks @spurav
Get GitHub Actions CI Running - PR: #68 + #70 - Thanks @adhaamehab

2019.10.22
Codename: Cairo, Egypt 🇪🇬
@adhaamehab who first time contributed is from there! Thanks!

Print more helpful error when file does not exist for coverage check - PR: #51
Officially Support Python 3.8 and enforce tests passing
Disable pyre-check for >=3.8 due to no support

Issue to Track: https://github.com/facebook/pyre-check/issues/213



2019.10.6
Codename: Stanford, CA
@cooperlees tailgated there yesterday 🚐🏈

Handle OSX /private in coverage report - PR: #60

2019.9.14
Codename: Clowntown 🤡

Fix bad bool passing for --print-cov + --error-on-warnings

2019.9.11
Codename: Rathbone Square
@cooperlees is releasing this release from FB London Office

Add CLI option to enable erroring on all warnings - Issue: #57

2019.8.7
Codename: Ellicott City
@omikader who reported and tested the fix for AST parsing is from there.
Ellicott City is also home to one of the oldest surviving train stations in the US!

Fix AST parsing to ignore AST targets that do not have an id attribute - Issue: #54
Renable black by default on 3.7 now that it runs on > 3.7.2 - Issue: #41
Fix --venv-timeout to store ints
Refactor _get_test_modules function to remove lint error - PR: #53 - Thanks @TomasFeeney

2019.7.16
Codename: California City
An over developed desert with infrastructure and little people

Add --print-non-configured to find non ptr modules in repos - Issue: #50

2019.6.15
Codename: Concord
🛫 @cooperlees is playing Aussie Football @ Concord, CA today 🏈

Add ability to disable test suites and add a --run-disabled option to force the run - Issue: #46
Azure CI now runs on Mac and Windows again

2019.5.3
Codename: Cleveland
🇺🇸 @cooperlees + @jreese are at PyCon US in Cleveland, OH 🦅

Print step name in failure output - Issue: #31
Add in pyre Type Checking step support - Issue: #38 + #40

Known Bug: black.exe does not run in Windows 3.7 - disabled by default on Python 3.7 on Windows
2019.3.5
Codename: Jaipur
🇮🇳 @cooperlees releasing whilst in Jaipur, India for a wedding 💒

Preliminary Windows support now ready for testing - Issue: #2
Run tests/mypy/flake8 etc. in CWD of setup.py path - Issue #23 - Thanks @jreese
Add support for linting with flake8 and pylint - Issue #20 - Thanks @jreese
Ignore dotted directories when running black - PR #19 - Thanks @jreese

Known Bug: black.exe does not run in Windows 3.7 - disabled by default on Python 3.7 on Windows
2019.2.12
Codename: Forbes
Forbes, NSW, Australia is the home of @aijayadams 👨🏻‍🦰🇦🇺

Added suite file coverage % to statistics JSON file - Issue: #16 - Thanks @aijayadams
Ignore hidden '.' (dot) directories when running black - PR: #19 - Thanks @jreese

2019.2.10
Codename: Carnival
🇧🇷 @cooperlees was in Rio de Janeiro, Brazil for Carnival 3 years ago today 🇧🇷

Added ptr setup.cfg support for ptr_params - Issue: #1
Added JSON stats validation to ci.py - Issue: #7
Fixed bug that allowed a step to run by default - Issue: #11

2019.2.8.post1/2

Fix setup.py URL to ptr GitHub
Other various setup.py fixes - e.g. Classifiers + License information

2019.2.8
Codename: Snowbird
Recent shredding of Snowbird, UT, USA took place 🏂 🇺🇸

Initial Release to the world!