GitLocker: The Coding Marketplace

Description:

pipstripper 0.3.0

TLDR: requirements without unnecessary pip packages.
For the purpose of this exercise, an unnecessary pip package is any package
that is not being imported by YOUR own Python code.
Unless you've provided a configuration override stating that you want it.

Simplest example: initialize, scan and build in current directory
$ pipstripper --init --build

pip-stripper configuration generated @ sample/tst.seedworkdir01/pip-stripper.yaml

build phase - generating requirements at:
sample/tst.seedworkdir01/requirements.dev.txt
sample/tst.seedworkdir01/requirements.txt

$cat requirements.txt
Django==2.2
Jinja2==2.10.1
celery==4.3.0
cx-Oracle==6.4.1
psycopg2==2.8.2
python-dateutil==2.8.0

$ cat requirements.dev.txt
pyquery==1.4.0

How it works
Lots of your packages may not be needed. Let's say you've installed black 18.9b0.
A linter and autoformatter has no need to be on a server:
pip-stripper most likely won't find import black anywhere, so it will not put it in requirements
options
usage: pipstripper [-h] [--config CONFIG] [--noscan] [--build] [--init]
[--workdir WORKDIR] [--verbose]

optional arguments:
-h, --help show this help message and exit
--config CONFIG config file. if not provided will look for pip-
stripper.yaml in --workdir, current directory
--noscan don't scan to classify packages. build phase will re-use
existing pip-stripper.scan.yaml. [False].
--build read pip-stripper.scan.yaml to create
requirements.prod/dev.txt [False]
--init initialize the config file (as pip-stripper.yaml) if it
doesn't exist
--workdir WORKDIR work directory [defaults to config file's value or
current directory]
--verbose verbose mode. adds extra zzz_debug: entry to pip-
stripper.scan.yaml [False]

Three phases, initialization, scan and build.
Initialization (defaults to False)
The first option --init will create pip-stripper.yaml, the configuration file for pip-stripper.
This is the only file you should edit manually!!!
Scan phase (will run unless you specify --noscan)
This will scan your Python source files in --workdir and use it to create pip-stripper.scan.yaml.
This is the file that contains instructions for the build phase.
Don't edit pip-stripper.scan.yaml!
Instead:

adjust the configuration in pip-stripper.yaml
re-run the scan

Scanning also creates 2 work files, tmp.pip-stripper.freeze.rpt and tmp.pip-stripper.imports.rpt, tracking pip packages and its best guesses at python imports, respectively.
Build (defaults to False)
--build takes what it finds in pip-stripper.scan.yaml and uses it to populate
requirements.txt and requirements.dev.txt.
If those requirements files don't suit you, you may need to edit pip-stripper.yaml.
Editing pip-stripper.yaml
This allows you to specify:

pip vs import aliases
specify which packages are just workstation-level and shouldn't go into requirements.
hardcode packages that need to go into either.
Associating your source directories to either prod or dev.

Aliases
You may have to enter pip to python import alias names manually (alias matching is something that needs work).
hardcoded_aliases:
PyYAML: yaml

Hardcoding package to requirement mapping:
Because psycopg2 is typically never really imported in a Django or SQLAlchemy context, but rather derived from the configuration, you need to specify it yourself as below. Same thing with django-redis-cache which is configured in django's settings.py as package path rather than an import.
ClassifierPip:

#the following are used to "hardcode" package names to given buckets.
buckets:
prod:
- psycopg2
- django-redis-cache

tests:
- nose
- pytest

workstation:
# that's a workstation only package, so it's held back
- black

Associating Python directories to requirements:
This is a typical regex-based configuration telling which buckets the directories count as:
prod is the default outcome. First match wins, and tests is the only one needed.
ClassifierImport:
regex_dirs:
workstation: []
dev: []
tests:
- "/tests/"
prod: []

default_bucket: "prod"

Walkthrough:
case 1: hardcoding
Given a pip freeze line like
psycopg2==2.7.7

First, it looks for a matching entry in ClassifierPip:buckets in pip-stripper.yaml, (basically a hardcoded decision by the user of where to put it).
ClassifierPip:
buckets:
workstation:
- black
prod:
- psycopg2

This will result in psycopg2==2.7.7 going into requirements.prod.txt (when needed, requirements lines are always copied from the pip freeze output).
case 2: import classification.
grep-ing the Python code found this line:
./tests/helper_pyquery.py:57: from pyquery import PyQuery

Each file path is run against the regex specified by ClassifierImport:regex_dirs, so pyquery ends up in the tests bucket.
ClassifierImport:
regex_dirs:
tests:
- "/tests/"
prod: []
default_bucket: "prod"

finally, the --build pass looks at where buckets get mapped:
Builder:
req_mapper:
dev:
buckets:
- dev
- tests
prod:
buckets:
- prod

which puts pyquery in requirements.dev.txt.
case 3 multiple imports
./tests/helper_pyquery.py:57: from pyquery import PyQuery
./myserver/foobar.py:22: from pyquery import PyQuery

As before, pyquery is put in tests and prod buckets. But also in prod as myserver did not match any ClassifierImport:regex_dirs, meaning that default_bucket: "prod" was used.
enter bucket precedence
ClassifierPip:
bucket_precedence:
- prod
- tests
- dev
- workstation

prod beats tests so pyquery ends up only in prod bucket.
case 4. no import match was found and nothing was hardcoded.
Babel==2.6.0
will get left out of requirements. Which is not to say that it won't end up pip installed on your server if it is a dependency of some other package (pipdeptree can help you there).
Babel==2.6.0
- pytz [required: >=0a, installed: 2018.9]

Build Phase - the result of the --scan phase gets put into pip-stripper.scan.yaml:
Notice our friend black? We've explicitly classified it as workstation, so the scan didn't label it as unknown.
pips:
buckets:
dev:
....
tests:
- pyquery
prod:
- psycopg2
unknown:
- Babel
workstation:
- black

Look at the end for the warnings: section. In this case, repr was used with Python 2.7 but isn't necessary with Python 3, so I won't worry about it. A typical reason for a missing import is that automatic aliasing to link the pip name and import name didn't work.
warnings:
- missing import:repr

And that's it. The outcome?
my raw pip freeze weighs in at 158 packages:
$ wc -l requirements.freeze_raw.txt
158 requirements.freeze_raw.txt

my stripped down requirements ended up with 24 packages total:
$ wc -l requirements*txt | egrep 'prod|dev'
6 requirements.dev.txt
18 requirements.prod.txt

On my test environment, pip install -r requirements.prod.txt -r requirements.dev.txt got me 48 packages, after dependencies were pulled in.
$ pip freeze | wc -l
48

This is all very nice, but hopefully you have sufficient tests to allow you to be confident you didn't miss anything!
The way you can test is to create a new virtualenv, pip install both requirements files and then run your tests.
WARNING: be cautious in hardcoding package associations in dev/tests buckets rather than prod. Your tests could run to success, but production would still fail.
These are pretty conservative as nose and pytest are really only testing tools.
ClassifierPip:
buckets:
tests:
- nose
- nose2
- pytest