GitLocker: The Coding Marketplace

Description:

pythonpackagesynctool 0.5.3

PythonPackageSyncTool
PythonPackageSyncTool is a Python utility to "fix" requirements.txt.
It is used to make manipulation on requirements.txt that is produces by pip freeze
Getting Help
QuickStart
First of all it is better to install alex_ber_utils (this dependency is used in setup.py).
pip3 install -U alex_ber_utils

Than:
pip3 install -U python-package-sync-tool

In some directory create config.yml. Here is example
(https://github.com/alex-ber/alpine-anaconda3/blob/master/python_package_sync_tool_config.yml)
treeroot:
source: installed.txt
destination: rm.txt
mutual_exclusion: True
remove:
- conda
- conda-package-handling
- icc-rt
- intel-openmp
- mkl
- mkl-fft
- mkl-random
- mkl-service
- pycosat
- sip
- tbb
add:
# - graphviz==2.38.0
# - entrypoints==0.2.3
# - numpy==1.16.2
# - PyYAML==5.1
# - sasl==0.2.1
# - scipy==1.2.1
# - tables==3.4.4
# - terminado==0.8.2
# - tornado==5.1.1
# - Twisted==19.2.0

Create installed.txt with installed packages (in pip freeze format).
Now, you can run
python3 -m alexber.reqsync.data

You can also override parameters in the yaml file:
python3 -m alexber.reqsync.data --treeroot.add=some_new_package:1.0.0

shorthand
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0

is also supported.
Of course, you can provide all parameters via CLI and not to use YAML file.
You can also supply --general.log (no shorthand here) parameter (you can do it also via config.yml).
There is some default log configuration (to write everything to stdout).
Note: config.yml now also supports format that alex-ber-utils
https://github.com/alex-ber/AlexBerUtils support for a long time.
Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for
full supported list of general parameters.
See documentation here
My parser module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718
My ymlparsers module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6
My major init_app_conf module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c
Note:
Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.
You can specified multiple packages using comma delimiter.
You can specify path to your config file using --config_file (or --general.config_file).
It can be absolute or relative.

Alternatively,
cd /opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/

Note: This is Path where you're actually install my utility, it can be different in your machine.
If you use venv it will look something like:
cd /opt/MyProject/venv/Lib/site-packages/alexber/reqsync

Alternatively you can create script file for yourself, named, say, driver.py:
#!/usr/bin/python3

import alexber.reqsync.app as app

if __name__ == "__main__":
app.main()

Than create file config.yml near your script (see data/config.yml) or provide all parameter using command line
argruments. Use ':' in places where you should naturally write '==' (see explanation below).
Parameters 'source' and 'destination' are required. You should also provide (requirements) file for 'source'.
mutual_exclusion has default value True.

Now, type
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0

or if you're using script (driver.py) go the directory with the script and type
./driver.py --add=some_new_package:1.0.0

or if you install my tool to Anaconda/Python/venv that has it's bin folder is in the Path
you can run
python_package_sync_tool --add=some_new_package:1.0.0

or alternativley
you can run
reqsync --add=some_new_package:1.0.0

This will run quick check whether package is not in remove list. If it is, the utility will fail. You can override this
beahivor by supplying --mutual_exclusion=False.
Then, this will add some_new_package with version 1.0.0 to the requirements-dest.txt
Note:
Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.
You can specified multiple packages using comma delimiter.
You can specifiy path to your config file using --config_file (or --general.config_file).
It can be absolute or relative. If you're running using script (driver.py), that it can be relative to the directory
whether you put your script. If you're running as the module (python3 -m), it can be relative to
/opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/ (exact path can be different, see above).

You can supplied multiply packages by using comma:
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0,another_new_package:2.0.0

or if you're using script (driver.py) go the directory with the script and type
./driver.py --add=some_new_package:1.0.0,another_new_package:2.0.0

Installing from Github
python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip

Optionally installing tests requirements.
python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip#egg=python-package-sync-tool[tests]

Or explicitly:
wget https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip -O master.zip; unzip master.zip; rm master.zip

And then installing from source (see below).
Installing from source
python3 -m pip install . # only installs "required"

python3 -m pip install .[test] # installs dependencies for tests

Using Docker
alexberkovich/python_package_sync_tool:latest contains all python_package_sync_tool dependencies.
This Dockerfile is very simple, you can take relevant part for you and put them into your Dockerfile.

Alternatively, you can use it as base Docker image for your project and add/upgrade
another dependencies as you need.
For example:
FROM alexberkovich/python_package_sync_tool:latest

COPY requirements.txt etc/requirements.txt

RUN set -ex && \
#latest pip,setuptools,wheel
pip install --upgrade pip setuptools wheel && \
pip install python_package_sync_tool
pip install -r etc/requirements.txt

CMD ["/bin/sh"]
#CMD tail -f /dev/null

where requirements.txt is requirements for your project.

From the directory with setup.py
python3 setup.py test #run all tests

pytest

Requirements
PythonPackageSyncTool requires the following modules.

Python 3.7+

alex-ber-utils==0.6.2

alex-ber-utils[yml]==0.6.2

See https://github.com/alex-ber/AlexBerUtils
Changelog
All notable changes to this project will be documented in this file.
#https://pypi.org/manage/project/python-package-sync-tool/releases/
[0.5.3] - 2020-12-17
Changed

REAMDE.md, requiremetes section.

[0.5.2] - 2020-12-17
Removed

alexber.reqsync.utils.app_conf removed.

Changed

Pytest version updated (also some tests dependencies changed)
alex-ber-utils dependency updated.
Logger information was moved from app.py to config.yml file.
config.yml now also supports format that alex-ber-utils
https://github.com/alex-ber/AlexBerUtils support for a long time.

Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for
full supported list of general parameters.
See documentation here
My parser module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718
My ymlparsers module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6
My major init_app_conf module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c

Logger is enabled, relative path is resolved correctly, and other little tweaks, see

Integrating Python’s logging and warnings packages
[https://medium.com/analytics-vidhya/integrating-pythons-logging-and-warnings-packages-7ffd6f65e02d]
Making relative path to file to work
[https://alex-ber.medium.com/making-relative-path-to-file-to-work-d5d0f1da67bf]

config.xml - to remove package was added mkl*-related packagages and anaconda-specific packages.

REAMDE.md was updated to reflect all changes above.

Added

Optional Docker support added.

[0.5.1] - 2020-10-25
Added

Support for Python 3.8

Changed

[0.4.3] - 2019-10-17
Changed

anaconda-navigator and conda-build added to config.yml.

[0.4.2] - 2019-10-16
Changed

navigator-updater added to config.yml.

[0.4.1] - 2019-05-30
Changed

Bug fix: adding packages before all existing one works incorrect.
Removing alexber.reqsync.utils.parsers. It was fully duplicated by alexber.utils.parsers.
So, all usage was change to the latest (part of alex-ber-utils).

[0.3.1] - 2019-05-23
Changed

Bug fix: adding packages before all existing one works incorrect.

Added

Unit test for bug fix that check adding packages before all existing one.
More detail assertion to integration tests.
Unit-test for non-sorted requirements-src.
Unit-test that check run with empty add and empty remvoe.
Unit-test that check removing single package.
Unit-test that check remove first package in requirements-src.
Unit-test that check remove last package in requirements-src.
Unit-test that check that empty lines in requirements-src are ignored.
Unit-test that check correct usage of file input buffer and file output buffer.

[0.2.11] - 2019-05-22
Changed

Dependency alex-ber-utils bumped up to 0.2.5.

[0.2.8] - 2019-05-22
Changed

Dependency alex-ber-utils bumped up to 0.2.4.

[0.2.6] - 2019-05-22
Changed

Dependency alex-ber-utils bumped up to 0.2.3.

[0.2.5] - 2019-05-22
Changed

Fixed bug in setup.py, incorrect order between VERSION and UploadCommand (no tag was created on upload)
Dependency alex-ber-utils bumped up to 0.2.2.

[0.2.4] - 2019-05-22
Changed

Adding dependency alex-ber-utils 0.2.1 to README.md.

[0.2.3] - 2019-05-22
Changed

Upgrading urllib3, SQLAlchemy, pycrypto dependenies beacause of volnurabilities issues.

[0.2.2] - 2019-05-22
Changed

Fixing python-package-sync-tool.
Creating alias reqsync to python-package-sync-tool.
Some minour fixed.

[0.2.1] - 2019-05-22
Changed

Changing dependency version of alex-ber-utils to 0.2.1.

[0.2.0] - 2019-05-22
Changed

Only bumping up version.

[0.1.9] - 2019-05-22
Changed

Added alex-ber-utils as dependency.
Deleting old README-old.rst file.
requirements-src.txt updated.
Clarification added to README.md that alex_ber_utils should be installed first.
formatting CHANGELOG.MD (minor fix)
Added alternative of usage of python_package_sync_tool to README.md
Fixing bug that --add is empty
Factor out tests_data to seperate folder, use importlib.resources API.
Updated README-old.rst

[0.1.8] - 2019-05-20
Changed

README.md change, key '-U' added to pip3 install.

[0.1.7] - 2019-05-20
Removed

Some project cleanup.

Changed

CHANGELOG and REAMDE now use Markdown format.
REAMDE totally rewritten.
Fixing bugs in the core algorithm. Simplifying code.
Fixing correct handling of package adding to the buttom of the list.

[0.1.6] - 2019-05-20
Added

__init__.py file added to alexber.reqsync.data.

[0.1.5] - 2019-05-20
Added

Small tool to sync package from different machines.