GitLocker: The Coding Marketplace

Description:

githeimdall 0.0.4.dev0

Git-Heimdall is a guardian/gatekeeper tool for scanning sensitive data before
committing files to github. It also provides functionality for creating
configuration files from existing template files.

Contents

Table of Contents:

Contents
What is Git-Heimdall?
Why Git-Heimdall?
Installation
More about Git-Heimdall secretfy
Development
How to Use
Config template file samples
FAQ
Support
License

What is Git-Heimdall?
Git-Heimdall is a tool for scanning sensitive data before the staged files are
added to a commit. So, if a developer makes some changes in his repository and
tries to commit those changes via git commit, those files will be first
scanned by Git-Heimdall to check whether there are any sensitive data in those
changes and informs developer about it.
This process is automated. You don’t have to run extra commands in order to scan those files. Infact those files are scanned automatically when git commit is called.

Why Git-Heimdall?
Committing sensitive data has been one of the most common vulnerabilities in
security world. Developers commit sensitive data and expose it to internet
without even realising about it.
This is where Git-Heimdall comes in. Git-Heimdall provides a set of functionalities like :-

Scanning files before commit to track secrets
Keep configuration files in template format so that there are less chances
of everytime modifying the config file and committing of secrets.(More about
Git-Heimdall secretfy)

Installation
This section provides quick steps of how to setup this tool.

Create a virtual Python environment and install Git-heimdall in it.
python3 -m venv vheimdall
. vheimdall/bin/activate
pip3 install git-heimdall
heimdall -i

Run Sanity test
heimdall secretfy -m
The above command creates mock templates, secrets file at
/tmp/git-heimdall directory. The -c or --config
option is for providing your config.yaml file.

Register current project with git-heimdall
heimdall -a <absolute-path-of-project>
The above command registers your project with heimdall scans. Next time when
you try to commit staged files, it would start scanning those files before
it gets added to commit and pushed to github.

More about Git-Heimdall secretfy
git-heimdall provides secretfy option for generating config files
dynamically from your template files. The templates are nothing but
configuration files, which holds your configuration in mustache format.
secretfy tool generates the required configuration file with help of secrets
file which would contain the real values required for actual config/properties
file.
Let’s just say you have a set of configuration which you keep in a file
config.yaml, config.json, application.properties etc. These configuration might
have some highly sensitive information required to execute your project like
your user credentials, email, phone number, private key etc. Everytime, in your
development process, you need to add these sensitive values to the config file
and remove them before committing the code into github.
This process is pretty painful and often you endup committing one or the other
sensitive information to git.
So, instead of having a config file, you can have a template which resembles
your config file. Now before executing your project. All you need to do is
generate the desired config file with the help of this tool and then follow the
usual approach of running the project. The best part is that you don’t have to
worry about accidently commit the actual config file to the git repo. That file
won’t be shown in git status unless you forcibly add it.

Development
This section describes how to set up a development environment for
Git-Heimdall. This section is useful for those who would like to contribute to
Git-Heimdall or run Git-Heimdall directly from its source.
We use primarily three tools to perform development on this project: Python 3,
Git, and Make. Your system may already have these tools. But if not, here are
some brief instructions on how they can be installed.

On macOS, if you have Homebrew installed, then these tools can be be installed easily with the following command:
brew install python git
On a Debian GNU/Linux system or in another Debian-based Linux distribution, they can be installed with the following commands:
apt-get update
apt-get install python3 python3-venv git make
On any other system, we hope you can figure out how to install these tools yourself.

Clone the project repository and enter its top-level directory:
git clone https://github.com/sunnysharmagts/git-heimdall
cd git-heimdall

Create a virtual Python environment for development purpose:
make vheimdall deps
This creates a virtual Python environment at ~/.vheimdall/git-heimdall.
Additionally, it also creates a convenience script named vheimdall in
the current directory to easily activate the virtual Python
environment which we will soon see in the next point.
To undo this step at anytime in future, i.e., delete the virtual
Python environment directory, either enter
rm -rf vheimdall ~/.vheimdall/.

Activate the virtual Python environment:
. ./vheimdall

In the top-level directory of the project, enter this command:
python3 -m heimdall -i
This initializes git-heimdall tool. This is just a one time process and
need not be run everytime, unless if there is any change in the template
resources. This command just updates in the location of the git templateDir
in git configuration.
python3 -m heimdall secretfy -m
This generates mock data at /tmp/git-heimdall. This step serves as a
sanity check that ensures that the development environment is correctly set
up. Also, it gives a brief idea of how to create a config in form of
template.

Now to simulate the environment and test the tool. Do the following:-
. ~/.vheimdall/git-heimdall/bin/activate
mkdir /tmp/git-heimdall-tool-test
cd /tmp/git-heimdall-tool-test
echo -n "print('<insert-some-sensitive-value>')" >> sample.py
git init
git add .
git commit -m "Sample commit"
This will start scanning the sample.py file and will provide you with the sensitive data that you have in this file. Add more files and play with the tool to familiar yourself.

How to Use
This section provides samples of how to use this tool.
heimdall is initialized once after installation. That can be done for the following :-
heimdall -i
After initialization, any new repository created or cloned will be scanned by
heimdall once any staged files are committed.
You can register existing repositories to heimdall to integrate scanning process.
heimdall -reg <absolute-path-of-repo>
heimdall secretfy consist of 3 components :-
Secrets file - This file can be in yaml, json and xml format.
Template files - These files are configuration files in template format. For
eg:- If you have a file config.json then your template file will be
config.json.mustache.
Extension - This is the file extension of your configuration file. Following
are the example config files and their respective extension.
a. config.yaml : yaml
b. config.xml : xml
c. config.json : json
d. config.properties : properties
These parameters can be added to a baseconfig.yaml file in the following way
heimdall:
secret: res/secrets.yaml
templates:
-
file: res/example.yaml.mustache
extension: yaml
-
file: res/example.json.mustache
extension: json
-
file: res/example.xml.mustache
extension: xml
The baseconfig.yaml file starts with heimdall tag.
1. secret is the absolute path of the secrets file containing sensitive
values.
2. templates tag is an array of template files. All these files are in
.mustache format whose sensitive values resides in secrets.yaml file.

file is the absolute path of the template file.
extension is the extension of the configuration file which will be
generated from the template file.

NOTE: Make sure that the template file are in <file_name>.<extension>.<mustache> format.
Run the following command to generate the config files.
heimdall secretfy -c baseconfig.yaml
This will create config files in the respective directories. Note that these
configurations won’t be seen in git history. You can check that by doing git status.

Config template file samples
secrets.yaml
secrets:
item:
val1: foo@bar.com
val2: my_password
item1:
val3: username
val4: my_private_key
example.yaml.mustache
secrets:
item:
val1: {{secrets.item.val1}}
val2: {{secrets.item.val2}}
result: This is just a dummy description.
item1:
val3: {{secrets.item1.val3}}
val4: {{secrets.item1.val4}}
result: This is another dummy description.
The secrets.yaml file contains the sensitive information and
example.yaml.mustache is the template file which contains the keys in
mustache format. Hence the key secrets.item.val2 has value my_password
which will be populated via heimdall secretfy tool.
NOTE: You can run `heimdall secretfy -m` to get more sample baseconfig, templates, secret files. These files will get generated at `/tmp/git-heimdall`.

FAQ
How can i deploy my code in CICD pipeline or on remote server since it
doesn’t have config files and needs to be generated.
You can generate all the config files required for your repository to compile
and run in CICD pipeline or at remote server by the following command.
heimdall secretfy -e mustache -s <secrets_file_path> -r <repository_path>
-e is the template extension, -s is the absolute path of the secrets file
and -r is absolute path of the repository

Support
To report bugs, suggest improvements, or ask questions, please create a
new issue at https://github.com/sunnysharmagts/git-heimdall/issues.

License
This is free software. You are permitted to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of it, under the
terms of the MIT License. See LICENSE.md for the complete license.
This software is provided WITHOUT ANY WARRANTY; without even the implied
warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See
LICENSE.md for the complete disclaimer.