GitLocker: The Coding Marketplace

Description:

healthchecker 1.1.1

HealthChecker
A very simple python script to check the health of a service: make a GET request to it and if it answers it on time, then it's alive.
Additionally, it can write the result of the operation in a git repo hosted on Github or in a local file. Or notify dead services through a POST to some JSON API.
Actions are logged using Python's standard logger and displayed according to the log level set.
This app was thought mainly for our status page. It's currently being maintained in our repository.
Requirements

Python 3.7+
PyGithub 1.43+
requests 2.21+

Usage
usage: healthchecker [-h] [--gh-repo GH_REPO] [--gh-filename GH_FILENAME]
[--gh-branch GH_BRANCH] [--gh-token GH_TOKEN]
[--gh-email GH_EMAIL] [--notify-url NOTIFY_URL]
[--notify-payload NOTIFY_PAYLOAD]
[--notify-header NOTIFY_HEADER] [--notify-json]
[--version] [--validation VALIDATION] [--output OUTPUT]
[url [url ...]]

HealthChecker v1.1.0 by HacKan (https://hackan.net) FOSS under GNU GPL v3.0 or
newer. Checks URLs through HTTP GET requests to verify their availability.
Optionally writes the status result to a file in Github. Using Github requires
the repository name, the filename and an API token. Besides the ones listed
below, the following env vars exist: HEALTHCHECKER_LOG_LEVEL sets the minimal
logging level and defaults to info (can be: debug, info, warning, error,
critical); HEALTHCHECKER_REQUESTS_TIMEOUT sets the amount of time in seconds
to wait for services to respond and defaults to 10 seconds (setting a very low
value might cause several false positives). Note: command-line parameters will
always supersede env vars.

positional arguments:
url (HEALTHCHECKER_URLS (comma-separated)) URL to check

optional arguments:
-h, --help show this help message and exit
--version show version and exit
--validation VALIDATION
(HEALTHCHECKER_URLS_VALIDATION (comma-separated))
string to find in the body of a request to an URL as a
validation, one per URL (or the last one is used for
the remaining URLs) (this parameter can be repeated as
needed)
--output OUTPUT, -o OUTPUT
(HEALTHCHECKER_OUTPUT) store result in a file,
overwriting if exists (use `-` for standard output)

github options:
--gh-repo GH_REPO (HEALTHCHECKER_GITHUB_REPO) repository in the form of
<user|org>/<repo> (case insensitive), i.e.:
HacKanCuBa/b2rsum
--gh-filename GH_FILENAME
(HEALTHCHECKER_GITHUB_FILENAME) filename to modify
(include path if it is in a subdir such as
path/to/file.ext)
--gh-branch GH_BRANCH
(HEALTHCHECKER_GITHUB_BRANCH) branch where commits are
done (defaults to main)
--gh-token GH_TOKEN (HEALTHCHECKER_GITHUB_API_TOKEN) API token or
client_id,client_secret (bypasses the one supplied
through the environment)
--gh-email GH_EMAIL (HEALTHCHECKER_GITHUB_COMMITTER_EMAIL) git committer
email (the committer name is hardcoded to
HealthChecker)

notify options:
--notify-url NOTIFY_URL
(HEALTHCHECKER_NOTIFY_URL) URL to POST the status
notification
--notify-payload NOTIFY_PAYLOAD
(HEALTHCHECKER_NOTIFY_PAYLOAD) payload to send to the
notify URL: it is prepended to the comma-separated
list of URLs that failed validation, unless that it
contains the string HEALTHCHECKER_FAILED_URLS (case
sensitive), where it will replace that string by the
comma-separated list of URLs, and send the entire
payload
--notify-header NOTIFY_HEADER
(HEALTHCHECKER_NOTIFY_HEADERS (comma-separated))
header to send to the notify URL, which must be
specified as name and value separated by a semicolon:
<header name>:<header value> (this parameter can be
repeated as needed)
--notify-json (HEALTHCHECKER_NOTIFY_JSON (true/false)) send the
payload JSON encoded (it also adds the proper Content-
Type header)

Environment variables

HEALTHCHECKER_GITHUB_REPO: repository in the form of <user|org>/ (case insensitive), i.e.: HacKanCuBa/b2rsum.
HEALTHCHECKER_GITHUB_FILENAME: filename to modify (include path if it is in a subdir such as path/to/file.ext).
HEALTHCHECKER_GITHUB_BRANCH: branch where commits are done (defaults to main).
HEALTHCHECKER_GITHUB_API_TOKEN: API token or client_id,client_secret (bypasses the one supplied through the environment).
HEALTHCHECKER_GITHUB_COMMITTER_EMAIL: git committer email (the committer name is hardcoded to HealthChecker).
HEALTHCHECKER_URLS: URLs to check, comma-separated.
HEALTHCHECKER_LOG_LEVEL: minimal logging level, defaults to info (can be: debug, info, warning, error, critical).
HEALTHCHECKER_REQUESTS_TIMEOUT: amount of time in decimal seconds to wait for services to respond and defaults to 10 seconds (setting a very low value might cause several false positives).
HEALTHCHECKER_NOTIFY_URL: URL to send failed checks via POST as notification, comma-separated.
HEALTHCHECKER_URLS_VALIDATION: comma-separated list of validations to run on given URLs.
HEALTHCHECKER_NOTIFY_PAYLOAD: payload to send to the notify URL: it is prepended to the comma-separated list of URLs that failed validation, unless that it contains the string HEALTHCHECKER_FAILED_URLS (case sensitive), where it will replace that string by the comma-separated list of URLs, and send the entire payload.

Example 1: HEALTHCHECKER_NOTIFY_PAYLOAD=here comes the failed urls...
Example 2: HEALTHCHECKER_NOTIFY_PAYLOAD={"data": "HEALTHCHECKER_FAILED_URLS"}

HEALTHCHECKER_NOTIFY_HEADERS: headers to send to the notify URL, which must be specified as name and value separated by a semicolon: header name:header value, and successive headers separated by comma.

Example 1: HEALTHCHECKER_NOTIFY_HEADERS=X-Auth:4c18a291d7d8e7946cb9db9cbb3e1f49
Example 2: HEALTHCHECKER_NOTIFY_HEADERS=Content-Type:application/json,X-MyVal:1

HEALTHCHECKER_NOTIFY_JSON: (true/false) send the payload JSON encoded (it also adds the proper Content-Type header).
HEALTHCHECKER_OUTPUT: store result in a file (use - for standard output).

Responses
HealthChecker reports two values for each service checked:

alive: means that the service has responded (no timeout and status code lower than 500).
ok: means that the service has responded and has a status code of 2xx; additionally, if checks are given for the service this means that checks passed.

The app will follow any 3xx response accordingly.
Examples
Simply print checks result:
:~$ healthchecker https://rlab.be adm.rlab.be
INFO 2019-04-09 01:06:37 Begin checking URL https://rlab.be...
INFO 2019-04-09 01:06:37 Begin checking URL http://adm.rlab.be...
INFO 2019-04-09 01:06:37 Finish checking URL https://rlab.be: alive and OK
INFO 2019-04-09 01:06:37 Finish checking URL http://adm.rlab.be: alive and OK

Or show output through stdout, hiding logs:
:~$ HEALTHCHECKER_LOG_LEVEL=critical healthchecker -o - https://rlab.be adm.rlab.be 2> /dev/null
[
{
"uid": "416ff55a",
"url": "https://rlab.be",
"alive": true,
"ok": true
},
{
"uid": "5d961608",
"url": "http://adm.rlab.be",
"alive": true,
"ok": true
}
]

That output can be parsed by jq or any other JSON tool! Additionally, it can be saved to a file for later reference:
:~$ healthchecker -o checks.json https://rlab.be adm.rlab.be
INFO 2019-04-09 01:06:37 Begin checking URL https://rlab.be...
INFO 2019-04-09 01:06:37 Begin checking URL http://adm.rlab.be...
INFO 2019-04-09 01:06:37 Finish checking URL https://rlab.be: alive and OK
INFO 2019-04-09 01:06:37 Finish checking URL http://adm.rlab.be: alive and OK
INFO 2019-04-09 01:06:37 Result stored as "/home/hackan/Workspace/healthchecker/checks.json"

:~$ cat checks.json
[
{
"uid": "416ff55a",
"url": "https://rlab.be",
"alive": true,
"ok": true
},
{
"uid": "5d961608",
"url": "http://adm.rlab.be",
"alive": true,
"ok": true
}
]

Notify failed services to an endpoint: healthchecker --notify-url https://eoc.rlab.be/api/v1/status/ https://rlab.be http://wiki.rlab.be
Write to a file in Github and be very verbose:
:~$ HEALTHCHECKER_LOG_LEVEL=debug healthchecker --gh-repo rlyehlab/sysadmins --gh-filename data/healthcheck.json --gh-token ab410...2cc https://git.rlab.be
INFO 2019-04-09 01:07:32 Begin checking URL https://git.rlab.be...
ERROR 2019-04-09 01:07:32 Error GETing data from/to https://git.rlab.be: ConnectionError(MaxRetryError("HTTPConnectionPool(host='git.rlab.be', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x6ac59ffbd001>: Failed to establish a new connection: [Errno -2] Name or service not known'))"))
DEBUG 2019-04-09 01:07:32 Request to https://git.rlab.be took 0.18 seconds
INFO 2019-04-09 01:07:32 Finish checking URL http://git.rlab.be: dead
INFO 2019-04-09 01:07:33 Getting repository information...
INFO 2019-04-09 01:07:38 File data/healthcheck.json updated: 0335a8088f5aff42f078a9396916c8adbcc1a6c3

Parameters can be passed through env vars and/or through command-line indistinctly (command-line parameters will always supersede env vars):
:~$ HEALTHCHECKER_URLS_VALIDATION="Services | Administration,Adventurous writings by R'lyeh Sysadmins" healthchecker --notify-url 127.0.0.1:8000 https://adm.rlab.be https://blog.adm.rlab.be
INFO 2019-04-09 00:59:44 Begin checking URL https://adm.rlab.be...
INFO 2019-04-09 00:59:44 Begin checking URL https://blog.adm.rlab.be...
INFO 2019-04-09 00:59:44 Finish checking URL https://blog.adm.rlab.be: alive and OK
INFO 2019-04-09 00:59:44 Finish checking URL https://adm.rlab.be: alive and OK

All checks were OK, but should one fail...:
HEALTHCHECKER_LOG_LEVEL=debug HEALTHCHECKER_URLS_VALIDATION="Services | Administration,Adventurous writings by R'lyeh Sysadmins" python -m healthchecker --validation "non-existent string" --notify-url 127.0.0.1:8000 https://adm.rlab.be https://blog.adm.rlab.be
INFO 2019-04-09 01:03:39 Begin checking URL https://adm.rlab.be...
INFO 2019-04-09 01:03:39 Begin checking URL https://blog.adm.rlab.be...
DEBUG 2019-04-09 01:03:40 Request to https://blog.adm.rlab.be took 0.18 seconds
INFO 2019-04-09 01:03:40 Finish checking URL https://blog.adm.rlab.be: alive but not OK
DEBUG 2019-04-09 01:03:40 Request to https://adm.rlab.be took 0.30 seconds
INFO 2019-04-09 01:03:40 Finish checking URL https://adm.rlab.be: alive but not OK
DEBUG 2019-04-09 01:03:40 Notifying http://127.0.0.1:8000 with headers: {} and payload: https://adm.rlab.be,https://blog.adm.rlab.be
ERROR 2019-04-09 01:03:40 Error POSTing data from/to http://127.0.0.1:8000: ConnectionError(MaxRetryError("HTTPConnectionPool(host='127.0.0.1', port=8000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f49d9767f28>: Failed to establish a new connection: [Errno 111] Connection refused'))"))
DEBUG 2019-04-09 01:03:40 Request to http://127.0.0.1:8000 took 0.00 seconds
ERROR 2019-04-09 01:03:40 Could not notify http://127.0.0.1:8000

Note that nobody was listening at 127.0.0.1:8000 so the notification failed as well.
Run
Can be run as a python module python -m healthchecker ... or directly as a CLI tool healthchecker .... Read below on how to run the docker image.
PyPi
HealthChecker is in PyPi and can be installed with any standard tool such as pip (pip install healthchecker) or poetry (poetry add healthchecker). To add it as a dependency in your project it's recommended to use the hash parameter for pip. Hashes (and more) are listed in every release.
Repo
Clone the repo and install requirements:

installing requirements with poetry (recommended): poetry install. Then start virtualenv with poetry shell.

Docker
Build
Build locally with docker build --compress --pull --rm --tag registry.rlab.be/sysadmins/healthchecker:latest . or invoke build
Pull
You can pull from our registry with docker pull registry.rlab.be/sysadmins/healthchecker:latest
Run
Run with docker run --rm registry.rlab.be/sysadmins/healthchecker:latest ...
Alternatively, use env vars by creating an env file and passing it to docker:
cp sample.env .env
vim .env # edit and populate vars
docker run --rm --env-file .env registry.rlab.be/sysadmins/healthchecker:latest

Deploy
This can be deployed in a server creating a SystemD service and timer (the image will be pulled by Docker on first run):

Service

Create an env file where ever you want, as in /srv/healthchecker/.env (protect access to it with linux permissions). It can be anywhere with any name, just point it in the parameter --env-file in the service file. Then create the service file /etc/systemd/system/healthchecker.service:
[Unit]
Description=HealthChecker Service
Requires=docker.service
After=network.target docker.service

[Service]
Type=simple
ExecStart=/usr/bin/docker run --rm --env-file /srv/healthchecker/.env registry.rlab.be/sysadmins/healthchecker:latest
User=root
Group=docker

Alternatively, you can skip env file usage and write every parameter in the ExecStart line, but writing the API token there means it will be visible in the process list which is usually not a good idea.

Timer:

Create the timer file /etc/systemd/system/healthchecker.timer (use the same name as the service but with the .timer extension):
[Unit]
Description=HealthChecker Service Timer

[Timer]
OnBootSec=600
OnUnitActiveSec=5m

[Install]
WantedBy=multi-user.target

This sample is set to run the service 5 minutes after boot and then every 5 minutes. Read the documentation if you need to set different parameters.

Reload SystemD services: systemctl daemon-reload
Enable the timer: systemctl enable healthchecker.timer
Start it: systemctl start healthchecker.timer

Tip: you can see the execution log with journalctl -u healthchecker (or the service name used). Check journalctl help for additional filtering options.
Thanks @snkisuke for your help with this section.
Developing and PRing
For more information and use cases, refer to DEVELOPERS.md.
Collaborators
Many thanks to those that collaborate with this project (in alphabetical order): @erus, @seykron, @snkisuke.
License
HealthChecker is made by HacKan under GNU GPL v3.0+. You are free to use, share, modify and share modifications under the terms of that license.
Copyright (C) 2019 HacKan (https://hackan.net)

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.