mjbackup 1.0.1

Description:

mjbackup 1.0.1

MJBackup - Morning Joe Software Linux Server Backup System.
This script is called to perform a backup and in turn calls other scripts which perform
specialized task such as back up files, MySql databases, Postgresql databases, and anything
else that this can be extended to support.
There is also a script that can be used to rsync the backup files to the remote location. The length
these remote copies can be retained can be set by days.
Both scripts are meant to be used with crontab in a single linux server environment with database servers running locally.
Requirements

Linux server (Geared more towards standalone web server)
Python 3.6+ installed
Mysql installed (optional)
PostgreSql installed (optional)
rsync installed (optional)

Features

supports backing up files into a tar.gz archive
supports full database backup dumps of PosgreSql and MySql databases.
all backups are timestamped in the file name.
file backups can be grouped into different target archives and locations.
databases backup location can be set.
limits on the backup copies retained on the server.
logging is supported.
backup status/errors can be emailed to a list of email addresses.
rsync based script to help with coping the backups to a remote location.
support for clearing old backups after a set amount of days on the remote location.

TODO

more database support such as Mongo DB, SQL Server
database backups from multiple configs much like the file archives.
better logging and error reporting

Installation (Server)
Run the following command on the server:
user@linux:~$ mkdir mjbackup
user@linux:~$ cd mjbackup
user@linux:~$ pip install mjbackup -t .
user@linux:~$ cd mjbackup

There should be the following files for the server part:

mjbackup.py
mjbackupConfig.py
mjbackupFiles.py
mjbackupPgsql.py
mjbackupMysql.py

There should be the following files for the remote location:

mjsync.py

Copy the files for the server part to a location such as /opt/mjbackup/
user@linux:~$ sudo mkdir /opt/mjbackup
user@linux:~$ sudo cp mjbackup* /opt/mjbackup
user@linux:~$ sudo chown root /opt/mjbackup -R
user@linux:~$ sudo chgrp root /opt/mjbackup -R
user@linux:~$ sudo chmod 700 /opt/mjbackup -R

Make sure this location is only accessible to the superuser.
Make the configuration directory:
user@linux:~$ sudo mkdir /etc/mjbackup

Configuration (Server)
First add this job to crontab. This will perform the backup every Sunday at 3AM.
user@linux:~$ sudo crontab -e

add:
0 3 * * 0 python3 /opt/mjbackup/mjbackup.py

Go to the directory where you downloaded the package from pip and go to the "conf" subdirecotry.
There should be many sample configuration files. These need to be editied for your system.
mjbackupConfig.xml
<MJBackupConfig>
<BackupList>
<Path>backup1.xml</Path>
<Path>backup2.xml</Path>
<Path>backup3.xml</Path>
</BackupList>
<EmailNotifyList>
<Email>remote_email1@dont_use_this.com</Email>
<Email>remote_email2@dont_use_this.com</Email>
<Email>remote_email3@dont_use_this.com/Email>
</EmailNotifyList>
<EmailSource>example_email@dont_use_this.com</EmailSource>
<LimitCopies>3</LimitCopies>
<Verbose>1</Verbose>
</MJBackupConfig>

BackupList - List of paths to configuration files for each indivisual archive.
EmailNotifyList - List of emails to send a copy of the log every time a backup is done.
EmailSource - This is the "From" field when emails are sent out.
LimitCopies - Limit to how many backup copies are kept on the server. Default is 3.
Verbose - If set, information will be printed to the console. For testing.
backup(X).xml
<BackupList name="examplebackup1" target="example1" targetdir="/targetdir/backups1">
<Backup>
<Name>user1</Name>
<Path>/home/user1</Path>
<Note>user1 backup</Note>
</Backup>
<Backup>
<Name>user2</Name>
<Path>/home/user2</Path>
<Note>user2 backup</Note>
</Backup>
</BackupList>

Configuration for an individual archive. Backuplist has three attributes:
name - Name of backup group to display in logs/emails.
target - Name of the archive. A date/time stamp and the extension tar.gz will be appended to this in the file name e.g. backup.2020-04-20_050006.tar.gz
targetdir - The directory to put this archive into. Use absolute path such as /home/archive/backups.
Each Backup is a directory or file to add to the archive.
Name - Name of backup to display in logs/emails.
Path - Absolute path of directory or file to backup.
Note - Extra field to put more information about this backup.
mjbackupMysql.conf
[config]
username = mysql-login
password = mysql-password
hostname = localhost
backupdir = backups-mysql/

username - MySql admin login name
Password - MySql admin password.
hostname - hostname MySql server resides on.
backupdir - The directory to put this archive into. Use absolute path.
mjbackupPgsql.conf
[config]
username = postgres
password = postgres-password
hostname = localhost
backupdir = backups-pgsql/


username - PostgreSql admin login name
Password - PostgreSql admin password.
hostname - hostname MySql server resides on.
backupdir - The directory to put this archive into. Use absolute path.
After configuration
Copy all .xml and .conf files to /etc/mjbackup and make sure this directory is only accessible by the superuser.
user@linux:~$ sudo cp *.xml /etc/mjbackup
user@linux:~$ sudo cp *.conf /etc/mjbackup

Test it to see if it works correctly. If Verbose is set, you can see the output.
user@linux:~$ sudo python3 /opt/mjbackup/mjbackup.py

Installation (Remote Sync)
Before going further, make sure rsync is configured for passwordless login for the remote backup server.
Copy the files for the server part to a location such as /opt/mjbackup/
user@linux:~$ sudo mkdir /opt/mjbackup
user@linux:~$ sudo cp mjsync.py /opt/mjbackup
user@linux:~$ sudo chown root /opt/mjbackup -R
user@linux:~$ sudo chgrp root /opt/mjbackup -R
user@linux:~$ sudo chmod 700 /opt/mjbackup -R
user@linux:~$ sudo chmod 700 /opt/mjbackup -R

Make sure this location is only accessible to the superuser.
Make the configuration directory:
user@linux:~$ sudo mkdir /etc/mjbackup

Configuration (Remote Sync)
First add this job to crontab. This will perform the backup every Sunday at 5AM. (Give the server a few hours to do the job.)
user@linux:~$ sudo crontab -e

add:
0 5 * * 0 python3 /opt/mjbackup/mjsync.py

Go to the directory where you downloaded the package from pip and go to the "conf" subdirecotry.
There should be many sample configuration files. You only need mjsyn.conf.
mjsync.conf
[config]
username = exampleuser
hostname = 123.123.123.123
remote_backupdir = /server/backupdir
local_backupdir = /local/backupdir
days_to_keep = 30
verbose = True

username - rsync login name for the server.
hostname - rsync host to rsync to.
remote_backupdir - The backup directory on the server. Use absolute path.
local_backupdir - The local directory to copy to. Use absolute path.
days_to_keep - the days to keep old backups downloaded.
verbose - Show the status or not. For testing.
Copy all mjsync.conf files to /etc/mjbackup and make sure this directory is only accessible by the superuser.
user@linux:~$ sudo cp *.conf /etc/mjbackup

Test it to see if it works correctly. If Verbose is set, you can see the output.
user@linux:~$ sudo python3 /opt/mjbackup/mjsync.py