amulog 0.3.10

Description:

amulog 0.3.10

Amulog is a tool to support system log management.
The main function is to classify log messages with automatically generated log templates (formats and variable locations),
and to store the data in a database.
This system works on python3.

Source: https://github.com/amulog/amulog
Bug Reports: https://github.com/amulog/amulog/issues
Author: Satoru Kobayashi
License: BSD-3-Clause


Main features

Support multiple databases: sqlite and mysql
Smart log segmentation with log2seq
Multiple template generation algorithms such as: Drain, SHISO, LenMa, FT-tree, Dlog, etc.
Support Online (incremental) and Offline (hindsight) use
Suspend and resume the template generation process
Import and Export log templates if you need
Edit log templates manually if you need
Search API with datetime, hostname and log template IDs



Tutorial

Install
$ pip install amulog


Generate config
For the first step, save following config as test.conf on an empty directory.
[general]
src_path = logfile.txt
src_recur = false
logging = auto.log

[database]
database = sqlite3
sqlite3_filename = log.db

[log_template]
lt_methods = drain
indata_filename = ltgen.dump
Then modify general.src_path option to a logfile you want to load.
(If you want to use multiple files, change general.src_recur into true and specify directory name to general.src_path.)


Generate database
Try following command to generate database:
$ python -m amulog db-make -c test.conf


Check database
$ python -m amulog show-db-info -c test.conf
shows status of the generated database.
$ python -m amulog show-lt -c test.conf
shows all generated log templates in the given logfile.
$ python -m amulog show-log -c test.conf ltid=2
shows all log messages corresponding to log template ID 2.


Resume generating database
Try following command to resume generating database:
$ python -m amulog db-add -c test.conf logfile2.txt


Export and Import templates
Following command exports all log templates in the database:
$ python3 -m amulog show-db-import -c test.conf > exported_tpl.txt
You can modify the exported templates manually.
Note that some special letters (\\, @, *) are escaped in the exported templates.
To import the templates, save following config as test2.conf.
[general]
src_path = logfile.txt
src_recur = false
logging = new_auto.log

[database]
database = sqlite3
sqlite3_filename = new_log.db

[log_template]
lt_methods = import
indata_filename = new_ltgen.dump

[log_template_import]
def_path = exported_tpl.txt
Then, try generating database again:
python -m amulog db-make -c test2.conf


Further usage
see help with following command:
python -m amulog -h



Reference
This tool is demonstrated at International Journal of Network Management and CNSM2020.
If you use this code, please consider citing:
@article{Kobayashi_IJNM2022,
author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke},
title = {amulog: A general log analysis framework for comparison and combination of diverse template generation methods*},
journal = {International Journal of Network Management},
volume = {32},
number = {4},
pages = {e2195},
doi = {https://doi.org/10.1002/nem.2195},
year = {2022}
}

@inproceedings{Kobayashi_CNSM2020,
author = {Kobayashi, Satoru and Yamashiro, Yuya and Otomo, Kazuki and Fukuda, Kensuke},
booktitle = {Proceedings of the 16th International Conference on Network and Service Management (CNSM'20)},
title = {amulog: A General Log Analysis Framework for Diverse Template Generation Methods},
pages={1-5},
year = {2020}
}