python-logging-rabbitmq 2.3.0

Description:

pythonloggingrabbitmq 2.3.0

python-logging-rabbitmq

Logging handler to ships logs to RabbitMQ. Compatible with Django.
Installation
Install using pip.
pip install python_logging_rabbitmq

Versions



Version
Dependency




>= 2.x
Pika == 0.13


<= 1.1.1
Pika <= 0.10



Handlers
This package has two built-in handlers that you can import as follows:
from python_logging_rabbitmq import RabbitMQHandler

or (thanks to @wallezhang)
from python_logging_rabbitmq import RabbitMQHandlerOneWay




Handler
Description




RabbitMQHandler
Basic handler for sending logs to RabbitMQ. Every record will be delivered directly to RabbitMQ using the exchange configured.


RabbitMQHandlerOneWay
High throughput handler. Initializes an internal queue where logs are stored temporarily. A thread is used to deliver the logs to RabbitMQ using the exchange configured. Your app doesn't need to wait until the log is delivered. Notice that if the main thread dies you might lose logs.



Standalone python
To use with python first create a logger for your app, then create an instance of the handler and add it to the logger created.
import logging
from python_logging_rabbitmq import RabbitMQHandler

logger = logging.getLogger('myapp')
logger.setLevel(logging.DEBUG)

rabbit = RabbitMQHandler(host='localhost', port=5672)
logger.addHandler(rabbit)

logger.debug('test debug')

As result, a similar message as follows will be sent to RabbitMQ:
{
"relativeCreated":280.61580657958984,
"process":13105,
"args":[],
"module":"test",
"funcName":"<module>",
"host":"albertomr86-laptop",
"exc_text":null,
"name":"myapp",
"thread":140032818181888,
"created":1482290387.454017,
"threadName":"MainThread",
"msecs":454.01692390441895,
"filename":"test.py",
"levelno":10,
"processName":"MainProcess",
"pathname":"test.py",
"lineno":11,
"msg":"test debug",
"exc_info":null,
"levelname":"DEBUG"
}

Sending logs
By default, logs will be sent to RabbitMQ using the exchange 'log', this should be of type topic. The routing key used is formed by concatenating the logger name and the log level. For example:
import logging
from python_logging_rabbitmq import RabbitMQHandler

logger = logging.getLogger('myapp')
logger.setLevel(logging.DEBUG)
logger.addHandler(RabbitMQHandler(host='localhost', port=5672))

logger.info('test info')
logger.debug('test debug')
logger.warning('test warning')

The messages will be sent using the following routing keys:

myapp.INFO
myapp.DEBUG
myapp.WARNING

For an explanation about topics and routing keys go to https://www.rabbitmq.com/tutorials/tutorial-five-python.html
When create the handler, you're able to specify different parameters in order to connect to RabbitMQ or configure the handler behavior.
Overriding routing-key creation
If you wish to override routing-key format entirely, you can pass
routing_key_formatter function which takes LogRecord objects and returns
routing-key. For example:
RabbitMQHandler(
host='localhost',
port=5672,
routing_key_formatter=lambda r: (
'some_exchange_prefix.{}'.format(r.levelname.lower())
)
)

Configuration
These are the configuration allowed:



Parameter
Description
Default




host
RabbitMQ Server hostname or ip address.
localhost


port
RabbitMQ Server port.
5672


username
Username for authentication.
None


password
Provide a password for the username.
None


exchange
Name of the exchange to publish the logs. This exchange is considered of type topic.
log


declare_exchange
Whether or not to declare the exchange.
False


remove_request
If True (default), remove request & exc info.
True


routing_key_format
Customize how messages are routed to the queues.
{name}.{level}


routing_key_formatter
Customize how routing-key is constructed.
None


connection_params
Allow extra params to connect with RabbitMQ.
None


formatter
Use custom formatter for the logs.
python_logging_rabbitmq.JSONFormatter


close_after_emit
Close the active connection after send a log. A new connection is open for the next log.
False


fields
Dict to add as a field in each logs send to RabbitMQ. This is useful when you want fields in each log but without pass them every time.
None


fields_under_root
When is True, each key in parameter 'fields' will be added as an entry in the log, otherwise they will be logged under the key 'fields'.
True


message_headers
A dictionary of headers to be published with the message.
None


record_fields
A set of attributes that should be preserved from the record object.
None


exclude_record_fields
A set of attributes that should be ignored from the record object.
None


heartbeat
Lower bound for heartbeat timeout.
60


content_type
The format of the message sent to the queue.
text/plain



Examples
RabbitMQ Connection
rabbit = RabbitMQHandler(
host='localhost',
port=5672,
username='guest',
password='guest',
connection_params={
'virtual_host': '/',
'connection_attempts': 3,
'socket_timeout': 5000
}
)

Custom fields
rabbit = RabbitMQHandler(
host='localhost',
port=5672,
fields={
'source': 'MyApp',
'env': 'production'
},
fields_under_root=True
)

Custom formatter
By default, python_logging_rabbitmq implements a custom JSONFormatter; but if you prefer to format your own message you could do it as follow:
import logging
from python_logging_rabbitmq import RabbitMQHandler

FORMAT = '%(asctime)-15s %(message)s'
formatter = logging.Formatter(fmt=FORMAT)
rabbit = RabbitMQHandler(formatter=formatter)

For a custom JSON Formatter take a look at https://github.com/madzak/python-json-logger
Django
To use with Django add the handler in the logging config.
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'rabbit': {
'level': 'DEBUG',
'class': 'python_logging_rabbitmq.RabbitMQHandler',
'host': 'localhost'
}
},
'loggers': {
'myapp': {
'handlers': ['rabbit'],
'level': 'DEBUG',
'propagate': False
}
}
}

Configuration
Same as when use it with standalone python, you could configure the handle directly when declaring it in the config:
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'rabbit': {
'level': 'DEBUG',
'class': 'python_logging_rabbitmq.RabbitMQHandler',
'host': 'localhost',
'port': 5672,
'username': 'guest',
'password': 'guest',
'exchange': 'log',
'declare_exchange': False,
'connection_params': {
'virtual_host': '/',
'connection_attempts': 3,
'socket_timeout': 5000
},
'fields': {
'source': 'MainAPI',
'env': 'production'
},
'fields_under_root': True
}
},
'loggers': {
'myapp': {
'handlers': ['rabbit'],
'level': 'DEBUG',
'propagate': False
}
}
}

Custom formatter
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'standard': {
'format': '%(levelname)-8s [%(asctime)s]: %(message)s'
}
},
'handlers': {
'rabbit': {
'level': 'DEBUG',
'class': 'python_logging_rabbitmq.RabbitMQHandler',
'host': 'localhost',
'formatter': 'standard'
}
},
'loggers': {
'myapp': {
'handlers': ['rabbit'],
'level': 'DEBUG',
'propagate': False
}
}
}

JSON formatter
pip install python-json-logger

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'formatters': {
'json': {
'()': 'pythonjsonlogger.jsonlogger.JsonFormatter',
'fmt': '%(name)s %(levelname) %(asctime)s %(message)s'
}
},
'handlers': {
'rabbit': {
'level': 'DEBUG',
'class': 'python_logging_rabbitmq.RabbitMQHandler',
'host': 'localhost',
'formatter': 'json'
}
},
'loggers': {
'myapp': {
'handlers': ['rabbit'],
'level': 'DEBUG',
'propagate': False
}
}
}

Releases
CHANGELOG
Similar efforts

MQHandler (https://github.com/ziXiong/MQHandler)
http://stackoverflow.com/a/25479008/2716524
LogBook (http://pydoc.net/Python/Logbook/0.12.5/logbook.queues/)
https://pypi.python.org/pypi/py-amqp-logging/0.1