Description:
openedxcalipertracking 0.14.3
Open edX Caliper Tracking can be used to transform the edX traditional event logs into Caliper Analytics Specifications provided by IMS Global. Generated logs can be consumed by any analytics application which is compatible with Caliper Standard. All the transformed logs are Caliper Certified.
Installation
To install openedx-caliper-tracking in your Open edX instance, please add the following line to your requirements file. (For most Open edX installations it should be located at edx-platform/requirements/edx/base.txt).
openedx-caliper-tracking==0.14.3
For manual installation:
pip install openedx-caliper-tracking
Configuration
To enable and use openedx-caliper-tracking:
Add ENABLE_EVENT_CALIPERIZATION flag under FEATURES in the following files:
lms.env.json
cms.env.json
These files should be located at /edx/app/edxapp/ directory, see the example below:
"FEATURES": {
...
"ENABLE_EVENT_CALIPERIZATION": true,
...
}
2. Add the following lines of code:
if FEATURES.get('ENABLE_EVENT_CALIPERIZATION'):
INSTALLED_APPS.insert(
INSTALLED_APPS.index('eventtracking.django.apps.EventTrackingConfig'),
'openedx_caliper_tracking'
)
in the following files:
lms/envs/production.py (aws.py for hawthorn release)
cms/envs/production.py (aws.py for hawthorn release)
Note:
Must make sure that after doing all the required changes in env (lms.env.json and cms.env.json) and auth (lms.auth.json and cms.auth.json) files you restart the server to reflect the changes that you have made.
Sending logs to external APIs (Optional)
There are two ways we can send caliper transformed logs to any external API
Rest API
Kafka
Using REST API
To do this, we have to add the following configurations
Add ENABLE_CALIPER_EVENTS_DELIVERY flag under FEATURES in the following files:
lms.env.json
cms.env.json
These files should be located at /edx/app/edxapp/ directory, see the example below:
"FEATURES": {
...
"ENABLE_CALIPER_EVENTS_DELIVERY": true,
...
}
Add the key CALIPER_DELIVERY_ENDPOINT and its value in the env files (lms.env.json and cms.env.json).
Add the key CALIPER_DELIVERY_AUTH_TOKEN and its value in the auth files ( lms.auth.json and cms.auth.json).
Add the following lines of code:
if FEATURES.get('ENABLE_CALIPER_EVENTS_DELIVERY'):
CALIPER_DELIVERY_ENDPOINT = ENV_TOKENS.get('CALIPER_DELIVERY_ENDPOINT')
CALIPER_DELIVERY_AUTH_TOKEN = AUTH_TOKENS.get('CALIPER_DELIVERY_AUTH_TOKEN')
in the following files:
lms/envs/production.py (aws.py for hawthorn release)
cms/envs/production.py (aws.py for hawthorn release)
Using Kafka Broker API
To do this, we have to add the following configurations
Add ENABLE_KAFKA_FOR_CALIPER flag under FEATURES in the following files:
lms.env.json
cms.env.json
These files should be located at /edx/app/edxapp/ directory, see the example below:
"FEATURES": {
...
"ENABLE_KAFKA_FOR_CALIPER": true,
...
}
2. Add the following keys and their values in the lms.env.json and cms.env.json files.
Please note that all parameters in the PRODUCER_CONFIG are unique to the broker instances. You
can set whatever parameters are required for your instance.
"CALIPER_KAFKA_SETTINGS": {
"PRODUCER_CONFIG": {
"bootstrap_servers": [
"<List of Kafka Brokers URLs>"
],
...
},
"TOPIC_NAME": "<Kafka Topic Name>",
"ERROR_REPORT_EMAILS": [
"<Reporting Email Address 1>",
"<Reporting Email Address 2>",
...
]
"MAXIMUM_RETRIES": <An Integer>
},
Keys
Description
MAXIMUM_RETRIES
Number of times the app will try to send the logs to Kafka in case of failure
PRODUCER_CONFIG
Configurations for initializing the Kafka Producer
Can further contain:
“bootstrap_servers”:
List of Kafka Brokers URLs
Any other supported paramter in the Kafka-python docs
Please note that it’s better to store the sensitive information in
the *.auth.json files
TOPIC_NAME
Topic name for the Kafka broker
ERROR_REPORT_EMAILS
Email Addresses to notify when number of failures exceeds the MAXIMUM_RETRIES
3. Add the following keys and their values in the lms.auth.json and cms.auth.json files.
Please note that all parameters in the PRODUCER_CONFIG are unique to the broker instances. You
can set whatever parameters are required for you.
"CALIPER_KAFKA_AUTH_SETTINGS": {
"PRODUCER_CONFIG": {
...
"sasl_plain_username": "<Username>",
"sasl_plain_password": "<Password>",
"security_protocol": "<Secuirty Protocol>",
"ssl_cafile": "<Path/to/the/ca/file>",
...
}
}
Keys
Description
PRODUCER_CONFIG
Configurations for initializing the Kafka Producer. Use this confiration to
store all sensitive configuration like authentication parameters.
For example:
Use this to configure paramters like:
sasl_plain_username
sasl_plain_password
security_protocol
sasl_mechanism
It can further contain:
Any other supported paramter in the Kafka-python docs
Please note that it’s better to store the insensitive information
in the *.env.json files
Add the following lines of code:
if FEATURES.get('ENABLE_KAFKA_FOR_CALIPER'):
CALIPER_KAFKA_SETTINGS = ENV_TOKENS.get('CALIPER_KAFKA_SETTINGS')
CALIPER_KAFKA_AUTH_SETTINGS = AUTH_TOKENS.get('CALIPER_KAFKA_AUTH_SETTINGS')
in the following files:
lms/envs/production.py (aws.py for hawthorn release)
cms/envs/production.py (aws.py for hawthorn release)
Location of Transformed Logs
Note: This doesn’t work locally.
Transformed events are logged using 'logging.handlers.SysLogHandler' with 'facility: local2'.
We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:
1. Create a log file with read/write permissions given to syslog user (e.g: /edx/var/log/caliper-analytics/caliper.log).
cd /edx/var/log
mkdir -p caliper-analytics && cd caliper-analytics
touch caliper.log
chown syslog caliper.log
Create a mapping for 'local2' in the configuration files present in /etc/rsyslog.d/ (e.g: in 99-edx.conf).
local2.* /edx/var/log/caliper-analytics/caliper.log;tracking
Run the following command on server to restart the rsyslog daemon:
sudo service rsyslog restart
Location of Logs Whose Delivery to Kafka is failed
Note:
This doesn’t work locally. Do this only if you are sending logs to external source using Kafka broker API.
Transformed events are logged using 'logging.handlers.SysLogHandler' with 'facility: local3'.
We need to create output files manually and set appropriate permissions for syslog user. To do so, please follow the steps below:
1. Create a log file with read/write permissions given to syslog user (e.g: /edx/var/log/caliper-analytics/delivery_failure.log).
cd /edx/var/log
mkdir -p caliper-analytics && cd caliper-analytics
touch delivery_failure.log
chown syslog delivery_failure.log
Create a mapping for 'local3' in the configuration files present in /etc/rsyslog.d/ (e.g: in 99-edx.conf).
local3.* /edx/var/log/caliper-analytics/delivery_failure.log;tracking
Run the following command on server to restart the rsyslog daemon:
sudo service rsyslog restart
Running Tests Locally
To run the unit tests of this app locally, follow the following steps:
Clone the repository
git clone git@github.com:ucsd-ets/openedx-caliper-tracking.git
Run the following command in the same directory in which you have cloned the repository
sudo ./openedx-caliper-tracking/openedx_caliper_tracking/tests/local_test_script.sh
License
The code in this repository is licensed under the GPL v3.0 unless otherwise noted. Please see LICENSE for details.
How To Contribute
To contribute, please make a pull request in the repository on Github . If you have any questions or issues, please feel free to open an issue on Github: Open edX Caliper Tracking.
Contributors
Muhammad Zeeshan
Tasawer Nawaz
Aroosha Arif
Osama Arshad
Danial Malik
Hamza Farooq
Hassan Tariq
Muhammad Umar Khan
Tehreem Sadat
Muhammad Arslan
Saad Ali
Husnain Raza Ghaffar
License
For personal and professional use. You cannot resell or redistribute these repositories in their original state.
Customer Reviews
There are no reviews.
