pedlar 0.0.8

Description:

pedlar 0.0.8

:chart_with_upwards_trend: pedlar
Pedlar is an algorithmic trading platform for Python designed for trading events, competitions and sessions such as Algothons. It includes a live web interface with multiple accounts with account sharing and live chat, an HTTP API with example Python trading agents and a ZeroMQ based broker connection to MetaTrader5.

Getting Started
If there is already a ticker server and web server running, the client API under pedlar can be used. If not, follow instructions on how to get them in the Hosting section. The client or agent API resolves around connecting to the ticker server (where agents receive price updates) and the web server, in this case handles the broker connection. The ticker is separate to reduce overhead and latencies between making trades and just receiving price updates.
Installation
The client API is can be installed using:
pip3 install --no-cache-dir -U pedlar

Usage
There are some helpful examples in the pedlar folder. Here is a overview of the client API:
from pedlar.agent import Agent

class MyAgent(Agent):
"""A trading agent."""
def on_order(self, order):
"""Called on placing a new order."""
print("New order:", order)
print("Orders:", self.orders) # Agent orders only

def on_order_close(self, order, profit):
"""Called on closing an order with some profit."""
print("Order closed", order, profit)
print("Current balance:", self.balance) # Agent balance only

def on_tick(self, bid, ask, time=None):
"""Called on every tick update."""
print("Tick:", bid, ask, time)
# self.buy()
# self.sell()
# self.close()

def on_bar(self, bopen, bhigh, blow, bclose, time=None):
"""Called on every bar update."""
print("Bar:", bopen, bhigh, blow, bclose, time)

if __name__ == "__main__":
import logging
logging.basicConfig(level=logging.DEBUG)
agent = MyAgent.from_args()
agent.run()

The extra parameters are parsed from the command line and can be run using:
python3 -u myagent.py -h

Key things to keep in mind:

Every agent takes control of its own orders and balance, it is not synced across agents of a shared account. This setup is to keep agents isolated.
Agents try to close orders when they are quit, if hard stopped or an error occurs an open orphan order might remain. In this case, one option would be manually invoke self.close with the stale order id or simply reset the account.
The ticker connection receives from ZeroMQ whereas the trade requests are made via HTTP. There might some ticks dropped if the trade request takes too long.

Basic Backtesting
The agents can backtest against a CSV file of the following format, last column time is optional and time format is adjustable through time_format of the Agent class:
tick,1.26361,1.26375,2019.01.03 23:44:42
tick,1.2636,1.26374,2019.01.03 23:44:59
tick,1.2636,1.26378,2019.01.03 23:45:00
bar,1.26386,1.26398,1.26355,1.2636,2019.01.03 23:45:00
tick,1.26359,1.26377,2019.01.03 23:45:02
tick,1.26357,1.26375,2019.01.03 23:45:05
tick,1.26356,1.26374,2019.01.03 23:45:07
tick,1.26358,1.26376,2019.01.03 23:45:10

which can be used with an agent python3 myagent.py -b ticks.csv. Essentially each line will invoke corresponding on_tick or on_bar function. The actual profit and trade results are computed offline based on absolute price differences. This means the agent will run completely offline and the actual results are only useful to get an idea about the performance or train a neural network. Any extra broker commissions beyond bid-ask spread, price requotes etc are not factored.
Hosting
Pedlar involves 4 components that talk to each other to create a platform for agents to trade:

ticker: is a ZeroMQ PUB socket that publishes tick and bar updates that are received by the agents which eventually trigger on_tick and on_bar methods. This component runs independently from others to provide a continuous stream of price updates.
broker: is the component that actually executes live order requests on the market. Pedlar is broker agnostic and all it needs is methods to buy, sell and close an order with unique order ids.
pedlarweb: is the main web server that handles user accounts, live leaderboard etc updates and agent trade requests.
pedlar: is the client API that allows for Python based clients to receive updates from ticker and make trade requests to pedlarweb.

Repository Structure
The client and server packages are separated to avoid any assumptions between their implementations. The main folders include:

mt5: contains the MetaTrader 5 trading components that act as the ticker and the broker for live trading.
pedlar: contains the client API is the exposed API for building new agents.
pedlarweb: is the web server that bridges agent trade requests and broker while providing a web user interface and live updates.

Prerequisites
All the extra packages required can be installed using:
pip3 install --no-cache-dir -U -r requirements.txt

Running MT5
To get the ticker and broker components up and running you need to:

Install MetaTrader 5, or some existing installation would work.
Install ZeroMQ for Windows since the scripts require the libzmq.dll to run. You might need to install a compatible Visual C++ runtime to get ZeroMQ running.
Find the installed ZeroMQ bin folder and move the DLL to Library\libzmq.dll as that is where the libzmq.mqh header expects it.
Copy files MetaTrader 5 files to the expected folders of your installation. An easy way to find that is to open the MetaEditor (F4 from trader) and right click on a folder to select Open Folder. For development it is easier to write a script that syncs the repo folder with the expected folders.
Once the ticker and broker is compiled, you can attach them to any chart that you want the tick updates to be sent and trade orders to be handled. You can run multiple at different ports as well. From "Options" the "Allow DLL Imports" needs to be enabled for obvious reasons.
If you get DLL import error, it is most likely because the ZeroMQ DLL isn't happy or the C++ runtime is not compatible.

Running Local Broker
As an alternative to executing trades live, there is lbroker.py that implements same interface as the actual MT5 broker but instead executes the orders locally based on the latest ticks provided by MT5. It is a simple stand-alone script:
python3 lbroker.py -h

Running Web Server
The web server is a standard Flask application organised into the pedlarweb package. You need to create a instance/config.py to customise the default values. Once the config.py options are as desired, a database can be initialised:
mkdir instance && cp config.py instance/config.py # customise instance/config.py ex. database settings
python3 -c "from pedlarweb import db; db.create_all()"

If the in-memory default database is used, tables will be automatically created but data is lost when server is stopped using an in-memory database. Then the server can be run using standard Flask options:
python3 runpedlarweb.py -h

usage: runpedlarweb.py [-h] [-d] [--host HOST] [--port PORT]

Run pedlarweb.

optional arguments:
-h, --help show this help message and exit
-d, --debug Enable debug server.
--host HOST Host IP address.
--port PORT Host port.

Due to Flask-SocketIO the eventlet server would be run. For convinience, a new user is created if none with the username exist from the login page. This choice is done to get people on-board as easy as possible without heavy registration and email confirmation schemes.
FAQ

Why ticker is separated from the web server? This design choice is done to reduce overhead and latency in receiving price updates for agents. As a result agents need to connect to both the ticker and the web server to function unless backtesting.
Why not use ZeroMQ for trade requests instead of HTTP? HTTP APIs provide a more approachable and unified way of providing a service. HTTP does have more overhead but it makes it easier for other non-pedlar clients that talk to pedlarweb to be built as well. Finally, it has an authentication mechanism that is enforced for every request.
Why is the source code 2 space indented? The answer is a combination of personal style and to stop direct copy-paste from other resources. The code is linted using PyLint although there are cases it is disabled on purpose.

Limitations & To-Dos

Currently the API supports trading only on a single instrument mainly because the web server can talk to a single broker and a single broker can only be attached to single chart. The MT5 broker is fixed to the current chart Symbol() but there is space for passing the symbol, instrument from agent -> server -> broker. For the target audience of live sessions and competitions, a single instrument seems enough.
Although polling is employed for ZeroMQ sockets, they are still blocking in nature to avoid data loss. In that case, an agent can look like it has frozen against Ctrl+C. This case also occurs for MT5 ticker and broker scripts. The timeout on the polls can be adjust or a little patience helps.
There is a mixture of line endings due MT5 running on Windows and development is somewhat split between Windows and Linux environments. Hopefully, we live in a day and age this is not a problem anymore.

Built With

Flask - pedlarweb web framework
ZeroMQ - ticker and broker messaging framework
MetaTrader5 - as the default trading platform
SocketIO - server client messaging for dynamic updates
Vue.JS - dynamic front-end rendering
W3.CSS - front-end CSS library