BanzaiDB 0.3.0

Description:

BanzaiDB 0.3.0

(Using landscape.io and drone.io)

News
API changes (v1 -> v2 -> v3 -> v?). The population of a mapping run
into BanzaiDB was dependent on a nesoni run. Originally we (API v1) parsed
the reports.txt for each strain. In API v2 we parse the nway.any (assumes you
have ran nesoni nway). BanzaiDB API v3 assumes that you still have accessto
the consensus.fa (called a consensus). We need this data to store information
in BanzaiDB about coverage.


What is BanzaiDB?
Please use the releases (https://github.com/mscook/BanzaiDB/releases). All
versions including most recent made some significant assumptions that are
currently being improved.
BanzaiDB is a tool for pairing Microbial Genomics Next Generation Sequencing
(NGS) analysis with a NoSQL database. We use the RethinkDB NoSQL database.

BanzaiDB:

initialises the NoSQL database and associated tables,
populates the database with results of NGS experiments/analysis and,
provides a set of query functions to wrangle with the data stored within
the database.





Why BanzaiDB?
The analysis (primary/secondary/tertiary) of large collections of draft
microbial genomes from NGS typically generates many separate flat files.

The bioinformatician will:

write scripts to parse and extract the important information from
the results files (often trying to standardise the output from
similar programs),
store these results in further flat files,
write scripts to link the results of one analysis step to another,
store these results in further flat files,
modify scripts as hypothesis is improved as a direct consequence of
incorporating the knowledge from the previous steps,
…
…
…
end up with thousands of flat files, many scripts and generally get
confused as to how and where everything came from.



The idea around BanzaiDB is to run once, store once analyse many times.


About BanzaiDB
BanzaiDB is geared towards outputs of Bioinformatics software employed by
the Banzai NGS pipeline.

BanzaiDB is thus geared towards handling data generated from:

Velvet and SPAdes (assembly),
BWA and Nesoni (mapping/variant calling),
Mugsy (whole genome alignment),
BRATTNextGen (recombination detection) and,
Prokka (annotation).



The present focus is on storing and manipulating the results of SNP and
recombination analysis.
Banzai is not a stable API.
See the ReadTheDocs site for BanzaiDB documentation (User & API).


About RethinkDB

We choose RethinkDB as our underlying database for a few reasons:

RethinkDB is both developer and operations friendly. This sits well with
the typical bioinformatician,
NoSQL databases allow for a flexible schema. We can store/collect now,
think later. This is much like how science is performed.
Not every bioinformatician or lab has a system administrator. RethinkDB
is easy to setup and administer
We don’t know how big our complex our datasets could get in the future.
It is easy to scale RethinkDB into a cluster.
ReQL the underlying query language is nice and simple to
learn/understand. We’re also very comfortable with Python and the
availability of official python drivers (also JavaScript & Ruby, and a
heap for user contributed for a swag of languages) is a big bonus.





BanzaiDB Requirements

You will need:

(probably) administrator access to your machine(s)
a RethinkDB server/instance. This can be running locally or on a VPS,
git (to clone this repository) and
pip
bedtools, samtools and tabix (for pybedtools)


You will also need a few Python modules:

rethinkdb
biopython
reportlab
fabric
tablib
argparse (if Python 2.6)
pybedtools (you will probably also need to install cython)



The Python modules should/will be pulled down automatically when installing
BanzaiDB.
We recommend you increase the rethinkdb python driver performance. We have
found that in some cases the installation of C++ backend fails. We provide
a simple protocol that we have found works.


BanzaiDB Installation
Something like this:
$ git clone https://github.com/mscook/BanzaiDB.git
$ cd BanzaiDB
$ python setup.py install


Getting BanzaiDB talking to RethinkDB
You provide information about you RethinkDB instance and database using the
file ~/.BanzaiDB.cfg (~/ is shorthand for $HOME).
The configuration file supports:
db_host = [def = localhost]
port = [def = 28015]
db_name = [def = Banzai]
auth_key = [def = '']


BanzaiDB usage
Note: Please refer to the BanzaiDB documentation (via ReadTheDocs) for
more detailed information (under active development).
Once both RethinkDB and BanzaiDB are installed and the configuration is set:
$ python BanzaiDB.py -h
usage: BanzaiDB.py [-h] [-v] {init,populate,update,query} ...

BanzaiDB v 0.3.0 - Database for Banzai NGS pipeline tool
(http://github.com/mscook/BanzaiDB)

positional arguments:
{init,populate,update,query}
Available commands:
init Initialise a DB
populate Populates a database with results of an experiment
update Updates a database with results from a new experiment
query List available or provide database query functions

optional arguments:
-h, --help show this help message and exit
-v, --verbose verbose output

Licence: ECL 2.0 by Mitchell Stanton-Cook <[email protected]>