GitLocker: The Coding Marketplace

Description:

pypiemma 4.0.1

Emma

Emma Memory and Mapfile Analyser (Emma)

Conduct static (i.e. worst case) memory consumption analyses based on arbitrary linker map files. It produces extensive .csv files which are easy to filter and post-process. Optionally .html and markdown reports as well as neat figures help you visualising your results.

Given a map file input (Green Hills map files are the default but others - like GCC - are supported via configuration options; examples are enclosed) Emma maps the addresses of sections (aka images) and/or objects (aka modules) to memory regions (all addresses given via map files must be known during compile time). Those memory regions are classified into two levels of granularity respectively. The first level defines arbitrary groups based on your personal taste (however using names similar to those defined by your microcontroller vendor makes most sense). Later each of those regions (second level) are assigned to one of four generalised predefined memory regions (those are: INT_RAM, INT_FLASH, EXT_RAM, EXT_FLASH). In case of virtual memory objects and sections lying within virtual address spaces (VASes) get translated back into physical memory. This is depicted in the figure above (lower part).
Categorisation can be used to assign consumers (a consumer would usually represent a software component) to each object or section. This is useful for subsequent steps in order to display memory consumption per consumer type. See the upper part of the figure shown above. Mechanisms are provided to batch categorise object and section names. "Objects in sections" provides ways to obtain a finer granularity of the categorisation result. Therefore categorised sections containing (smaller) objects of a different category got split up and result into a more accurate categorisation.
As a result you will get output files in form of a .csv file which sets you up to do later processing on this data easily. In this file additional information is added like:

Overlaps (of sections/objects)
Containments (e.g. sections containing objects)
Duplicates
All meta data about the origin of each section/object (mapfile, addess space, ...)
...

Holding the aforementioned augmented data makes it easy to detect issues in linker scripts and get an in-depth understanding of your program's memory consumption. Including a lot of additional and "corrected" data can cause confusion. Thus all original (unmodified) data is preserved in the output files simultaneously.
The Emma visualiser helps you to create nice plots and reports in a .png and .html and markdown file format.
The whole Emma tool suite contains command line options making it convenient to be run on a build server like --Werror (treat all warnings as errors) or --no-prompt (exit and fail on user prompts; user prompts can happen when ambiguous configurations appear such as multiple matches for one configured map files).

Installation
pip3 install pypiemma

Dependencies: Python 3.6 or higher; pip3 install Pygments Markdown matplotlib pandas pypiscout

Optional: Cython
For bigger projects escpecially the number of objects will grow. We provide an optional Cython implementation which can speed-up your analysis (you will gain typically about **30 % speed-up**).
For now we do not provide the binaries with Emma, hence you have to compile (make sure a suitable compiler is installed) it yourself (don't worry it is quick and easy):
Install the Cython package (pip install Cython) and (in the Emma top level folder) execute (MSVC is recommended on Windows):
python setup.py build_ext --inplace --compiler=msvc

General Workflow
The following figure shows a possible workflow using Emma:

Emma - as the core component - produces an intermediate .csv file. Inputs are mapfiles and JSON files (for configuration (memory layout, sizes, ...)). From this point you are very flexible to choose your own pipeline. You could

use the Emma tools (Emma Visualiser, Emma Deltas, ...) for further processing (data aggregation and analysis),
simply your favourite spreadsheet software (like Microsoft Excel or LibreOffice Calc) or
use your own tool for the data analysis.

Quick Start Guide
At this point we want to give you a brief overview what to do in the below two scenarios. If you want to play around go to (project files are already present) and use our example projects in ./doc/test_project*.

if the Emma project is already set-up (JSON files were created) and you want to analyse your software with newly generated mapfiles proceed to -> Project files are already present or
you start your analysis from scratch and need to do configure Emma before you use it then go to -> Project files have to be created.

Example projects (including Emma* outputs/results) can be found in ./doc/test_project*.
Since version 3.1 Emma can be called in two ways (if you want to run it from the installation folder) where the following variant is recommended:
python Emma.py a --project doc/test_project --mapfiles doc/test_project/mapfiles --noprompt

The following table provides an overview how you call Emma:

Emma module
Entry point + <options> (if installed via pip)
Top level sub-command (tlsc) (python Emma.py <tlsc>)
Module (python -m + <module> <options>)

Analyser
emma
a
Emma.emma

Visualiser
emma_vis
v
Emma.emma_vis

Deltas
emma_deltas
d
Emma.emma_deltas

Project files are already present
Try python Emma.py a --help to see all possible options or refer to the documentation (./doc/*).

Create intermediate .csv from mapfiles with Emma:

python Emma.py a -p .\MyProjectFolder --map .\MyProjectFolder\mapfiles --dir .\MyProjectFolder\analysis --subdir Analysis_1

Generate reports and graphs with Emma Visualiser:

python Emma.py v -p .\MyProjectFolder --dir .\MyProjectFolder\analysis --subdir Analysis_1 -q

Project files that have to be created
To create a new project, the following files must be created:

globalConfig.json
budgets.json
categories.json
categoriesKeywords.json
categoriesSections.json
categoriesSectionsKeywords.json

You will find example projects in ./doc/test_project*. In-depth documentation can be found in the full documentation (see ./doc/).
A basic configuration can be short per file. For complex systems you can choose from many optional keywords/options that will provide you means to adjust your analysis as fine grained as you wish.
One main concept includes the globalConfig.json. You can see this as meta-config. Each configuration ID (configID) is a separately conducted analysis. Per configID you state individually the configuration files you want to use for this exact analysis. Herewith you can mix and match any combination of subconfigs you prefer.

A globalConfig.json could look like this:
{
"configID1": {
"addressSpacesPath": "addressSpaces.json",
"sectionsPath": "sections.json",
"patternsPath": "patterns.json"
}
}

Full documentation
For the full documentation please refer to the ./doc/ directory.
Contribute
We are glad if you want to participate. In ./doc/dev-guide.md you will find a guide telling you everything you need to know including coding conventions and more.
Mailing List
emma-dev (. at) googlegroups.com

Dependencies & Licences

Library (version)
pip package name
Licence
URL

Markdown (v3.0.1+)
Markdown
BSD-3-Clause
https://github.com/Python-Markdown/markdown; https://python-markdown.github.io/

Pandas (v0.23.4+)
pandas
BSD-3-Clause
https://github.com/pandas-dev/pandas/; http://pandas.pydata.org/getpandas.html

Pygments (v2.3.1+)
Pygments
BSD-2-Clause
https://bitbucket.org/birkenfeld/pygments-main/src/default/; http://pygments.org/download/

Matplotlib (v3.0.0+)
matplotlib
Matplotlib License (BSD compatible)
https://matplotlib.org/users/installing.html; https://github.com/matplotlib/matplotlib

SCout (v2.0+)
pypiscout
MIT
https://github.com/holzkohlengrill/SCout

svgwrite (v1.4+)
svgwrite
MIT License (MIT License)
https://github.com/mozman/svgwrite; https://svgwrite.readthedocs.io/en/latest/

Optional dependencies:
Utility scripts in ./doc/ need additional dependencies. As a normal user you can ignore this.

Library (version)
pip package name
Licence
URL

gprof2dot (v2017.9.19+)
gprof2dot
LGPL-3.0
https://github.com/jrfonseca/gprof2dot

pylint (v2.3.1+)
pylint
GPL-2.0
https://github.com/PyCQA/pylint

Cython (v0.29.13+)
Cython
Apache-2.0
https://cython.org/

Please refer to the gprof2dot project site and install its dependencies (this has to be done even if you install Emma via pip).
Note that those modules are invoked via subprocess calls within the ./genDoc/ scripts.
Dependencies used to generate documentation for GitHub pages (separate, independent branch gh-pages):
Utility scripts used to build GitHub pages documentation. As a normal user you can ignore this.

Library (version)
pip package name
Licence
URL

MkDocs (v1.0.4+)
mkdocs
BSD-3Clause
https://github.com/mkdocs/mkdocs

Material for MkDocs (v4.4.1+)
mkdocs-material
MIT
https://github.com/squidfunk/mkdocs-material

Code snippets etc.:

Name (version)
Kind
Modified?
Licence
URL

pygmentize (v2.2.0+)
Auto-generated .css file
Yes
BSD-2-Clause
http://pygments.org/download/; https://bitbucket.org/birkenfeld/pygments-main/issues/1496/question-licence-of-auto-generated-css

toHumanReadable (--)
Code snippet
No
MIT
https://github.com/TeamFlowerPower/kb/wiki/humanReadable

For the full documentation please refer to the ./doc/ directory.