temci 0.8.5

Description:

temci 0.8.5

An advanced benchmarking tool written in Python 3 that supports
setting up an environment for benchmarking
and the generation of visually appealing reports.
It runs on Linux systems and (rudimentarily) on macOS.

Why should you use temci?
temci allows you to easily measure the execution time (and other things)
of programs and compare them against each other resulting in a pretty
HTML5 based report. Furthermore it can set up the environment to ensure
benchmarking results with a low variance. The latter feature can be used
without using temci for benchmarking
by using temci short shell.


Usage
The main commands of temci are temci exec and
temci report.
Suppose you want to see whether grepping for the strings that consist of a and b in the current
folder is slower than for strings that consist only of a.
First we have to install temci (using Nix, see below for more instructions):
nix-env -f https://github.com/parttimenerd/temci/archive/master.tar.gz -i
After this, we can benchmark both commands with temci:
# benchmark both commands 20 times
temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10

# append --watch to get report (in which you can move with the arrow keys and scroll)
# after every benchmark completed (use --watch_every to decrease interval)
temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10 --watch

# if you want to improve the stability your benchmarks, run them with root privileges
# the benchmarked programs are run with your current privileges
temci short exec "grep '[ab]*' -R ." "grep 'a*' -R ." --runs 10 --sudo --preset usable
This results in a run_output.yaml file that should look like:
- attributes: {description: 'grep ''[ab]*'' -R .'}
data:
etime: [0.03, 0.02, 0.02, 0.03, 0.03, 0.03, 0.02, 0.03, 0.03, 0.02]
… # other properties
- attributes: {description: grep 'a*' -R .}
data:
etime: [0.02, 0.03, 0.02, 0.03, 0.03, 0.02, 0.03, 0.03, 0.02, 0.02]
… # other properties
- property_descriptions: {etime: elapsed real (wall clock) time, … }
For more information on the support measurement tools (like
perf stat and
rusage),
the supported plugins for setting up the environment
and more, see temci exec.
We can now create a report from these benchmarking results using
temci report.
We use the option --properties to include only the elapsed time in the
report to keep the report simple:
> temci report run_output.yaml --properties etime
Report for single runs
grep '[ab]*' -R . ( 10 single benchmarks)
etime mean = 2(6).(000)m, deviation = 18.84223%

grep 'a*' -R . ( 10 single benchmarks)
etime mean = 2(5).(000)m, deviation = 20.00000%

Equal program blocks
grep '[ab]*' -R . ⟷ grep 'a*' -R .
etime confidence = 67%, speed up = 3.85%
We see that there is no significant difference between the two commands.
There are multiple reporters besides the default
console reporter.
Another reporter is the html2 reporter
that produces an HTML report, use it by adding the --reporter html2 option:



Installation
The simplest way is to use the Nix package manager, after installing Nix, run:
nix-env -f https://github.com/parttimenerd/temci/archive/master.tar.gz -i
Using pip requiring at least Python 3.6:
sudo pip3 install temci
For more information see the Installation page.

Auto completion
Temci can generate auto completion files for bash and zsh. Add the following line to your .bashrc or .zshrc:
. `temci_completion $0`



Using temci to set up a benchmarking environment
Use the temci short shell COMMAND to run a command (sh by default) in a shell that is inside
the benchmarking environment. Most options of temci short exec are supported.
For more information, see temci shell.


Why is temci called temci?
The problem in naming programs is that most good program names are
already taken. A good program or project name has (in my opinion) the
following properties:

it shouldn’t be used on the relevant platforms (in this case: github and pypi)
it should be short (no one wants to type long program names)
it should be pronounceable
it should have at least something to do with the program

temci is such a name. It’s lojban for time (i.e. the time duration between two moments or events).


Contributing
Bug reports and
code contributions are highly appreciated.
For more information, see the Contributing page.