pyjls 0.10.0

Description:

pyjls 0.10.0

JLS
Welcome to the Joulescope® File Format project.
The goal of this project is to provide performant data storage for huge,
simultaneous, one-dimensional signals. This repository contains:

The JLS file format specification
The implementation in C
Language bindings for Python

Features

Cross-platform

Microsoft Windows x64
Apple macOS x64
Apple macOS ARM
Linux x64
Linux aarch64 (ARM 64-bit). Supports Raspberry Pi 4.


Support for multiple, simultaneous data sources
Support for multiple, simultaneous signal waveforms
Fixed sample rate signals (FSR)

Handles missing samples gracefully (interpolate) 🔜
Multiple data types including:

Floating point: f32, f64
Unsigned integers: u1, u4, u8, u16, u24, u32, u64
Signed integers: i4, i8, i16, i24, i32, i64
Fixed-point, signed integers (same bit sizes as signed integers)
Boolean (digital) 1-bit signals = u1




Variable sample rate (VSR) signals 🔜
Fast read performance

Signal Summaries

"Zoomed out" view with mean, min, max, standard deviation
Provides fast waveform load without any additional processing steps


Automatic load by summary level
Fast seek, next, previous access


Sample ID to Wall-clock time (UTC) for FSR signals
Annotations

Global VSR annotations
Signal annotations, timestamped to sample_id for FSR and UTC time for VSR
Support for text, marker, and user-defined (text, binary, JSON)


User data

Arbitrary data included in the same file
Support for text, binary, and JSON


Reliability

Integrated integrity checks using CRC32C
File data still accessible in the case of improper program termination 🔜
Uncorrupted data is still accessible in presence of file corruption 🔜
Write once, except for indices and the doubly-linked list pointers


Compression options 🔜

lossless 🔜
lossy 🔜
lossy with downsampling below threshold 🔜
Support level 0 DATA not written (only INDEX & SUMMARY) 🔜



Items marked with 🔜 are under development and coming soon.
Items marked with ⏳ are planned for future release.
As of June 2023, the JLS v2 file structure is well-defined and stable.
However, the compression storage formats are not yet defined and
corrupted file recovery is not yet implemented.
Why JLS?
The world is already full of file formats, and we would rather not create
another one. However, we could not identify a solution that met these
requirements. HDF5 meets the
large storage requirements, but not the reliability and rapid load requirements.
The Saleae binary export file format v2
is also not suitable since it buffers stores single, contiguous blocks.
Sigrok v2 is similar.
The Sigrok v3 format
(under development as of June 2023) is better in that it stores sequences of
"packets" containing data blocks, but it still will does not allow for
fast seek or summaries.
Timeseries databases, such as InfluxDB, are
powerful tools. However, they are not well-designed for fast sample-rate
data.
Media containers are another option, especially the ISO base media file format
used by MPEG4 and many others:

ISO/IEC 14496-14:2020 Specification
Overview

However, the standard does not include the ability to store the signal summaries
and our specific signal types. While we could add these features, these formats
are already complicated, greatly reducing the advantage of repurposing them.
Why JLS v2?
This file format is based upon JLS v1 designed for
pyjoulescope and used by the
Joulescope test instrument. We leveraged
the lessons learned from v1 to make v2 better, faster, and more extensible.
The JLS v1 format has been great for the Joulescope ecosystem and has
accomplished the objective of long data captures (days) with fast
sampling rates (MHz). However, it now has a long list of issues including:

Inflexible storage format (always current, voltage, power, current range, GPI0, GPI1).
Unable to store from multiple sources.
Unable to store other sources and signals.
No annotation support:
41,
93.
Inflexible user data support.
Inconsistent performance across sampling rates, zoom levels, and file sizes:
48,
103.
Unable to correlate sample times with UTC:
55.

The JLS v2 file format addressed all of these issues, dramatically
improved performance, and added new capabilities, such as signal compression.
How?
At its lowest layer, JLS is an enhanced
tag-length-value (TLV)
format. TLV files form the foundation of many reliable image and video formats,
including MPEG4 and PNG. The enhanced header contains additional fields
to speed navigation and improve reliability. The JLS file format calls
each TLV a chunk. The enhanced tag-length component the chunk header
or simply header. The file also contains a file header, not to be
confused with the chunk header. A chunk may have zero payload length,
in which case the next header follows immediately. Otherwise, a
chunk consists of a header followed by a payload.
The JLS file format defines sources that produce data. The file allows
the application to clearly define and label the source. Each source
can have any number of associated signals.
Signals are 1-D sequences of values over time consisting of a single,
fixed data type. Each signal can have multiple tracks that contain
data associated with that signal. The JLS file supports two signal types:
fixed sample rate (FSR) and variable sample rate (VSR). FSR signals
store their sample data in the FSR track using FSR_DATA and FSR_SUMMARY.
FSR time is denoted by samples using timestamp. FSR signals also support:

Sample time to UTC time mapping using the UTC track.
Annotations with the ANNOTATION track.

VSR signals store their sample data in the VSR track. VSR signals
specify time in UTC (wall-clock time). VSR signals also
support annotations with the ANNOTATION track.
The JLS file format supports VSR signals that only use the
ANNOTATION track and not the VSR track. Such signals are commonly
used to store UART text data where each line contains a UTC timestamp.
Signals support DATA chunks and SUMMARY chunks.
The DATA chunks store the actual sample data. The SUMMARY chunks
store the reduced statistics, where each statistic entry represents
multiple samples. FSR tracks store the mean, min, max,
and standard deviation. Although standard deviation requires the
writer to compute the square root, standard deviation keeps the
same units and bit depth requirements as the other fields. Variance
requires twice the bit size for integer types since it is squared.
Before each SUMMARY chunk, the JLS file will contain the INDEX chunk
which contains the starting time and offset for each chunk that
contributed to the summary. This SUMMARY chunk enables fast O(log n)
navigation of the file. For FSR tracks, the starting time is
calculated rather than stored for each entry.
The JLS file format design supports SUMMARY of SUMMARY. It supports
the DATA and up to 15 layers of SUMMARIES. timestamp is given as a
64-bit integer, which allows each summary to include only 20 samples
and still support the full 64-bit integer timestamp space. In practice, the
first level summary increases a single value to 4 values, so summary
steps are usually 50 or more.
Many applications, including the Joulescope UI, prioritize read performance,
especially visualizing the waveform quickly following open,
over write performance. Waiting to scan through a 1 TB file is not a
valid option. The reader opens the file and scans for sources and signals.
The application can then quickly load the highest summary of summaries
for every signal of interest. The application can very quickly display this
data, and then start to retrieve more detailed information as requested.
Example file structure
sof
header
USER_DATA(0, NULL) // Required, point to first real user_data chunk
SOURCE_DEF(0) // Required, internal, reserved for global annotations
SIGNAL_DEF(0, 0.VSR) // Required, internal, reserved for global annotations
TRACK_DEF(0.VSR)
TRACK_HEAD(0.VSR)
TRACK_DEF(0.ANNO)
TRACK_HEAD(0.ANNO)
SOURCE_DEF(1) // input device 1
SIGNAL_DEF(1, 1, FSR) // our signal, like "current" or "voltage"
TRACK_DEF(1.FSR)
TRACK_HEAD(1.FSR)
TRACK_DEF(1.ANNO)
TRACK_HEAD(1.ANNO)
TRACK_DEF(1.UTC)
TRACK_HEAD(1.UTC)
USER_DATA // just because
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_INDEX(1.FSR, lvl=1)
TRACK_SUMMARY(1.FSR, lvl=2)
USER_DATA // just because
END
eof

Note that TRACK_HEAD(1.FSR) points to the first TRACK_INDEX(1.FSR, lvl=0) and
TRACK_INDEX(1.FSR, lvl=1).
Each TRACK_DATA(1.FSR) is in a doubly-linked list with its next and previous
neighbors. Each TRACK_INDEX(1.FSR, lvl=0) is likewise in a separate doubly-linked
list, and the payload of each TRACK_INDEX points to the summarized TRACK_DATA
instances. TRACK_INDEX(1.FSR, lvl=1) points to each TRACK_INDEX(1.FSR, lvl=0) instance.
As more data is added, the TRACK_INDEX(1.FSR, lvl=1) will also get added to
the INDEX chunks at the same level.
Resources

source code
documentation
pypi
Joulescope (Joulescope web store)
forum

References

JLS v1:
lower-layer,
upper-layer.
Sigrok/v3, which shares
many of the same motivations.
Tag-length-value: Wikipedia.
Doubly linked list: Wikipedia.

License
This project is Copyright © 2017-2023 Jetperch LLC and licensed under the
permissive Apache 2.0 License.