opensbt 0.1.4

Description:

opensbt 0.1.4

OpenSBT - A Modular Framework for Search-based Testing of Automated Driving Systems
Intro
OpenSBT provides a modular and extandable code base to facilitate search-based testing of automated driving systems. It provides interfaces to integrate search algorithms, fitness/criticality functions and simulation environments in a modular way. It allows to visualize testing results and analyse the critical behaviour of the tested system.
An introductory video of OpenSBT can be found here: https://www.youtube.com/watch?v=qi_CTTzrk5s.
Overview

OpenSBT builds upon Pymoo 0.6.0 and extends internal optimization related models, such as Problem, Result, to tailor heuristic search algorithms for testing ADS systems.
Installation
OpenSBT requires Python to be installed and its compatibality has been tested with Python 3.8. OpenSBT can be run as a standalone application or can be imported as a library.
Installation instructions are available in the following jupyter notebook.
Usage
You can find several tutorials as jupyter notebooks which explain step-by-step of how to use OpenSBT. In these tutorials, we have integrated:

A simplified SUT simulated in very simplistic simulator (linear motion planning, no GPU required)
A real-world FMI-based AEB agent developed in the fortiss Mobility lab which is simulated in CARLA using the simulator adapter CARLA Runner Extension.

Note: We have also implemented a simulator adapter for testing Simulink-based SUTs in Prescan.
Output
OpenSBT produces several result artefacts. All artefacts are written into the results folder in a folder named as the problem name.
OpenSBT generates the following outputs:



Type
Description
Example




Design Space Plot
Visualization of all evaluated test cases in the input space + of predicted critical regions using the decision tree algorithm, pairwise. Constraints of derived regions are stored in CSV file bounds_regions.csv and the learned tree in tree.pdf



Scenario 2D Visualization
Visualization of traces of the ego vehicle and adversaries in a two-dimensional GIF animation



Objective Space Plot
Visualization of fitness values of evaluated test cases, pairwise.



All Testcases
CSV file of all test inputs of all evaluated testcases
all_testcases.csv


All Critical Testcases
CSV file of all critical test inputs of all evaluated testcases
all_critical_testcases.csv


Calculation Properties
CSV file of all experiment configuration parameters (e.g. algorithm parameters, such as population size, number iterations; search space, fitness function etc..).
calculation_properties.csv


Evaluation Results
CSV file containing performance values of the algorithm, e.g., number critical test cases found in relation to all evaluations, execution time.
summary_results.csv



Application Use Cases
OpenSBT has been already applied:

For a replication experiment to replicate the results of a surrogate assisted testing technique, s. here.
In an industrial case study to validate a systems behaviour for different operating scenarios, s. here and here: https://doi.org/10.1007/978-3-031-46002-9_15

Contribution
If you like to contribute please contact one of the developers listed below or create a pull request. If you face any issue with OpenSBT feel free to create an issue or contact the developers.
Acknowledgements
OpenSBT has been developed by Lev Sorokin, Tiziano Munaro and Damir Safin within the
FOCETA Project. Special thanks go to Brian Hsuan-Cheng Liao and Adam Molin from DENSO AUTOMOTIVE Deutschland GmbH for their valuable feedback and evaluation of OpenSBT on the AVP Case Study in the Prescan simulator.
Reference
If you use or extend OpenSBT please cite our framework. Here is an example BibTeX entry:
@misc{sorokin2023opensbt,
title={OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems},
author={Lev Sorokin and Tiziano Munaro and Damir Safin and Brian Hsuan-Cheng Liao and Adam Molin},
year={2023},
eprint={2306.10296},
archivePrefix={arXiv},
primaryClass={cs.SE}
}

License
OpenSBT is licensed under the Apache License, Version 2.0.

License

For personal and professional use. You cannot resell or redistribute these repositories in their original state.

Share

• Share on Facebook
• Share on Twitter