search-data-collector 0.6.1

Description:

searchdatacollector 0.6.1

Search Data Collector











Thread-safe and atomic collection of tabular data into csv-files.


Introduction
The search-data-collector provides a single class with following methods to manage data:

save
append
load
remove

The Search-Data-Collector was created as a utility function for the Gradient-Free-Optimizers- and Hyperactive-package. It is intended to be used as a tool to collect search-data from the optimization run. The search-data can be collected during the optimization run as a dictionary via append or after the run as a dataframe with the save-method.
The append-method is thread-safe to work with hyperactive-multiprocessing. The save-method is atomic to avoid accidental data-loss, when interupting the save-process.
For the Hyperactive-package the search-data-collector handles functions in the data by converting them to strings. If the data is loaded you can pass the search-space to convert the strings back to functions.

Disclaimer
This project is in an early development stage and is sparsely tested. If you encounter bugs or have suggestions for improvements, then please open an issue.

Installation
pip install search-data-collector


Examples

Append search-data
import numpy as np
from hyperactive import Hyperactive
from search_data_collector import CsvSearchData

collector = CsvSearchData("./search_data.csv") # the csv is created automatically


def parabola_function(para):
loss = para["x"] * para["x"] + para["y"] * para["y"]

data_dict = dict(para) # copy the parameter dictionary
data_dict["score"] = -loss # add the score to the dictionary
collector.append(data_dict) # you can append a dictionary to the csv

return -loss


search_space = {
"x": list(np.arange(-10, 10, 0.1)),
"y": list(np.arange(-10, 10, 0.1)),
}


hyper = Hyperactive()
hyper.add_search(parabola_function, search_space, n_iter=1000)
hyper.run()
search_data = hyper.search_data(parabola_function)

search_data = collector.load(search_space) # load data

print("\n search_data \n", search_data)


Save search-data
import numpy as np
from hyperactive import Hyperactive
from search_data_collector import CsvSearchData

collector = CsvSearchData("./search_data.csv") # the csv is created automatically


def parabola_function(para):
loss = para["x"] * para["x"] + para["y"] * para["y"]

return -loss


search_space = {
"x": list(np.arange(-10, 10, 0.1)),
"y": list(np.arange(-10, 10, 0.1)),
}


hyper = Hyperactive()
hyper.add_search(parabola_function, search_space, n_iter=1000)
hyper.run()
search_data = hyper.search_data(parabola_function)

collector.save(search_data) # save a dataframe instead

search_data = collector.load(search_space) # load data

print("\n search_data \n", search_data)


Functions in the search-space/search-data
import numpy as np
from hyperactive import Hyperactive
from search_data_collector import CsvSearchData

collector = CsvSearchData("./search_data.csv") # the csv is created automatically


def parabola_function(para):
loss = para["x"] * para["x"] + para["y"] * para["y"]

return -loss


# just some dummy functions to show how this works


def function1():
print("this is function1")


def function2():
print("this is function2")


def function3():
print("this is function3")


search_space = {
"x": list(np.arange(-10, 10, 0.1)),
"y": list(np.arange(-10, 10, 0.1)),
"string.example": ["string1", "string2", "string3"],
"function.example": [function1, function2, function3],
}


hyper = Hyperactive()
hyper.add_search(parabola_function, search_space, n_iter=30)
hyper.run()
search_data = hyper.search_data(parabola_function)

collector.save(search_data) # save a dataframe instead of appending a dictionary

search_data = collector.load() # load data

print(
"\n In this dataframe the 'function.example'-column contains strings, which are the '__name__' of the functions. \n search_data \n ",
search_data,
"\n",
)

search_data = collector.load(search_space) # load data with search-space

print(
print(
"\n In this dataframe the 'function.example'-column contains the functions again. \n search_data \n ",
search_data,
"\n",
)
)