osculari 0.0.4

Description:

osculari 0.0.4

Exploring artificial neural networks with psychophysical experiments.
Overview
The osculari package provides an easy interface for different techniques to explore and interpret
the internal presentation of deep neural networks.

Supporting the following pretrained models:

All classification and segmentation networks
from PyTorch's official website.
All OpenAI CLIP language-vision models.
All Taskonomy networks.


Managing convolution and transformer architectures.
Allowing to readout the network at any given depth.
Training a linear classifier on top of the extract features from any network/layer.
Experimenting with 2AFC and 4AFC paradigms.

At a granular level, Kornia is a library that consists of the following components:



Module
Description




osculari
Open source library to explore and interpret pretrained deep neural networks.


osculari.datasets
A module to create datasets and dataloaders to train and test linear probes.


osculari.models
A module to readout pretrained networks and add linear layers on top of them.


osculari.paradigms
A module to implement psychophysical paradigms to experiment with deep networks.



Installation
From pip
pip install osculari


Alternative installation options
From source with symbolic links:
pip install -e .

From source using pip:
pip install git+https://github.com/ArashAkbarinia/osculari


Examples
Please check the example page of our
documentation with many notebooks that can also be executed on Google Colab.

Quick start
Pretrained features
Let's create a linear classifier on top of the extracted features from a pretrained network to
perform a binary classification task (i.e., 2AFC – two-alternative-force-choice). This is easily
achieved by calling the paradigm_2afc_merge_concatenate from the osculari.models module.
architecture = 'resnet50' # network's architecture
weights = 'resnet50' # the pretrained weights
img_size = 224 # network's input size
layer = 'block0' # the readout layer
readout_kwargs = {
'architecture': architecture,
'weights': weights,
'layers': layer,
'img_size': img_size,
}
net_2afc = osculari.models.paradigm_2afc_merge_concatenate(**readout_kwargs)

Datasets
The osculari.datasets module provides datasets that are generated randomly on the fly with
flexible properties that can be dynamically changed based on the experiment of interest.
For instance, by passing a appearance_fun to the ShapeAppearanceDataset class, we can
dynamically merge foreground masks with background images to generate stimuli of interest.
def appearance_fun(foregrounds, backgrounds):
# implementing the required appearance (colour, texture, etc.) on foreground and merging
# to background.
return merged_imgs, ground_truth

num_samples = 1000 # the number of random samples generated in the dataset
num_imgs = net_2afc.input_nodes # the number of images in each sample
background = 128 # the background type
dataset = osculari.datasets.geometrical_shapes.ShapeAppearanceDataset(
num_samples, num_imgs, img_size, background, appearance_fun,
unique_bg=True, transform=net_2afc.preprocess_transform()
)

Linear probe
The osculari.paradigms module implements a set of psychophysical paradigms. The train_linear_probe
function trains the network on a dataset following the paradigm passed to the function.
# experiment-dependent function to train on an epoch of data
epoch_fun = osculari.paradigms.forced_choice.epoch_loop
# calling the generic train_linear_probe function
training_log = osculari.paradigms.paradigm_utils.train_linear_probe(
net_2afc, dataset, epoch_fun, './osculari_test/'
)

Psychophysical experiment
The osculari.paradigms module also implements a set of psychophysical experiments similar to the
experiments conducted with human participants. In this example, we use the staircase function
to gradually measure the network's sensitivity.
# experiment-dependent function to test an epoch of data
test_epoch_fun = osculari.paradigms.forced_choice.test_dataset
# the test dataset implementing desired stimuli.
class TestDataset(TorchDataset):
def __getitem__(self, idx):
return stimuli

test_log = osculari.paradigms.staircase(
net_2afc, test_epoch_fun, TestDataset(), low_val=0, high_val=1
)


Contribution
We welcome all contributions to the project that extend or improve code and/or documentation!
Please read the CONTRIBUTING page and follow the Code of Conduct.