plenoptic

plenoptic is a python library for model-based synthesis of perceptual stimuli, built on top of pytorch. For plenoptic, models are those of visual [1] information processing: they accept an image[2] as input, perform some computations, and return some output, which can be mapped to neuronal firing rate, fMRI BOLD response, behavior on some task, image category, etc. The intended audience is researchers in neuroscience, psychology, and machine learning. For any pytorch model, users can generate stimuli which enable interpretation of model properties through examination of features that are enhanced, suppressed, or discarded. More importantly, these stimuli facilitate the scientific process, through use in further perceptual or neural experiments aimed at validating or falsifying model predictions.

Breaking API Changes!

Plenoptic 2.0 introduces breaking changes to the API and will not work with code written to work with earlier versions! See the Plenoptic 2.0 Migration Guide for details, including a script to update your code.

Installation

Quickstart

Video Introduction

Citation Guide

Contents

Synthesis methods

• Metamers: given a model and a reference image, stochastically generate a new image whose model representation is identical to that of the reference image (a “metamer”, as originally defined in the literature on Trichromacy). This method makes explicit those features that the model retains/discards.

  • Example papers: Portilla and Simoncelli, 2000, Freeman and Simoncelli, 2011, Deza et al., 2019, Feather et al., 2019, Wallis et al., 2019, Ziemba and Simoncelli, 2021

• Eigendistortions: given a model and a reference image, compute the image perturbations that produce the smallest/largest change in the model response space. These are the image changes to which the model is least/most sensitive, respectively.

  • Example papers: Berardino et al., 2017

• Maximal differentiation (MAD) competition: given a reference image and two models that measure distance between images, generate pairs of images that optimally differentiate the models. Specifically, synthesize a pair of images that are equi-distant from the reference image according to model-1, but maximally/minimally distant according to model-2. Synthesize a second pair with the roles of the two models reversed. This method allows for efficient comparison of two metrics, highlighting the aspects in which their sensitivities most differ.

  • Example papers: Wang and Simoncelli, 2008

Models, Metrics, and Model Components

• Steerable pyramid, Simoncelli and Freeman, 1995, Simoncelli et al., 1992, is a multi-scale oriented image decomposition. Images are decomposed with a family of oriented filters, localized in space and frequency, similar to the “Gabor functions” commonly used to model receptive fields in primary visual cortex. The critical difference is that the pyramid organizes these filters so as to efficiently cover the 4D space of (x,y) positions, orientations, and scales, enabling efficient interpolation and interpretation (further info). See the pyrtools documentation for more details on python tools for image pyramids in general and the steerable pyramid in particular.

• Portilla-Simoncelli texture model, Portilla and Simoncelli, 2000, which computes a set of image statistics that capture the appearance of visual textures (further info).

• Structural Similarity Index (SSIM), Wang et al., 2004, is a perceptual similarity metric, that takes two images and returns a value between -1 (totally different) and 1 (identical) reflecting their similarity (further info).

• Multiscale Structural Similarity Index (MS-SSIM), Wang et al., 2003, is an extension of SSIM that operates jointly over multiple scales.

• Normalized Laplacian distance, Laparra et al., 2016, Laparra et al., 2017, is a perceptual distance metric based on transformations associated with the early visual system: local luminance subtraction and local contrast gain control, at six scales (further info).

Getting help

We communicate via several channels on Github:

• To report a bug, open an issue.

• To send suggestions for extensions or enhancements, please post in the ideas section of discussions first. We’ll discuss it there and, if we decide to pursue it, open an issue to track progress.

• To ask usage questions, discuss broad issues, or show off what you’ve made with plenoptic, go to Discussions.

• To contribute to the project, see the contributing guide.

In all cases, we request that you respect our code of conduct.

Citing us

If you use plenoptic in a published academic article or presentation, please cite us! See the Citation Guide and Bibliography for more details.

Related packages

If you are curious about plenoptic, these other packages may also be of interest:

• pytorch: optimized tensor library for deep learning using GPUs and CPUs. plenoptic depends on pytorch for automatic differentiation. Any valid pytorch model is compatible with plenoptic’s methods (see Model requirements for more details).

• pyrtools: a python package for multi-scale image processing. Includes numpy implementations of several of the image pyramids and image-processing tools found in plenoptic.

• lucent: pytorch adaptation of the lucid TensorFlow library, which contains a variety of tools for research in neural network interpretability. Similar to plenoptic, contains methods for synthesizing images to better understand how models process visual information.

This package is supported by the Center for Computational Neuroscience, in the Flatiron Institute of the Simons Foundation.

---

[1]

These methods also work with auditory models, such as in Feather et al., 2019 though we haven’t yet implemented examples. If you’re interested, please post in Discussions!

[2]

Here and throughout the documentation, we use “image” to describe the input. The models and metrics that are included in plenoptic are intended to work on images, represented as 4d tensors. However, the synthesis methods should also work on videos (5d tensors), audio (3d tensors) and more! If you have a problem using a tensor with different dimensionality, please open an issue!