API

Synthesis objects

Synthesis objects generate novel stimuli which allow researchers to better understand how their computational models make sense of their inputs.

Metamer	Synthesize metamers for image-computable differentiable models.
MetamerCTF	Synthesize model metamers with coarse-to-fine synthesis.
Eigendistortion	Synthesize eigendistortions induced by a model on a given input image.
MADCompetition	Synthesize a single maximally-differentiating image for two metrics.

Models

Models give a response to a single stimulus and are compatible with Metamer and Eigendistortion, and can be turned into Metrics by using the model_metric_factory function.

See Model requirements for more details.

PortillaSimoncelli

Portila-Simoncelli texture statistics.

LGN-inspired Models

These “front end” models are inspired by the lateral geniculate nucleus (LGN; the first non-retinal stage of the primate visual system), come from Berardino et al., 2017, and are nested, increasing in complexity as you move down the list.

LinearNonlinear	Linear-Nonlinear model.
LuminanceGainControl	Linear center-surround followed by luminance gain control and activation.
LuminanceContrastGainControl	Center-surround followed by luminance and contrast gain control, then activation.
OnOff	On-off and off-on center-surround with contrast and luminance gain control.

The following models are used to construct the models above. They are probably most useful in the construction of other, more complex models, but they are compatible with our synthesis methods.

Identity	Simple class that just returns a copy of the image.
Linear	Simplistic linear convolutional model.
Gaussian	Isotropic Gaussian convolutional filter.
CenterSurround	Center-Surround, Difference of Gaussians (DoG) filter model.

Metrics

Metrics compare two stimuli and are compatible with MADCompetition.

See Metric requirements for more details.

mse	Return the MSE between img1 and img2.
model_metric_factory	Create a metric function which returns the root mean squared error in model space.
ssim	Compute the structural similarity index.
ms_ssim	Multiscale structural similarity index (MS-SSIM).
nlpd	Compute the normalized Laplacian Pyramid Distance.

Commonly-used helper functions

The following helper functions are commonly-used when preparing data and models for use with plenoptic.

remove_grad	Detach all parameters and buffers of model (in place).
set_seed	Set the seed.
load_images	Load in images.
to_numpy	Cast tensor to numpy in the most conservative way possible.
convert_float_to_int	Convert numpy array from float to 8 or 16 bit integer.

Plotting functions

Synthesis objects

These functions all are intended to help visualize the status and outputs of synthesis objects. The following accept all synthesis classes:

synthesis_imshow	Display image of synthesis object.
synthesis_histogram	Plot histogram of values of synthesis objects.
synthesis_status	Make a plot showing synthesis status.

The following accept Metamer / MetamerCTF and MADCompetition objects:

synthesis_loss	Plot synthesis loss.
synthesis_animate	Animate synthesis progress.

The following accept only objects of a single class.

Metamer / MetamerCTF

metamer_representation_error

Plot representation error showing how close we are to convergence.

MADCompetition

mad_imshow_all	Display all MAD Competition images.
mad_loss_all	Plot loss for full set of MAD Competiton instances.

Eigendistortion

eigendistortion_imshow_all

Display base image, eigendistortions alone, and eigendistortions added to the image.

Tensors

The following functions can be used to visualize torch.Tensor objects directly: images, videos, and model representations.

imshow	Show image(s), avoiding interpolation.
animshow	Animate video(s), avoiding interpolation.
pyrshow	Display steerable pyramid coefficients in orderly fashion.
plot_representation	Plot model representation.
stem_plot	Create a simple stem plot.
histogram	Plot histogram of values from tensor.
update_plot	Update the information in some axes (for an animation).

Image-processing methods

These classes and functions all process images in some way, and thus may be helpful for constructing your own models or metrics. As is, they are not compatible with any of the synthesis methods.

Metric components

These functions are used by the Metrics. While they are not compatible with any of our synthesis methods, they may be useful to better understand the behavior of their respective metrics.

ssim_map	Structural similarity index map.
normalized_laplacian_pyramid	Compute the normalized Laplacian Pyramid using pre-optimized parameters.

Image pyramids

Image pyramids decompose images into bands corresponding to different spatial frequencies and orientations. As we often think of neurons in the early visual system as similarly only responding to a range of spatial frequencies and/or orientations, these pyramids can be used to construct a model of those early areas.

See pyrtools for more information, including links to resources for learning more.

LaplacianPyramid	Laplacian Pyramid in Torch.
SteerablePyramidFreq	Steerable frequency pyramid in Torch.

Filter construction functions

These convenience functions make it easier to construct some commonly-used filters.

circular_gaussian2d

Create normalized, centered circular 2D gaussian tensor with which to convolve.

Coordinate transformations

This related set of functions convert between the rectangular and polar representations of a signal (or computing the norm and direction, which are the analogues for real-valued signals).

rectangular_to_polar	Rectangular to polar coordinate transform.
polar_to_rectangular	Polar to rectangular coordinate transform.
local_gain_control	Spatially local gain control.
local_gain_release	Spatially local gain release.
rectangular_to_polar_dict	Return the complex amplitude and the phase of each complex tensor in a dictionary.
polar_to_rectangular_dict	Return the real and imaginary parts of tensor in a dictionary.
local_gain_control_dict	Spatially local gain control, for each element in a dictionary.
local_gain_release_dict	Spatially local gain release, for each element in a dictionary.

The following table summarizes the relationship among these functions (where ➡ denotes from rectangular to polar and ⬅ is the inverse):

function	dtype	input object	direction
rectangular_to_polar	complex*	tensor	➡
polar_to_rectangular	complex*	tensor	⬅
local_gain_control	float*	tensor	➡
local_gain_release	float*	tensor	⬅
rectangular_to_polar_dict	complex*	dict	➡
polar_to_rectangular_dict	complex*	dict	⬅
local_gain_control_dict	float*	dict	➡
local_gain_release_dict	float*	dict	⬅

Image resizing

The following functions return a version of their input whose size has increased or decreased without further modifying its contents.

correlate_downsample	Correlate with a filter and downsample by a factor of 2.
blur_downsample	Correlate with a named filter and downsample by 2.
upsample_convolve	Upsample by 2 and convolve with a filter.
upsample_blur	Upsample by 2 and convolve with named filter.
same_padding	Pad a tensor so that 2D convolution will result in output with same dims.
shrink	Shrink a signal by a factor.
expand	Expand a signal by a factor.

Image modification

The following functions return a modified version of their input.

rescale	Linearly rescale the dynamic range of the input to [a, b].
add_noise	Add normally distributed noise to an image.
center_crop	Crop out the center of a signal.
modulate_phase	Modulate the phase of a complex signal.

Image statistics

The following functions compute summary statistics of their inputs.

autocorrelation	Compute the autocorrelation of x.
variance	Calculate sample variance.
skew	Calculate sample estimate of asymmetry about input's mean.
kurtosis	Calculate sample estimate of tailedness (presence of outliers).

Loading, creating, and accessing images

Example images

Plenoptic includes a small number of example images, which are used throughout the documentation.

einstein	Return an example grayscale image of Albert Einstein.
curie	Return an example grayscale image of Marie Curie.
parrot	Return an example grayscale image of a parrot.
reptile_skin	Return an example grayscale image of reptile skin.
color_wheel	Return an example image of a color wheel.

Plenoptic includes a helper function for downloading additional data, which includes example synthesis objects and additional images.

fetch_data	Download data, using pooch.
DOWNLOADABLE_FILES	Built-in mutable sequence.

Synthetic images

The following functions create synthetic image tensors. See pyrtools.tools.synthetic_images for more synthetic images.

disk	Create a circular mask with softened edges.
polar_radius	Make distance-from-origin (r) matrix.
polar_angle	Make polar angle matrix (in radians).

Validation

The following functions are used to validate that the user-supplied inputs are compatible with our synthesis objects.

validate_model	Determine whether model can be used for synthesis.
validate_input	Determine whether input_tensor can be used for synthesis.
validate_metric	Determine whether a metric can be used for MADCompetition synthesis.
validate_coarse_to_fine	Determine whether a model can be used for coarse-to-fine synthesis.
validate_convert_tensor_dict	Determine if a model can properly convert between tensor and dict representations.
validate_penalty	Determine whether penalty_function can be used for regularization in synthesis.

Optimization

Loss functions

The following functions can be used as the loss_function argument for Metamer and MetamerCTF.

mse	Calculate the MSE between synth_rep and ref_rep.
l2_norm	Calculate the L2-norm of the difference between ref_rep and synth_rep.
relative_sse	Calculate the relative sum of squared errors between two tensors.

Loss function factories

The following functions return a function that can be used as a loss_function, as above.

portilla_simoncelli_loss_factory	Create the loss function required for PortillaSimoncelli metamer synthesis.
groupwise_relative_l2_norm_factory	Create loss function that computes groupwise relative L2 norm for synthesis.

Penalty functions

The following functions operate as penalties.

penalize_range

Calculate quadratic penalty on values outside of allowed_range.

Debugging

Functions to help understand when things have gone wrong.

examine_saved_synthesis	Examine saved synthesis object.
LoadWarning	Custom warning to raise if there's an issue with loading.

External

Functions for interacting with other code and packages.

plot_MAD_results

Plot original MAD results, provided by Zhou Wang.