plenoptic.synthesize package

Submodules

plenoptic.synthesize.autodiff module

plenoptic.synthesize.autodiff.jacobian(y, x)

Explicitly compute the full Jacobian matrix. N.B. This is only recommended for small input sizes (e.g. <100x100 image)

Parameters:

• y (Tensor) – Model output with gradient attached

• x (Tensor) – Tensor with gradient function model input with gradient attached

Returns:

Jacobian matrix with torch.Size([len(y), len(x)])

Return type:

J

plenoptic.synthesize.autodiff.jacobian_vector_product(y, x, V, dummy_vec=None)

Compute Jacobian Vector Product: \(\text{jvp} = (\partial y/\partial x) v\)

Forward Mode Auto-Differentiation (Rop in Theano). PyTorch does not natively support this operation; this function essentially calls backward mode autodiff twice, as described in [1].

See vector_jacobian_product() docstring on why we and pass arguments for retain_graph and create_graph.

Parameters:

• y (Tensor) – Model output with gradient attached, shape is torch.Size([m, 1])

• x (Tensor) – Model input with gradient attached, shape is torch.Size([n, 1]), i.e. same dim as input tensor

• V (Tensor) – Directions in which to compute product, shape is torch.Size([n, k]) where k is number of vectors to compute

• dummy_vec (Tensor) – Vector with which to do jvp trick [1]. If argument exists, then use some pre-allocated, cached vector, otherwise create a new one and move to device in this method.

Returns:

Jacobian-vector product, torch.Size([n, k])

Return type:

Jv

Notes

[1] https://j-towns.github.io/2017/06/12/A-new-trick.html

plenoptic.synthesize.autodiff.vector_jacobian_product(y, x, U, retain_graph=True, create_graph=True, detach=False)

Compute vector Jacobian product: \(\text{vjp} = u^T(\partial y/\partial x)\)

Backward Mode Auto-Differentiation (Lop in Theano)

Note on efficiency: When this function is used in the context of power iteration for computing eigenvectors, the vector output will be repeatedly fed back into :meth: vector_jacobian_product() and jacobian_vector_product(). To prevent the accumulation of gradient history in this vector (especially on GPU), we need to ensure the computation graph is not kept in memory after each iteration. We can do this by detaching the output, as well as carefully specifying where/when to retain the created graph.

Parameters:

• y (Tensor) – Output with gradient attached, torch.Size([m, 1]).

• x (Tensor) – Input with gradient attached, torch.Size([n, 1]).

• U (Tensor) – Direction, shape is torch.Size([m, k]), i.e. same dim as output tensor.

• retain_graph (bool) – Whether or not to keep graph after doing one vector_jacobian_product(). Must be set to True if k>1.

• create_graph (bool) – Whether or not to create computational graph. Usually should be set to True unless you’re reusing the graph like in the second step of jacobian_vector_product().

• detach (bool) – As with create_graph, only necessary to be True when reusing the output like we do in the 2nd step of jacobian_vector_product().

Returns:

vector-Jacobian product, torch.Size([m, k]).

Return type:

vJ

plenoptic.synthesize.eigendistortion module

class plenoptic.synthesize.eigendistortion.Eigendistortion(image, model)

Bases: Synthesis

Synthesis object to compute eigendistortions induced by a model on a given input image.

Parameters:

• image (Tensor) – Image, torch.Size(batch=1, channel, height, width). We currently do not support batches of images, as each image requires its own optimization.

• model (Module) – Torch model with defined forward and backward operations.

batch_size

Type:

int

n_channels

Type:

int

im_height

Type:

int

im_width

Type:

int

jacobian

Is only set when synthesize() is run with method='exact'. Default to None.

Type:

Tensor

eigendistortions

Tensor of eigendistortions (eigenvectors of Fisher matrix), ordered by eigenvalue, with Size((n_distortions, n_channels, im_height, im_width)).

Type:

Tensor

eigenvalues

Tensor of eigenvalues corresponding to each eigendistortion, listed in decreasing order.

Type:

Tensor

eigenindex

Index of each eigenvector/eigenvalue.

Type:

listlike

Notes

This is a method for comparing image representations in terms of their ability to explain perceptual sensitivity in humans. It estimates eigenvectors of the FIM. A model, \(y = f(x)\), is a deterministic (and differentiable) mapping from the input pixels \(x \in \mathbb{R}^n\) to a mean output response vector \(y\in \mathbb{R}^m\), where we assume additive white Gaussian noise in the response space. The Jacobian matrix at x is:

\(J(x) = J = dydx\), \(J\in\mathbb{R}^{m \times n}\) (ie. output_dim x input_dim)

The matrix consists of all partial derivatives of the vector-valued function f. The Fisher Information Matrix (FIM) at x, under white Gaussian noise in the response space, is:

\(F = J^T J\)

It is a quadratic approximation of the discriminability of distortions relative to \(x\).

References

[1]

Berardino, A., Laparra, V., Ballé, J. and Simoncelli, E., 2017. Eigen-distortions of hierarchical representations. In Advances in neural information processing systems (pp. 3530-3539). https://www.cns.nyu.edu/pub/lcv/berardino17c-final.pdf https://www.cns.nyu.edu/~lcv/eigendistortions/

Attributes:

eigendistortions

Tensor of eigendistortions (eigenvectors of Fisher matrix), ordered by eigenvalue.

eigenindex

Index of each eigenvector/eigenvalue.

eigenvalues

Tensor of eigenvalues corresponding to each eigendistortion, listed in decreasing order.

image

jacobian

Is only set when synthesize() is run with method='exact'.

model

Methods

compute_jacobian()	Calls autodiff.jacobian and returns jacobian.
load(file_path[, map_location])	Load all relevant stuff from a .pt file.
save(file_path)	Save all relevant variables in .pt file.
synthesize([method, k, max_iter, p, q, ...])	Compute eigendistortions of Fisher Information Matrix with given input image.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.

compute_jacobian()

Calls autodiff.jacobian and returns jacobian. Will throw error if input too big.

Returns:

Jacobian of representation wrt input.

Return type:

J

property eigendistortions

Tensor of eigendistortions (eigenvectors of Fisher matrix), ordered by eigenvalue.

property eigenindex

Index of each eigenvector/eigenvalue.

property eigenvalues

Tensor of eigenvalues corresponding to each eigendistortion, listed in decreasing order.

property image

property jacobian

Is only set when synthesize() is run with method='exact'. Default to None.

load(file_path, map_location=None, **pickle_load_args)

Load all relevant stuff from a .pt file.

This should be called by an initialized Eigendistortion object – we will ensure that image and model are identical.

Note this operates in place and so doesn’t return anything.

Parameters:

• file_path (str) – The path to load the synthesis object from

• map_location (str, optional) – map_location argument to pass to torch.load. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device

• pickle_load_args – any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Examples

>>> eig = po.synth.Eigendistortion(img, model)
>>> eig.synthesize(max_iter=10)
>>> eig.save('eig.pt')
>>> eig_copy = po.synth.Eigendistortion(img, model)
>>> eig_copy.load('eig.pt')

Note that you must create a new instance of the Synthesis object and then load.

property model

save(file_path)

Save all relevant variables in .pt file.

See load docstring for an example of use.

Parameters:

file_path (str) – The path to save the Eigendistortion object to

synthesize(method='power', k=1, max_iter=1000, p=5, q=2, stop_criterion=1e-07)

Compute eigendistortions of Fisher Information Matrix with given input image.

Parameters:

• method (Literal['exact', 'power', 'randomized_svd']) – Eigensolver method. ‘exact’ tries to do eigendecomposition directly ( not recommended for very large inputs). ‘power’ (default) uses the power method to compute first and last eigendistortions, with maximum number of iterations dictated by n_steps. ‘randomized_svd’ uses randomized SVD to approximate the top k eigendistortions and their corresponding eigenvalues.

• k (int) – How many vectors to return using block power method or svd.

• max_iter (int) – Maximum number of steps to run for method='power' in eigenvalue computation. Ignored for other methods.

• p (int) – Oversampling parameter for randomized SVD. k+p vectors will be sampled, and k will be returned. See docstring of _synthesize_randomized_svd for more details including algorithm reference.

• q (int) – Matrix power parameter for randomized SVD. This is an effective trick for the algorithm to converge to the correct eigenvectors when the eigenspectrum does not decay quickly. See _synthesize_randomized_svd for more details including algorithm reference.

• stop_criterion (float) – Used if method='power' to check for convergence. If the L2-norm of the eigenvalues has changed by less than this value from one iteration to the next, we terminate synthesis.

to(*args, **kwargs)

Moves and/or casts the parameters and buffers.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to(), but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See below for examples.

Note

This method modifies the module in-place.

Args:

device (torch.device): the desired device of the parameters

and buffers in this module

dtype (torch.dtype): the desired floating point type of

the floating point parameters and buffers in this module

tensor (torch.Tensor): Tensor whose dtype and device are the desired

dtype and device for all parameters and buffers in this module

plenoptic.synthesize.eigendistortion.display_eigendistortion(eigendistortion, eigenindex=0, alpha=5.0, process_image=<function <lambda>>, ax=None, plot_complex='rectangular', **kwargs)

Displays specified eigendistortion added to the image.

If image or eigendistortions have 3 channels, then it is assumed to be a color image and it is converted to grayscale. This is merely for display convenience and may change in the future.

Parameters:

• eigendistortion (Eigendistortion) – Eigendistortion object whose synthesized eigendistortion we want to display

• eigenindex (int) – Index of eigendistortion to plot. E.g. If there are 10 eigenvectors, 0 will index the first one, and -1 or 9 will index the last one.

• alpha (float) – Amount by which to scale eigendistortion for image + (alpha * eigendistortion) for display.

• process_image (Callable[[Tensor], Tensor]) – A function to process the image+alpha*distortion before clamping between 0,1. E.g. multiplying by the stdev ImageNet then adding the mean of ImageNet to undo image preprocessing.

• ax (Axes | None) – Axis handle on which to plot.

• plot_complex (str) – Parameter for plenoptic.imshow() determining how to handle complex values. Defaults to ‘rectangular’, which plots real and complex components as separate images. Can also be ‘polar’ or ‘logpolar’; see that method’s docstring for details.

• kwargs – Additional arguments for po.imshow().

Returns:

matplotlib Figure handle returned by plenoptic.imshow()

Return type:

fig

plenoptic.synthesize.eigendistortion.fisher_info_matrix_eigenvalue(y, x, v, dummy_vec=None)

Compute the eigenvalues of the Fisher Information Matrix corresponding to eigenvectors in v:math:lambda= v^T F v

Return type:

Tensor

plenoptic.synthesize.eigendistortion.fisher_info_matrix_vector_product(y, x, v, dummy_vec)

Compute Fisher Information Matrix Vector Product: \(Fv\)

Parameters:

• y (Tensor) – Output tensor with gradient attached

• x (Tensor) – Input tensor with gradient attached

• v (Tensor) – The vectors with which to compute Fisher vector products

• dummy_vec (Tensor) – Dummy vector for Jacobian vector product trick

Returns:

Vector, Fisher vector product

Return type:

Fv

Notes

Under white Gaussian noise assumption, \(F\) is matrix multiplication of Jacobian transpose and Jacobian: \(F = J^T J\). Hence: \(Fv = J^T (Jv)\)

plenoptic.synthesize.geodesic module

class plenoptic.synthesize.geodesic.Geodesic(image_a, image_b, model, n_steps=10, initial_sequence='straight', range_penalty_lambda=0.1, allowed_range=(0, 1))

Bases: OptimizedSynthesis

Synthesize an approximate geodesic between two images according to a model.

This method can be used to visualize and refine the invariances of a model’s representation as described in [1].

NOTE: This synthesis method is still under construction. It will run, but it might not find the most informative geodesic.

Parameters:

• image_a (Tensor) – Start and stop anchor points of the geodesic, of shape (1, channel, height, width).

• image_b (Tensor) – Start and stop anchor points of the geodesic, of shape (1, channel, height, width).

• model (Module) – an analysis model that computes representations on signals like image_a.

• n_steps (int) – the number of steps (i.e., transitions) in the trajectory between the two anchor points.

• initial_sequence (Literal['straight', 'bridge']) – initialize the geodesic with pixel linear interpolation ('straight'), or with a brownian bridge between the two anchors ('bridge').

• range_penalty_lambda (float) – strength of the regularizer that enforces the allowed_range. Must be non-negative.

• allowed_range (tuple[float, float]) – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.

geodesic

the synthesized sequence of images between the two anchor points that minimizes representation path energy, of shape (n_steps+1, channel, height, width). It starts with image_a and ends with image_b.

Type:

Tensor

pixelfade

the straight interpolation between the two anchor points, used as reference

Type:

Tensor

losses

A list of our loss over iterations.

Type:

Tensor

gradient_norm

A list of the gradient’s L2 norm over iterations.

Type:

list

pixel_change_norm

A list containing the L2 norm of the pixel change over iterations (pixel_change_norm[i] is the pixel change norm in geodesic between iterations i and i-1).

Type:

list

step_energy

step lengths in representation space, stored along the optimization process.

Type:

Tensor

dev_from_line

deviation of the representation to the straight line interpolation, measures distance from straight line and distance along straight line, stored along the optimization process

Type:

Tensor

Notes

Manifold prior hypothesis: natural images form a manifold 𝑀ˣ embedded in signal space (ℝⁿ), a model warps this manifold to another manifold 𝑀ʸ embedded in representation space (ℝᵐ), and thereby induces a different local metric.

This method computes an approximate geodesics by solving an optimization problem: it minimizes the path energy (aka. action functional), which has the same minimum as minimizing path length and by Cauchy-Schwarz, reaches it with constant-speed minimizing geodesic

Caveat: depending on the geometry of the manifold, geodesics between two anchor points may not be unique and may depend on the initialization.

References

[1]

Geodesics of learned representations O J Hénaff and E P Simoncelli Published in Int’l Conf on Learning Representations (ICLR), May 2016. https://www.cns.nyu.edu/~lcv/pubs/makeAbs.php?loc=Henaff16b

Deprecated since version 1.1.0: Geodesic is not robust enough yet, see https://github.com/plenoptic-org/geodesics for ongoing development

Attributes:

allowed_range

dev_from_line

Deviation of representation each from of self.geodesic from a straight line.

geodesic

gradient_norm

Synthesis gradient’s L2 norm over iterations.

image_a

image_b

losses

Synthesis loss over iterations.

model

optimizer

pixel_change_norm

L2 norm change in pixel values over iterations.

range_penalty_lambda

step_energy

Squared L2 norm of transition between geodesic frames in representation space.

store_progress

Methods

calculate_jerkiness([geodesic])	Compute the alignment of representation's acceleration to model local curvature.
load(file_path[, map_location])	Load all relevant stuff from a .pt file.
objective_function([geodesic])	Compute geodesic synthesis loss.
save(file_path)	Save all relevant variables in .pt file.
synthesize([max_iter, optimizer, ...])	Synthesize a geodesic via optimization.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.

calculate_jerkiness(geodesic=None)

Compute the alignment of representation’s acceleration to model local curvature.

This is the first order optimality condition for a geodesic, and can be used to assess the validity of the solution obtained by optimization.

Parameters:

geodesic (Tensor | None) – Geodesic to check. If None, we use self.geodesic. Must have a gradient attached.

Return type:

jerkiness

property dev_from_line

Deviation of representation each from of self.geodesic from a straight line.

Has shape (np.ceil(synth_iter/store_progress), n_steps+1, 2), where synth_iter is the number of iterations of synthesis that have happened. For final dimension, the first element is the Euclidean distance along the straight line and the second is the Euclidean distance to the line.

property geodesic

property image_a

property image_b

load(file_path, map_location=None, **pickle_load_args)

Load all relevant stuff from a .pt file.

This should be called by an initialized Geodesic object – we will ensure that image_a, image_b, model, n_steps, initial_sequence, range_penalty_lambda, allowed_range, and pixelfade are all identical.

Note this operates in place and so doesn’t return anything.

Parameters:

• file_path (str) – The path to load the synthesis object from

• map_location (str, optional) – map_location argument to pass to torch.load. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device

• pickle_load_args – any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Examples

>>> geo = po.synth.Geodesic(img_a, img_b, model)
>>> geo.synthesize(max_iter=10, store_progress=True)
>>> geo.save('geo.pt')
>>> geo_copy = po.synth.Geodesic(img_a, img_b, model)
>>> geo_copy.load('geo.pt')

Note that you must create a new instance of the Synthesis object and then load.

property model

objective_function(geodesic=None)

Compute geodesic synthesis loss.

This is the path energy (i.e., squared L2 norm of each step) of the geodesic’s model representation, with the weighted range penalty.

Additionally, caches:

• self._geodesic_representation = self.model(geodesic)

• ``self._most_recent_step_energy = self._calculate_step_energy(

  self._geodesic_representation)``

These are cached because we might store them (if self.store_progress is True) and don’t want to recalcualte them

Parameters:

geodesic (Tensor | None) – Geodesic to check. If None, we use self.geodesic.

Return type:

loss

save(file_path)

Save all relevant variables in .pt file.

See load docstring for an example of use.

Parameters:

file_path (str) – The path to save the Geodesic object to

property step_energy

Squared L2 norm of transition between geodesic frames in representation space.

Has shape (np.ceil(synth_iter/store_progress), n_steps), where synth_iter is the number of iterations of synthesis that have happened.

synthesize(max_iter=1000, optimizer=None, store_progress=False, stop_criterion=None, stop_iters_to_check=50)

Synthesize a geodesic via optimization.

Parameters:

• max_iter (int) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).

• optimizer (Optimizer | None) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.001, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.

• store_progress (bool | int) – Whether we should store the step energy and deviation of the representation from a straight line. If False, we don’t save anything. If True, we save every iteration. If an int, we save every store_progress iterations (note then that 0 is the same as False and 1 the same as True).

• stop_criterion (float | None) – If pixel_change_norm (i.e., the norm of the difference in self.geodesic from one iteration to the next) over the past stop_iters_to_check has been less than stop_criterion, we terminate synthesis. If None, we pick a default value based on the norm of self.pixelfade.

• stop_iters_to_check (int) – How many iterations back to check in order to see if pixel_change_norm has stopped decreasing (for stop_criterion).

to(*args, **kwargs)

Moves and/or casts the parameters and buffers.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to(), but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See below for examples.

Note

This method modifies the module in-place.

Args:

device (torch.device): the desired device of the parameters

and buffers in this module

dtype (torch.dtype): the desired floating point type of

the floating point parameters and buffers in this module

tensor (torch.Tensor): Tensor whose dtype and device are the desired

dtype and device for all parameters and buffers in this module

plenoptic.synthesize.geodesic.plot_deviation_from_line(geodesic, natural_video=None, ax=None)

Visual diagnostic of geodesic linearity in representation space.

This plot illustrates the deviation from the straight line connecting the representations of a pair of images, for different paths in representation space.

Parameters:

• geodesic (Geodesic) – Geodesic object to visualize.

• natural_video (Tensor | None) – Natural video that bridges the anchor points, for comparison.

• ax (Axes | None) – If not None, the axis to plot this representation on. If None, we call plt.gca()

Returns:

Axes containing the plot

Return type:

ax

Notes

Axes are in the same units, normalized by the distance separating the end point representations.

Knots along each curve indicate samples used to compute the path.

When the representation is non-linear it may not be feasible for the geodesic to be straight (for example if the representation is normalized, all paths are constrained to live on a hypershpere). Nevertheless, if the representation is able to linearize the transformation between the anchor images, then we expect that both the ground truth natural video sequence and the geodesic will deviate from straight line similarly. By contrast the pixel-based interpolation will deviate significantly more from a straight line.

Deprecated since version 1.1.0: Geodesic is not robust enough yet, see https://github.com/plenoptic-org/geodesics for ongoing development

plenoptic.synthesize.geodesic.plot_loss(geodesic, ax=None, **kwargs)

Plot synthesis loss.

Parameters:

• geodesic (Geodesic) – Geodesic object whose synthesis loss we want to plot.

• ax (Axes | None) – If not None, the axis to plot this representation on. If None, we call plt.gca()

• kwargs – passed to plt.semilogy

Returns:

Axes containing the plot.

Return type:

ax

Notes

Deprecated since version 1.1.0: Geodesic is not robust enough yet, see https://github.com/plenoptic-org/geodesics for ongoing development

plenoptic.synthesize.mad_competition module

Run MAD Competition.

class plenoptic.synthesize.mad_competition.MADCompetition(image, optimized_metric, reference_metric, minmax, initial_noise=0.1, metric_tradeoff_lambda=None, range_penalty_lambda=0.1, allowed_range=(0, 1))

Bases: OptimizedSynthesis

Synthesize a single maximally-differentiating image for two metrics.

Following the basic idea in [1], this class synthesizes a maximally-differentiating image for two given metrics, based on a given image. We start by adding noise to this image and then iteratively adjusting its pixels so as to either minimize or maximize optimized_metric while holding the value of reference_metric constant.

MADCompetiton accepts two metrics as its input. These should be callables that take two images and return a single number, and that number should be 0 if and only if the two images are identical (thus, the larger the number, the more different the two images).

Note that a full set of images MAD Competition images consists of two pairs: a maximal and a minimal image for each metric. A single instantiation of MADCompetition will generate one of these four images.

Parameters:

• image (Tensor) – A 4d tensor, this is the image whose representation we wish to match. If this is not a tensor, we try to cast it as one.

• optimized_metric (Module | Callable[[Tensor, Tensor], Tensor]) – The metric whose value you wish to minimize or maximize, which takes two tensors and returns a scalar. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the MADCompetition object (i.e., it must be one of our built-in functions or defined using a def statement)

• reference_metric (Module | Callable[[Tensor, Tensor], Tensor]) – The metric whose value you wish to keep fixed, which takes two tensors and returns a scalar. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the MADCompetition object (i.e., it must be one of our built-in functions or defined using a def statement)

• minmax (Literal['min', 'max']) – Whether you wish to minimize or maximize optimized_metric.

• initial_noise (float) – Standard deviation of the Gaussian noise used to initialize mad_image from image.

• metric_tradeoff_lambda (float | None) – Lambda to multiply by reference_metric loss and add to optimized_metric loss. If None, we pick a value so the two initial losses are approximately equal in magnitude.

• range_penalty_lambda (float) – Lambda to multiply by range penalty and add to loss.

• allowable_range – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.

mad_image

The Maximally-Differentiating Image. This may be unfinished depending on how many iterations we’ve run for.

Type:

torch.Tensor

initial_image

The initial mad_image, which we obtain by adding Gaussian noise to image.

Type:

torch.Tensor

losses

A list of the objective function’s loss over iterations.

Type:

list

gradient_norm

A list of the gradient’s L2 norm over iterations.

Type:

list

pixel_change_norm

A list containing the L2 norm of the pixel change over iterations (pixel_change_norm[i] is the pixel change norm in mad_image between iterations i and i-1).

Type:

list

optimized_metric_loss

A list of the optimized_metric loss over iterations.

Type:

list

reference_metric_loss

A list of the reference_metric loss over iterations.

Type:

list

saved_mad_image

Saved self.mad_image for later examination.

Type:

torch.Tensor

References

[1]

Wang, Z., & Simoncelli, E. P. (2008). Maximum differentiation (MAD) competition: A methodology for comparing computational models of perceptual discriminability. Journal of Vision, 8(12), 1–13. https://dx.doi.org/10.1167/8.12.8

Attributes:

allowed_range

gradient_norm

Synthesis gradient’s L2 norm over iterations.

image

initial_image

losses

Synthesis loss over iterations.

mad_image

metric_tradeoff_lambda

minmax

optimized_metric

optimized_metric_loss

optimizer

pixel_change_norm

L2 norm change in pixel values over iterations.

range_penalty_lambda

reference_metric

reference_metric_loss

saved_mad_image

store_progress

Methods

load(file_path[, map_location])	Load all relevant stuff from a .pt file.
objective_function([mad_image, image])	Compute the MADCompetition synthesis loss.
save(file_path)	Save all relevant variables in .pt file.
synthesize([max_iter, optimizer, scheduler, ...])	Synthesize a MAD image.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.

property image

property initial_image

load(file_path, map_location=None, **pickle_load_args)

Load all relevant stuff from a .pt file.

This should be called by an initialized MADCompetition object – we will ensure that image, metric_tradeoff_lambda, range_penalty_lambda, allowed_range, minmax are all identical, and that reference_metric and optimize_metric return identical values.

Note this operates in place and so doesn’t return anything.

Parameters:

• file_path (str) – The path to load the synthesis object from

• map_location (str, optional) – map_location argument to pass to torch.load. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device

• pickle_load_args – any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Examples

>>> mad = po.synth.MADCompetition(img, model)
>>> mad.synthesize(max_iter=10, store_progress=True)
>>> mad.save('mad.pt')
>>> mad_copy = po.synth.MADCompetition(img, model)
>>> mad_copy.load('mad.pt')

Note that you must create a new instance of the Synthesis object and then load.

property mad_image

property metric_tradeoff_lambda

property minmax

objective_function(mad_image=None, image=None)

Compute the MADCompetition synthesis loss.

This computes:

\[\begin{split}t L_1(x, \hat{x}) &+ \lambda_1 [L_2(x, x+\epsilon) - L_2(x, \hat{x})]^2 \\ &+ \lambda_2 \mathcal{B}(\hat{x})\end{split}\]

where \(t\) is 1 if self.minmax is 'min' and -1 if it’s 'max', \(L_1\) is self.optimized_metric, \(L_2\) is self.reference_metric, \(x\) is self.image, \(\hat{x}\) is self.mad_image, \(\epsilon\) is the initial noise, \(\mathcal{B}\) is the quadratic bound penalty, \(\lambda_1\) is self.metric_tradeoff_lambda and \(\lambda_2\) is self.range_penalty_lambda.

Parameters:

• mad_image (Tensor | None) – Proposed mad_image, \(\hat{x}\) in the above equation. If None, use self.mad_image.

• image (Tensor | None) – Proposed image, \(x\) in the above equation. If None, use self.image.

Return type:

loss

property optimized_metric

property optimized_metric_loss

property reference_metric

property reference_metric_loss

save(file_path)

Save all relevant variables in .pt file.

Note that if store_progress is True, this will probably be very large.

See load docstring for an example of use.

Parameters:

file_path (str) – The path to save the MADCompetition object to

property saved_mad_image

synthesize(max_iter=100, optimizer=None, scheduler=None, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50)

Synthesize a MAD image.

Update the pixels of initial_image to maximize or minimize (depending on the value of minmax) the value of optimized_metric(image, mad_image) while keeping the value of reference_metric(image, mad_image) constant.

We run this until either we reach max_iter or the change over the past stop_iters_to_check iterations is less than stop_criterion, whichever comes first

Parameters:

• max_iter (int) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).

• optimizer (Optimizer | None) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.

• scheduler (_LRScheduler | None) – The learning rate scheduler to use. If None, we don’t use one.

• store_progress (bool | int) – Whether we should store the representation of the MAD image in progress on every iteration. If False, we don’t save anything. If True, we save every iteration. If an int, we save every store_progress iterations (note then that 0 is the same as False and 1 the same as True).

• stop_criterion (float) – If the loss over the past stop_iters_to_check has changed less than stop_criterion, we terminate synthesis.

• stop_iters_to_check (int) – How many iterations back to check in order to see if the loss has stopped decreasing (for stop_criterion).

to(*args, **kwargs)

Moves and/or casts the parameters and buffers.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to(), but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See below for examples.

Note

This method modifies the module in-place.

Args:

device (torch.device): the desired device of the parameters

and buffers in this module

dtype (torch.dtype): the desired floating point type of

the floating point parameters and buffers in this module

tensor (torch.Tensor): Tensor whose dtype and device are the desired

dtype and device for all parameters and buffers in this module

plenoptic.synthesize.mad_competition.animate(mad, framerate=10, batch_idx=0, channel_idx=None, zoom=None, fig=None, axes_idx={}, figsize=None, included_plots=['display_mad_image', 'plot_loss', 'plot_pixel_values'], width_ratios={})

Animate synthesis progress.

This is essentially the figure produced by mad.plot_synthesis_status animated over time, for each stored iteration.

This functions returns a matplotlib FuncAnimation object. See our documentation (e.g., [Quickstart](https://docs.plenoptic.org/docs/branch/main/tutorials/00_quickstart.html)) for examples on how to view it in a Jupyter notebook. In order to save, use anim.save(filename). In either case, this can take a while and you’ll need the appropriate writer installed and on your path, e.g., ffmpeg, imagemagick, etc). See [matplotlib documentation](https://matplotlib.org/stable/api/animation_api.html) for more details.

Parameters:

• mad (MADCompetition) – MADCompetition object whose synthesis we want to animate.

• framerate (int) – How many frames a second to display.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).

• zoom (float | None) – How much to zoom in / enlarge the synthesized image, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.

• fig (Figure | None) – If None, create the figure from scratch. Else, should be an empty figure with enough axes (the expected use here is have same-size movies with different plots).

• axes_idx (dict[str, int]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys: 'mad_image', 'loss', 'pixel_values', 'misc'. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in 'misc'.

• figsize (tuple[float] | None) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)

• width_ratios (dict[str, float]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys: [‘display_mad_image’, ‘plot_loss’, ‘plot_pixel_values’] and floats specifying their relative width. Any not included will be assumed to be 1.

Returns:

The animation object. In order to view, must convert to HTML or save.

Return type:

anim

Notes

By default, we use the ffmpeg backend, which requires that you have ffmpeg installed and on your path (https://ffmpeg.org/download.html). To use a different, use the matplotlib rcParams: matplotlib.rcParams[‘animation.writer’] = writer, see https://matplotlib.org/stable/api/animation_api.html#writer-classes for more details.

For displaying in a jupyter notebook, ffmpeg appears to be required.

plenoptic.synthesize.mad_competition.display_mad_image(mad, batch_idx=0, channel_idx=None, zoom=None, iteration=None, ax=None, title='MADCompetition', **kwargs)

Display MAD image.

You can specify what iteration to view by using the iteration arg. The default, None, shows the final one.

We use plenoptic.imshow to display the synthesized image and attempt to automatically find the most reasonable zoom value. You can override this value using the zoom arg, but remember that plenoptic.imshow is opinionated about the size of the resulting image and will throw an Exception if the axis created is not big enough for the selected zoom.

Parameters:

• mad (MADCompetition) – MADCompetition object whose MAD image we want to display.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we assume image is RGB(A) and show all channels.

• zoom (float | None) – How much to zoom in / enlarge the synthesized image, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ax (Axes | None) – Pre-existing axes for plot. If None, we call plt.gca().

• title (str) – Title of the axis.

• kwargs – Passed to plenoptic.imshow

Returns:

The matplotlib axes containing the plot.

Return type:

ax

plenoptic.synthesize.mad_competition.display_mad_image_all(mad_metric1_min, mad_metric2_min, mad_metric1_max, mad_metric2_max, metric1_name=None, metric2_name=None, zoom=1, **kwargs)

Display all MAD Competition images.

To generate a full set of MAD Competition images, you need four instances: one for minimizing and maximizing each metric. This helper function creates a figure to display the full set of images.

In addition to the four MAD Competition images, this also plots the initial image from mad_metric1_min, for comparison.

Note that all four MADCompetition instances must have the same image.

Parameters:

• mad_metric1_min (MADCompetition) – MADCompetition object that minimized the first metric.

• mad_metric2_min (MADCompetition) – MADCompetition object that minimized the second metric.

• mad_metric1_max (MADCompetition) – MADCompetition object that maximized the first metric.

• mad_metric2_max (MADCompetition) – MADCompetition object that maximized the second metric.

• metric1_name (str | None) – Name of the first metric. If None, we use the name of the optimized_metric function from mad_metric1_min.

• metric2_name (str | None) – Name of the second metric. If None, we use the name of the optimized_metric function from mad_metric2_min.

• zoom (int | float) – Ratio of display pixels to image pixels. See plenoptic.imshow for details.

• kwargs – Passed to plenoptic.imshow.

Returns:

Figure containing the images.

Return type:

fig

plenoptic.synthesize.mad_competition.plot_loss(mad, iteration=None, axes=None, **kwargs)

Plot metric losses.

Plots mad.optimized_metric_loss and mad.reference_metric_loss on two separate axes, over all iterations. Also plots a red dot at iteration, to highlight the loss there. If iteration=None, then the dot will be at the final iteration.

Parameters:

• mad (MADCompetition) – MADCompetition object whose loss we want to plot.

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• axes (list[Axes] | Axes | None) – Pre-existing axes for plot. If a list of axes, must be the two axes to use for this plot. If a single axis, we’ll split it in half horizontally. If None, we call plt.gca().

• kwargs – passed to plt.plot

Returns:

The matplotlib axes containing the plot.

Return type:

axes

Notes

We plot abs(mad.losses) because if we’re maximizing the synthesis metric, we minimized its negative. By plotting the absolute value, we get them all on the same scale.

plenoptic.synthesize.mad_competition.plot_loss_all(mad_metric1_min, mad_metric2_min, mad_metric1_max, mad_metric2_max, metric1_name=None, metric2_name=None, metric1_kwargs={'c': 'C0'}, metric2_kwargs={'c': 'C1'}, min_kwargs={'linestyle': '--'}, max_kwargs={'linestyle': '-'}, figsize=(10, 5))

Plot loss for full set of MAD Competiton instances.

To generate a full set of MAD Competition images, you need four instances: one for minimizing and maximizing each metric. This helper function creates a two-axis figure to display the loss for this full set.

Note that all four MADCompetition instances must have the same image.

Parameters:

• mad_metric1_min (MADCompetition) – MADCompetition object that minimized the first metric.

• mad_metric2_min (MADCompetition) – MADCompetition object that minimized the second metric.

• mad_metric1_max (MADCompetition) – MADCompetition object that maximized the first metric.

• mad_metric2_max (MADCompetition) – MADCompetition object that maximized the second metric.

• metric1_name (str | None) – Name of the first metric. If None, we use the name of the optimized_metric function from mad_metric1_min.

• metric2_name (str | None) – Name of the second metric. If None, we use the name of the optimized_metric function from mad_metric2_min.

• metric1_kwargs (dict) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where the first metric was being optimized.

• metric2_kwargs (dict) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where the second metric was being optimized.

• min_kwargs (dict) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where optimized_metric was being minimized.

• max_kwargs (dict) – Dictionary of arguments to pass to matplotlib.pyplot.plot to identify synthesis instance where optimized_metric was being maximized.

• figsize – Size of the figure we create.

Returns:

Figure containing the plot.

Return type:

fig

plenoptic.synthesize.mad_competition.plot_pixel_values(mad, batch_idx=0, channel_idx=None, iteration=None, ylim=False, ax=None, **kwargs)

Plot histogram of pixel values of reference and MAD images.

As a way to check the distributions of pixel intensities and see if there’s any values outside the allowed range

Parameters:

• mad (MADCompetition) – MADCompetition object with the images whose pixel values we want to compare.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) images).

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ylim (Union[tuple[float], Literal[False]]) – if tuple, the ylimit to set for this axis. If False, we leave it untouched

• ax (Axes | None) – Pre-existing axes for plot. If None, we call plt.gca().

• kwargs – passed to plt.hist

Returns:

Creates axes.

Return type:

ax

plenoptic.synthesize.mad_competition.plot_synthesis_status(mad, batch_idx=0, channel_idx=None, iteration=None, vrange='indep1', zoom=None, fig=None, axes_idx={}, figsize=None, included_plots=['display_mad_image', 'plot_loss', 'plot_pixel_values'], width_ratios={})

Make a plot showing synthesis status.

We create several subplots to analyze this. By default, we create two subplots on a new figure: the first one contains the MAD image and the second contains the loss.

There is an optional additional plot: pixel_values, a histogram of pixel values of the synthesized and target images.

All of these (including the default plots) can be toggled using their corresponding boolean flags, and can be created separately using the method with the name plot_{flag}.

Parameters:

• mad (MADCompetition) – MADCompetition object whose status we want to plot.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• vrange (tuple[float] | str) – The vrange option to pass to display_mad_image(). See docstring of imshow for possible values.

• zoom (float | None) – How much to zoom in / enlarge the synthesized image, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.

• fig (Figure | None) – if None, we create a new figure. otherwise we assume this is an empty figure that has the appropriate size and number of subplots

• axes_idx (dict[str, int]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys: 'mad_image', 'loss', 'pixel_values', 'misc'. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in 'misc'.

• figsize (tuple[float] | None) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)

• included_plots (list[str]) – Which plots to include. Must be some subset of 'display_mad_image', 'plot_loss', 'plot_pixel_values'.

• width_ratios (dict[str, float]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys: [‘display_mad_image’, ‘plot_loss’, ‘plot_pixel_values’] and floats specifying their relative width. Any not included will be assumed to be 1.

Return type:

tuple[Figure, dict[str, int]]

Returns:

• fig – The figure containing this plot

• axes_idx – Dictionary giving index of each plot.

plenoptic.synthesize.metamer module

Synthesize model metamers.

class plenoptic.synthesize.metamer.Metamer(image, model, loss_function=<function mse>, range_penalty_lambda=0.1, allowed_range=(0, 1), initial_image=None)

Bases: OptimizedSynthesis

Synthesize metamers for image-computable differentiable models.

Following the basic idea in [1], this class creates a metamer for a given model on a given image. We start with initial_image and iteratively adjust the pixel values so as to match the representation of the metamer and image.

All saved_ attributes are initialized as empty lists and will be non-empty if the store_progress arg to synthesize() is not False. They will be appended to on every iteration if store_progress=True or every store_progress iterations if it’s an int.

Parameters:

• image (Tensor) – A 4d tensor, this is the image whose representation we wish to match. If this is not a tensor, we try to cast it as one.

• model (Module) – A visual model, see Metamer notebook for more details

• loss_function (Callable[[Tensor, Tensor], Tensor]) – the loss function to use to compare the representations of the models in order to determine their loss. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the Metamer object (i.e., it must be one of our built-in functions or defined using a def statement)

• range_penalty_lambda (float) – strength of the regularizer that enforces the allowed_range. Must be non-negative.

• allowed_range (tuple[float, float]) – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.

• initial_image (Tensor | None) – 4d Tensor to initialize our metamer with. If None, will draw a sample of uniform noise within allowed_range.

target_representation

Whatever is returned by model(image), this is what we match in order to create a metamer

Type:

torch.Tensor

metamer

The metamer. This may be unfinished depending on how many iterations we’ve run for.

Type:

torch.Tensor

losses

A list of our loss over iterations.

Type:

list

gradient_norm

A list of the gradient’s L2 norm over iterations.

Type:

list

pixel_change_norm

A list containing the L2 norm of the pixel change over iterations (pixel_change_norm[i] is the pixel change norm in metamer between iterations i and i-1).

Type:

list

saved_metamer

Saved self.metamer for later examination.

Type:

torch.Tensor

References

[1]

J Portilla and E P Simoncelli. A Parametric Texture Model based on Joint Statistics of Complex Wavelet Coefficients. Int’l Journal of Computer Vision. 40(1):49-71, October, 2000. https://www.cns.nyu.edu/~eero/ABSTRACTS/portilla99-abstract.html https://www.cns.nyu.edu/~lcv/texture/

Attributes:

allowed_range

gradient_norm

Synthesis gradient’s L2 norm over iterations.

image

losses

Synthesis loss over iterations.

metamer

model

optimizer

pixel_change_norm

L2 norm change in pixel values over iterations.

range_penalty_lambda

saved_metamer

store_progress

target_representation

Model representation of image, the goal of synthesis is for model(metamer) to match this value.

Methods

load(file_path[, map_location])	Load all relevant stuff from a .pt file.
objective_function([metamer_representation, ...])	Compute the metamer synthesis loss.
save(file_path)	Save all relevant variables in .pt file.
synthesize([max_iter, optimizer, scheduler, ...])	Synthesize a metamer.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.

property image

load(file_path, map_location=None, **pickle_load_args)

Load all relevant stuff from a .pt file.

This should be called by an initialized Metamer object – we will ensure that image, target_representation (and thus model), and loss_function are all identical.

Note this operates in place and so doesn’t return anything.

Parameters:

• file_path (str) – The path to load the synthesis object from

• map_location (str, optional) – map_location argument to pass to torch.load. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device

• pickle_load_args – any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Examples

>>> metamer = po.synth.Metamer(img, model)
>>> metamer.synthesize(max_iter=10, store_progress=True)
>>> metamer.save('metamers.pt')
>>> metamer_copy = po.synth.Metamer(img, model)
>>> metamer_copy.load('metamers.pt')

Note that you must create a new instance of the Synthesis object and then load.

property metamer

property model

objective_function(metamer_representation=None, target_representation=None)

Compute the metamer synthesis loss.

This calls self.loss_function on metamer_representation and target_representation and then adds the weighted range penalty.

Parameters:

• metamer_representation (Tensor | None) – Model response to metamer. If None, we use self.model(self.metamer)

• target_representation (Tensor | None) – Model response to image. If None, we use self.target_representation.

Return type:

loss

save(file_path)

Save all relevant variables in .pt file.

Note that if store_progress is True, this will probably be very large.

See load docstring for an example of use.

Parameters:

file_path (str) – The path to save the metamer object to

property saved_metamer

synthesize(max_iter=100, optimizer=None, scheduler=None, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50)

Synthesize a metamer.

Update the pixels of initial_image until its representation matches that of image.

We run this until either we reach max_iter or the change over the past stop_iters_to_check iterations is less than stop_criterion, whichever comes first

Parameters:

• max_iter (int) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).

• optimizer (Optimizer | None) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.

• scheduler (_LRScheduler | None) – The learning rate scheduler to use. If None, we don’t use one.

• store_progress (bool | int) – Whether we should store the metamer image in progress on every iteration. If False, we don’t save anything. If True, we save every iteration. If an int, we save every store_progress iterations (note then that 0 is the same as False and 1 the same as True).

• stop_criterion (float) – If the loss over the past stop_iters_to_check has changed less than stop_criterion, we terminate synthesis.

• stop_iters_to_check (int) – How many iterations back to check in order to see if the loss has stopped decreasing (for stop_criterion).

property target_representation

Model representation of image, the goal of synthesis is for model(metamer) to match this value.

to(*args, **kwargs)

Moves and/or casts the parameters and buffers.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to(), but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See below for examples.

Note

This method modifies the module in-place.

Args:

device (torch.device): the desired device of the parameters

and buffers in this module

dtype (torch.dtype): the desired floating point type of

the floating point parameters and buffers in this module

tensor (torch.Tensor): Tensor whose dtype and device are the desired

dtype and device for all parameters and buffers in this module

class plenoptic.synthesize.metamer.MetamerCTF(image, model, loss_function=<function mse>, range_penalty_lambda=0.1, allowed_range=(0, 1), initial_image=None, coarse_to_fine='together')

Bases: Metamer

Synthesize model metamers with coarse-to-fine synthesis.

This is a special case of Metamer, which uses the coarse-to-fine synthesis procedure described in [1]_: we start by updating metamer with respect to only a subset of the model’s representation (generally, that which corresponds to the lowest spatial frequencies), and changing which subset we consider over the course of synthesis. This is similar to optimizing with a blurred version of the objective function and gradually adding in finer details. It improves synthesis performance for some models.

Parameters:

• image (Tensor) – A 4d tensor, this is the image whose representation we wish to match. If this is not a tensor, we try to cast it as one.

• model (Module) – A visual model, see Metamer notebook for more details

• loss_function (Callable[[Tensor, Tensor], Tensor]) – the loss function to use to compare the representations of the models in order to determine their loss. Because of the limitations of pickle, you cannot use a lambda function for this if you wish to save the Metamer object (i.e., it must be one of our built-in functions or defined using a def statement)

• range_penalty_lambda (float) – strength of the regularizer that enforces the allowed_range. Must be non-negative.

• allowed_range (tuple[float, float]) – Range (inclusive) of allowed pixel values. Any values outside this range will be penalized.

• initial_image (Tensor | None) – 4d Tensor to initialize our metamer with. If None, will draw a sample of uniform noise within allowed_range.

• coarse_to_fine (Literal['together', 'separate']) –

  • ‘together’: start with the coarsest scale, then gradually add each finer scale.

  • ’separate’: compute the gradient with respect to each scale separately (ignoring the others), then with respect to all of them at the end.

  (see Metamer tutorial for more details).

target_representation

Whatever is returned by model(image), this is what we match in order to create a metamer

Type:

torch.Tensor

metamer

The metamer. This may be unfinished depending on how many iterations we’ve run for.

Type:

torch.Tensor

losses

A list of our loss over iterations.

Type:

list

gradient_norm

A list of the gradient’s L2 norm over iterations.

Type:

list

pixel_change_norm

A list containing the L2 norm of the pixel change over iterations (pixel_change_norm[i] is the pixel change norm in metamer between iterations i and i-1).

Type:

list

saved_metamer

Saved self.metamer for later examination.

Type:

torch.Tensor

scales

The list of scales in optimization order (i.e., from coarse to fine). Will be modified during the course of optimization.

Type:

list or None

scales_loss

The scale-specific loss at each iteration

Type:

list or None

scales_timing

Keys are the values found in scales, values are lists, specifying the iteration where we started and stopped optimizing this scale.

Type:

dict or None

scales_finished

List of scales that we’ve finished optimizing.

Type:

list or None

Attributes:

allowed_range

coarse_to_fine

gradient_norm

Synthesis gradient’s L2 norm over iterations.

image

losses

Synthesis loss over iterations.

metamer

model

optimizer

pixel_change_norm

L2 norm change in pixel values over iterations.

range_penalty_lambda

saved_metamer

scales

scales_finished

scales_loss

scales_timing

store_progress

target_representation

Model representation of image, the goal of synthesis is for model(metamer) to match this value.

Methods

load(file_path[, map_location])	Load all relevant stuff from a .pt file.
objective_function([metamer_representation, ...])	Compute the metamer synthesis loss.
save(file_path)	Save all relevant variables in .pt file.
synthesize([max_iter, optimizer, scheduler, ...])	Synthesize a metamer.
to(*args, **kwargs)	Moves and/or casts the parameters and buffers.

property coarse_to_fine

load(file_path, map_location=None, **pickle_load_args)

Load all relevant stuff from a .pt file.

This should be called by an initialized Metamer object – we will ensure that image, target_representation (and thus model), and loss_function are all identical.

Note this operates in place and so doesn’t return anything.

Parameters:

• file_path (str) – The path to load the synthesis object from

• map_location (str, optional) – map_location argument to pass to torch.load. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device

• pickle_load_args – any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Examples

>>> metamer = po.synth.Metamer(img, model)
>>> metamer.synthesize(max_iter=10, store_progress=True)
>>> metamer.save('metamers.pt')
>>> metamer_copy = po.synth.Metamer(img, model)
>>> metamer_copy.load('metamers.pt')

Note that you must create a new instance of the Synthesis object and then load.

property scales

property scales_finished

property scales_loss

property scales_timing

synthesize(max_iter=100, optimizer=None, scheduler=None, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50, change_scale_criterion=0.01, ctf_iters_to_check=50)

Synthesize a metamer.

Update the pixels of initial_image until its representation matches that of image.

We run this until either we reach max_iter or the change over the past stop_iters_to_check iterations is less than stop_criterion, whichever comes first

Parameters:

• max_iter (int) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).

• optimizer (Optimizer | None) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, this must be None and we reuse the previous optimizer.

• scheduler (_LRScheduler | None) – The learning rate scheduler to use. If None, we don’t use one.

• store_progress (bool | int) – Whether we should store the metamer image in progress on every iteration. If False, we don’t save anything. If True, we save every iteration. If an int, we save every store_progress iterations (note then that 0 is the same as False and 1 the same as True).

• stop_criterion (float) – If the loss over the past stop_iters_to_check has changed less than stop_criterion, we terminate synthesis.

• stop_iters_to_check (int) – How many iterations back to check in order to see if the loss has stopped decreasing (for stop_criterion).

• change_scale_criterion (float | None) – Scale-specific analogue of change_scale_criterion: we consider a given scale finished (and move onto the next) if the loss has changed less than this in the past ctf_iters_to_check iterations. If None, we’ll change scales as soon as we’ve spent ctf_iters_to_check on a given scale

• ctf_iters_to_check (int) – Scale-specific analogue of stop_iters_to_check: how many iterations back in order to check in order to see if we should switch scales.

plenoptic.synthesize.metamer.animate(metamer, framerate=10, batch_idx=0, channel_idx=None, ylim=None, vrange=(0, 1), zoom=None, plot_representation_error_as_rgb=False, fig=None, axes_idx={}, figsize=None, included_plots=['display_metamer', 'plot_loss', 'plot_representation_error'], width_ratios={})

Animate synthesis progress.

This is essentially the figure produced by metamer.plot_synthesis_status animated over time, for each stored iteration.

This functions returns a matplotlib FuncAnimation object. See our documentation (e.g., [Quickstart](https://docs.plenoptic.org/docs/branch/main/tutorials/00_quickstart.html)) for examples on how to view it in a Jupyter notebook. In order to save, use anim.save(filename). In either case, this can take a while and you’ll need the appropriate writer installed and on your path, e.g., ffmpeg, imagemagick, etc). See [matplotlib documentation](https://matplotlib.org/stable/api/animation_api.html) for more details.

Parameters:

• metamer (Metamer) – Metamer object whose synthesis we want to animate.

• framerate (int) – How many frames a second to display.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).

• ylim (Union[str, None, tuple[float, float], Literal[False]]) –

  The y-limits of the representation_error plot:

  • If a tuple, then this is the ylim of all plots

  • If None, then all plots have the same limits, all symmetric about 0 with a limit of np.abs(representation_error).max() (for the initial representation_error)

  • If False, don’t modify limits.

  • If a string, must be ‘rescale’ or of the form ‘rescaleN’, where N can be any integer. If ‘rescaleN’, we rescale the limits every N frames (we rescale as if ylim = None). If ‘rescale’, then we do this 10 times over the course of the animation

• vrange (tuple[float, float] | str) – The vrange option to pass to display_metamer(). See docstring of imshow for possible values.

• zoom (float | None) – How much to zoom in / enlarge the metamer, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.

• plot_representation_error_as_rgb (bool) – The representation can be image-like with multiple channels, and we have no way to determine whether it should be represented as an RGB image or not, so the user must set this flag to tell us. It will be ignored if the representation doesn’t look image-like or if the model has its own plot_representation_error() method. Else, it will be passed to po.imshow(), see that methods docstring for details. since plot_synthesis_status normally sets it up for us

• fig (Figure | None) – If None, create the figure from scratch. Else, should be an empty figure with enough axes (the expected use here is have same-size movies with different plots).

• axes_idx (dict[str, int]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys: 'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values', 'misc'. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in 'misc'.

• figsize (tuple[float, float] | None) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)

• included_plots (list[str]) – Which plots to include. Must be some subset of 'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values'.

• width_ratios (dict[str, float]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys: 'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values' and floats specifying their relative width. Any not included will be assumed to be 1.

Returns:

The animation object. In order to view, must convert to HTML or save.

Return type:

anim

Notes

By default, we use the ffmpeg backend, which requires that you have ffmpeg installed and on your path (https://ffmpeg.org/download.html). To use a different, use the matplotlib rcParams: matplotlib.rcParams[‘animation.writer’] = writer, see https://matplotlib.org/stable/api/animation_api.html#writer-classes for more details.

For displaying in a jupyter notebook, ffmpeg appears to be required.

plenoptic.synthesize.metamer.display_metamer(metamer, batch_idx=0, channel_idx=None, zoom=None, iteration=None, ax=None, **kwargs)

Display metamer.

You can specify what iteration to view by using the iteration arg. The default, None, shows the final one.

We use plenoptic.imshow to display the metamer and attempt to automatically find the most reasonable zoom value. You can override this value using the zoom arg, but remember that plenoptic.imshow is opinionated about the size of the resulting image and will throw an Exception if the axis created is not big enough for the selected zoom.

Parameters:

• metamer (Metamer) – Metamer object whose synthesized metamer we want to display.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we assume image is RGB(A) and show all channels.

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ax (Axes | None) – Pre-existing axes for plot. If None, we call plt.gca().

• zoom (float | None) – How much to zoom in / enlarge the metamer, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.

• kwargs – Passed to plenoptic.imshow

Returns:

The matplotlib axes containing the plot.

Return type:

ax

plenoptic.synthesize.metamer.plot_loss(metamer, iteration=None, ax=None, **kwargs)

Plot synthesis loss with log-scaled y axis.

Plots metamer.losses over all iterations. Also plots a red dot at iteration, to highlight the loss there. If iteration=None, then the dot will be at the final iteration.

Parameters:

• metamer (Metamer) – Metamer object whose loss we want to plot.

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ax (Axes | None) – Pre-existing axes for plot. If None, we call plt.gca().

• kwargs – passed to plt.semilogy

Returns:

The matplotlib axes containing the plot.

Return type:

ax

plenoptic.synthesize.metamer.plot_pixel_values(metamer, batch_idx=0, channel_idx=None, iteration=None, ylim=False, ax=None, **kwargs)

Plot histogram of pixel values of target image and its metamer.

As a way to check the distributions of pixel intensities and see if there’s any values outside the allowed range

Parameters:

• metamer (Metamer) – Metamer object with the images whose pixel values we want to compare.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) images).

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ylim (Union[tuple[float, float], Literal[False]]) – if tuple, the ylimit to set for this axis. If False, we leave it untouched

• ax (Axes | None) – Pre-existing axes for plot. If None, we call plt.gca().

• kwargs – passed to plt.hist

Returns:

Created axes.

Return type:

ax

plenoptic.synthesize.metamer.plot_representation_error(metamer, batch_idx=0, iteration=None, ylim=None, ax=None, as_rgb=False, **kwargs)

Plot distance ratio showing how close we are to convergence.

We plot _representation_error(metamer, iteration). For more details, see plenoptic.tools.display.plot_representation.

Parameters:

• metamer (Metamer) – Metamer object whose synthesized metamer we want to display.

• batch_idx (int) – Which index to take from the batch dimension

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ylim (Union[tuple[float, float], None, Literal[False]]) – If ylim is None, we sets the axes’ y-limits to be (-y_max, y_max), where y_max=np.abs(data).max(). If it’s False, we do nothing. If a tuple, we use that range.

• ax (Axes | None) – Pre-existing axes for plot. If None, we call plt.gca().

• as_rgb (bool, optional) – The representation can be image-like with multiple channels, and we have no way to determine whether it should be represented as an RGB image or not, so the user must set this flag to tell us. It will be ignored if the response doesn’t look image-like or if the model has its own plot_representation_error() method. Else, it will be passed to po.imshow(), see that methods docstring for details.

• kwargs – Passed to metamer.model.forward

Returns:

List of created axes

Return type:

axes

plenoptic.synthesize.metamer.plot_synthesis_status(metamer, batch_idx=0, channel_idx=None, iteration=None, ylim=None, vrange='indep1', zoom=None, plot_representation_error_as_rgb=False, fig=None, axes_idx={}, figsize=None, included_plots=['display_metamer', 'plot_loss', 'plot_representation_error'], width_ratios={})

Make a plot showing synthesis status.

We create several subplots to analyze this. By default, we create three subplots on a new figure: the first one contains the synthesized metamer, the second contains the loss, and the third contains the representation error.

There is an optional additional plot: plot_pixel_values, a histogram of pixel values of the metamer and target image.

The plots to include are specified by including their name in the included_plots list. All plots can be created separately using the method with the same name.

Parameters:

• metamer (Metamer) – Metamer object whose status we want to plot.

• batch_idx (int) – Which index to take from the batch dimension

• channel_idx (int | None) – Which index to take from the channel dimension. If None, we use all channels (assumed use-case is RGB(A) image).

• iteration (int | None) – Which iteration to display. If None, the default, we show the most recent one. Negative values are also allowed.

• ylim (Union[tuple[float, float], None, Literal[False]]) – The ylimit to use for the representation_error plot. We pass this value directly to plot_representation_error

• vrange (tuple[float, float] | str) – The vrange option to pass to display_metamer(). See docstring of imshow for possible values.

• zoom (float | None) – How much to zoom in / enlarge the metamer, the ratio of display pixels to image pixels. If None (the default), we attempt to find the best value ourselves.

• plot_representation_error_as_rgb (bool, optional) – The representation can be image-like with multiple channels, and we have no way to determine whether it should be represented as an RGB image or not, so the user must set this flag to tell us. It will be ignored if the response doesn’t look image-like or if the model has its own plot_representation_error() method. Else, it will be passed to po.imshow(), see that methods docstring for details.

• fig (Figure | None) – if None, we create a new figure. otherwise we assume this is an empty figure that has the appropriate size and number of subplots

• axes_idx (dict[str, int]) – Dictionary specifying which axes contains which type of plot, allows for more fine-grained control of the resulting figure. Probably only helpful if fig is also defined. Possible keys: 'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values', 'misc'. Values should all be ints. If you tell this function to create a plot that doesn’t have a corresponding key, we find the lowest int that is not already in the dict, so if you have axes that you want unchanged, place their idx in 'misc'.

• figsize (tuple[float, float] | None) – The size of the figure to create. It may take a little bit of playing around to find a reasonable value. If None, we attempt to make our best guess, aiming to have each axis be of size (5, 5)

• included_plots (list[str]) – Which plots to include. Must be some subset of 'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values'.

• width_ratios (dict[str, float]) – By default, all plots axes will have the same width. To change that, specify their relative widths using the keys: 'display_metamer', 'plot_loss', 'plot_representation_error', 'plot_pixel_values' and floats specifying their relative width. Any not included will be assumed to be 1.

Return type:

tuple[Figure, dict[str, int]]

Returns:

• fig – The figure containing this plot

• axes_idx – Dictionary giving index of each plot.

plenoptic.synthesize.simple_metamer module

Simple Metamer Class

class plenoptic.synthesize.simple_metamer.SimpleMetamer(image, model)

Bases: Synthesis

Simple version of metamer synthesis.

This doesn’t have any of the bells and whistles of the full Metamer class, but does perform basic metamer synthesis: given a target image and a model, synthesize a new image (initialized with uniform noise) that has the same model output.

This is meant as a demonstration of the basic logic of synthesis.

Parameters:

• image (Tensor) – A 4d tensor, this is the image whose model representation we wish to match.

• model (Module) – The visual model whose representation we wish to match.

Notes

Deprecated since version 1.1.0: Use plenoptic.synthesize.metamer.Metamer instead

Methods

load(file_path[, map_location])	Load all relevant attributes from a .pt file.
save(file_path)	Save all relevant (non-model) variables in .pt file.
synthesize([max_iter, optimizer])	Synthesize a simple metamer.
to(*args, **kwargs)	Move and/or cast the parameters and buffers.

load(file_path, map_location=None)

Load all relevant attributes from a .pt file.

Note this operates in place and so doesn’t return anything.

Parameters:

file_path (str) – The path to load the synthesis object from

save(file_path)

Save all relevant (non-model) variables in .pt file.

Parameters:

file_path (str) – The path to save the SimpleMetamer object to.

synthesize(max_iter=100, optimizer=None)

Synthesize a simple metamer.

If called multiple times, will continue where we left off.

Parameters:

• max_iter (int) – Number of iterations to run synthesis for.

• optimizer (None | Optimizer) – The optimizer to use. If None and this is the first time calling synthesize, we use Adam(lr=.01, amsgrad=True); if synthesize has been called before, we reuse the previous optimizer.

Returns:

The synthesized metamer

Return type:

metamer

to(*args, **kwargs)

Move and/or cast the parameters and buffers.

This can be called as .. function:: to(device=None, dtype=None, non_blocking=False) .. function:: to(dtype, non_blocking=False) .. function:: to(tensor, non_blocking=False) Its signature is similar to torch.Tensor.to(), but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. When calling this method to move tensors to a CUDA device, items in attrs that start with “saved_” will not be moved. .. note:

This method modifies the module in-place.

Args:

device (torch.device): the desired device of the parameters

and buffers in this module

dtype (torch.dtype): the desired floating point type of

the floating point parameters and buffers in this module

tensor (torch.Tensor): Tensor whose dtype and device are the desired

dtype and device for all parameters and buffers in this module

attrs (list): list of strs containing the attributes of

this object to move to the specified device/dtype

Returns:

Module: self

plenoptic.synthesize.synthesis module

abstract synthesis super-class.

class plenoptic.synthesize.synthesis.OptimizedSynthesis(range_penalty_lambda=0.1, allowed_range=(0, 1))

Bases: Synthesis

Abstract super-class for synthesis objects that use optimization.

The primary difference between this and the generic Synthesis class is that these will use an optimizer object to iteratively update their output.

Attributes:

allowed_range

gradient_norm

Synthesis gradient’s L2 norm over iterations.

losses

Synthesis loss over iterations.

optimizer

pixel_change_norm

L2 norm change in pixel values over iterations.

range_penalty_lambda

store_progress

Methods

load(file_path[, map_location, ...])	Load all relevant attributes from a .pt file.
objective_function()	How good is the current synthesized object.
save(file_path[, attrs])	Save all relevant (non-model) variables in .pt file.
synthesize()	Synthesize something.
to(*args[, attrs])	Moves and/or casts the parameters and buffers.

property allowed_range

property gradient_norm

Synthesis gradient’s L2 norm over iterations.

property losses

Synthesis loss over iterations.

abstract objective_function()

How good is the current synthesized object.

See plenoptic.tools.optim for some examples.

property optimizer

property pixel_change_norm

L2 norm change in pixel values over iterations.

property range_penalty_lambda

property store_progress

class plenoptic.synthesize.synthesis.Synthesis

Bases: ABC

Abstract super-class for synthesis objects.

All synthesis objects share a variety of similarities and thus need to have similar methods. Some of these can be implemented here and simply inherited, some of them will need to be different for each sub-class and thus are marked as abstract methods here

Methods

load(file_path[, map_location, ...])	Load all relevant attributes from a .pt file.
save(file_path[, attrs])	Save all relevant (non-model) variables in .pt file.
synthesize()	Synthesize something.
to(*args[, attrs])	Moves and/or casts the parameters and buffers.

load(file_path, map_location=None, check_attributes=[], check_loss_functions=[], **pickle_load_args)

Load all relevant attributes from a .pt file.

This should be called by an initialized Synthesis object – we will ensure that the attributes in the check_attributes arg all match in the current and loaded object.

Note this operates in place and so doesn’t return anything.

Parameters:

• file_path (str) – The path to load the synthesis object from

• map_location (str | None) – map_location argument to pass to torch.load. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device

• check_attributes (list[str]) – List of strings we ensure are identical in the current Synthesis object and the loaded one. Checking the model is generally not recommended, since it can be hard to do (checking callable objects is hard in Python) – instead, checking the base_representation should ensure the model hasn’t functinoally changed.

• check_loss_functions (list[str]) – Names of attributes that are loss functions and so must be checked specially – loss functions are callables, and it’s very difficult to check python callables for equality so, to get around that, we instead call the two versions on the same pair of tensors, and compare the outputs.

• pickle_load_args – any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

save(file_path, attrs=None)

Save all relevant (non-model) variables in .pt file.

If you leave attrs as None, we grab vars(self) and exclude ‘model’. This is probably correct, but the option is provided to override it just in case

Parameters:

• file_path (str) – The path to save the synthesis object to

• attrs (list or None, optional) – List of strs containing the names of the attributes of this object to save. See above for behavior if attrs is None.

abstract synthesize()

Synthesize something.

abstract to(*args, attrs=[], **kwargs)

Moves and/or casts the parameters and buffers. Similar to save, this is an abstract method only because you need to define the attributes to call to on.

This can be called as .. function:: to(device=None, dtype=None, non_blocking=False) .. function:: to(dtype, non_blocking=False) .. function:: to(tensor, non_blocking=False) Its signature is similar to torch.Tensor.to(), but only accepts floating point desired dtype s. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When non_blocking is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices. When calling this method to move tensors to a CUDA device, items in attrs that start with “saved_” will not be moved. .. note:

This method modifies the module in-place.

Args:

device (torch.device): the desired device of the parameters

and buffers in this module

dtype (torch.dtype): the desired floating point type of

the floating point parameters and buffers in this module

tensor (torch.Tensor): Tensor whose dtype and device are the desired

dtype and device for all parameters and buffers in this module

attrs (list): list of strs containing the attributes of

this object to move to the specified device/dtype

Module contents