plenoptic.Eigendistortion#

class plenoptic.Eigendistortion(image, model)[source]#

Synthesize eigendistortions induced by a model on a given input image.

Following the basic idea in [1], this class synthesizes image perturbations that are considered the most and least noticeable for a model on a given image. Because these are perturbations on the input image, they are local in pixel space, i.e., they do not change the pixels much.

Parameters:

image (Tensor) – Image to perturb. We currently do not support batches of images, as each image requires its own optimization, so either image.ndimension()==1 or image.shape[0]==1.
model (Module) – Torch model with defined forward and backward operations.

Notes

This is a method for comparing image representations in terms of their ability to explain perceptual sensitivity in humans. It estimates eigenvectors of the Fisher Information Matrix. A model, \(y = f(x)\), is a deterministic (and differentiable) mapping from the input pixels \(x \in \mathbb{R}^n\) to a mean output response vector \(y\in \mathbb{R}^m\), where we assume additive white Gaussian noise in the response space.

The Jacobian matrix at \(x\) is: \(J(x) = J = dydx\), where \(J\in\mathbb{R}^{m \times n}\) (i.e. output_dim x input_dim).

The matrix consists of all partial derivatives of the vector-valued function \(f\). The Fisher Information Matrix (FIM) at \(x\), under white Gaussian noise in the response space, is: \(F = J^T J\) It is a quadratic approximation of the discriminability of distortions relative to \(x\).

References

Methods

`compute_jacobian`()	Compute, cache, and return jacobian.
`load`(file_path[, map_location, ...])	Load all relevant stuff from a .pt file.
`save`(file_path)	Save all relevant variables in .pt file.
`synthesize`([method, k, max_iter, p, q, ...])	Compute eigendistortions of Fisher Information Matrix with given input image.
`to`(args, *kwargs)	Move and/or cast the parameters and buffers.

Attributes

`eigendistortions`	Eigendistortions, ordered by eigenvalue.
`eigenindex`	Index of each eigenvector/eigenvalue.
`eigenvalues`	Eigenvalues corresponding to each eigendistortion, in decreasing order.
`image`	Target image of eigendistortion synthesis.
`jacobian`	Jacobian matrix of `model` with respect to `image`.
`model`	The model for which the eigendistortions are synthesized.

compute_jacobian()[source]#

Compute, cache, and return jacobian.

If the jacobian has not been cached: compute, cache, and return.

If the jacobian has already been cached, we simply return it.

Return type:: Tensor
Returns:: J – Jacobian of representation with respect to input.
Warns:: UserWarning – If input dimensionality is greater than 1e4, in which case we believe that this calculation will take too long.

load(file_path, map_location=None, raise_on_checks=True, tensor_equality_atol=1e-08, tensor_equality_rtol=1e-05, **pickle_load_args)[source]#

Load all relevant stuff from a .pt file.

This must be called by a Eigendistortion object initialized just like the saved object.

Note this operates in place and so doesn’t return anything.

Changed in version 1.2: load behavior changed in a backwards-incompatible manner in order to compatible with breaking changes in torch 2.6.

Changed in version 2.0.0: Adds raise_on_checks argument.

Parameters:

file_path (str) – The path to load the synthesis object from.
map_location (str | None (default: None)) – Argument to pass to torch.load as map_location. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device.
raise_on_checks (bool (default: True)) – During load, we perform several checks to ensure that the saved object was initialized in the same way as the loading object. This is to ensure that the model, image, etc. are all the same and avoid unpleasant surprises. If True, we raise a ValueError if any of these checks fail. If False, we instead raise a LoadWarning. The intended use here is if you’re loading something that was saved with an older version of plenoptic and you’re sure that you’re doing everything correctly. Note that different devices or dtypes will always result in a ValueError. See raise_on_checks on the “Reproducibility and Compatibility” page of the documentation for more info. Additionally, note that, if the Eigendistortion object itself has changed, we cannot ensure that methods are the same – proceed at your own risk.
tensor_equality_atol (float (default: 1e-08)) – Absolute tolerance to use when checking for tensor equality during load, passed to torch.allclose. It may be necessary to increase if you are saving and loading on two machines with torch built by different cuda versions. Be careful when changing this! See torch.finfo for more details about floating point precision of different data types (especially, eps); if you have to increase this by more than 1 or 2 decades, then you are probably not dealing with a numerical issue.
tensor_equality_rtol (float (default: 1e-05)) – Relative tolerance to use when checking for tensor equality during load, passed to torch.allclose. It may be necessary to increase if you are saving and loading on two machines with torch built by different cuda versions. Be careful when changing this! See torch.finfo for more details about floating point precision of different data types (especially, eps); if you have to increase this by more than 1 or 2 decades, then you are probably not dealing with a numerical issue.
**pickle_load_args (Any) – Any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Raises:

ValueError – If synthesize has been called before this call to load.
ValueError – If the object saved at file_path is not a Eigendistortion object.
ValueError – If the saved and loading Eigendistortion objects have a different value for image.
ValueError – If the behavior of model is different between the saved and loading objects.

See also

examine_saved_synthesis: Examine metadata from saved object: pytorch and plenoptic versions, name of the synthesis object, shapes of tensors, etc.

Examples

>>> import plenoptic as po
>>> img = po.data.einstein()
>>> model = po.models.Gaussian(30).eval()
>>> po.remove_grad(model)
>>> eig = po.Eigendistortion(img, model)
>>> eig.synthesize(max_iter=5)
>>> eig.save("eig.pt")
>>> eig_copy = po.Eigendistortion(img, model)
>>> eig_copy.load("eig.pt")

save(file_path)[source]#

Save all relevant variables in .pt file.

See load docstring for an example of use.

Parameters:: file_path (str) – The path to save the Eigendistortion object to.

synthesize(method='power', k=1, max_iter=1000, p=5, q=2, stop_criterion=1e-07)[source]#

Compute eigendistortions of Fisher Information Matrix with given input image.

There are three potential ways of computing the eigendistortion for a model; all have the same interpretation. See method argument for details.

Parameters:

method (Literal['exact', 'power', 'randomized_svd'] (default: 'power')) – Eigensolver method. 'exact' tries to do eigendecomposition directly (not recommended for very large inputs). 'power' uses the power method to compute first and last eigendistortions, with maximum number of iterations dictated by n_steps. 'randomized_svd' uses randomized SVD to approximate the top k eigendistortions and their corresponding eigenvalues.
k (int (default: 1)) – How many vectors to return using block power method or svd.
max_iter (int (default: 1000)) – Maximum number of steps to run for method='power' in eigenvalue computation. Ignored for other methods.
p (int (default: 5)) – Oversampling parameter for randomized SVD. k+p vectors will be sampled, and k will be returned. See docstring of _synthesize_randomized_svd for more details including algorithm reference.
q (int (default: 2)) – Matrix power parameter for randomized SVD. This is an effective trick for the algorithm to converge to the correct eigenvectors when the eigenspectrum does not decay quickly. See _synthesize_randomized_svd for more details including algorithm reference.
stop_criterion (float (default: 1e-07)) – Used if method='power' to check for convergence. If the L2-norm of the eigenvalues has changed by less than this value from one iteration to the next, we terminate synthesis.

Raises:

ValueError – If method takes an illegal value.

Warns:

UserWarning – If method == "power" but the Jacobian size is greater than 1e6 (which depends on the number of elements in the model’s representation and input image), in which case we’re worried about running out of memory.

to(*args, **kwargs)[source]#

Move and/or cast the parameters and buffers.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to, but only accepts floating point desired dtype. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When on_blocking` is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See torch.nn.Module.to for examples.

Note

This method modifies the module in-place.

Parameters:

device (torch.device) – The desired device of the parameters and buffers in this module.
dtype (torch.dtype) – The desired floating point type of the floating point parameters and buffers in this module.
tensor (torch.Tensor) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module.

property eigendistortions: Tensor#

Eigendistortions, ordered by eigenvalue.

Eigendistortions are the eigenvectors of Fisher matrix, will have size Size((n_distortions, *image.shape[1:])).

property eigenindex: Tensor#: Index of each eigenvector/eigenvalue.

property eigenvalues: Tensor#: Eigenvalues corresponding to each eigendistortion, in decreasing order.

property image: Tensor#: Target image of eigendistortion synthesis.

property jacobian: Tensor#

Jacobian matrix of model with respect to image.

Only set when synthesize is run with method='exact'. Else, None.

property model: Module#: The model for which the eigendistortions are synthesized.