plenoptic.MADCompetition#

class plenoptic.MADCompetition(image, optimized_metric, reference_metric, minmax, metric_tradeoff_lambda=None, penalty_function=<function penalize_range>, penalty_lambda=0.1)[source]#

Synthesize a single maximally-differentiating image for two metrics.

Following the basic idea in [1], this class synthesizes a maximally-differentiating image for two given metrics, based on a given image. We start by adding noise to this image and then iteratively adjusting its pixels so as to either minimize or maximize optimized_metric while holding the value of reference_metric constant.

MADCompetiton accepts two metrics as its input. These should be callables that take two images and return a single number, and that number should be 0 if and only if the two images are identical (thus, the larger the number, the more different the two images).

Note that a full set of MAD Competition images consists of two pairs: a maximal and a minimal image for each metric. A single instantiation of MADCompetition will generate one of these four images.

Parameters:

image (Tensor) – A tensor, this is the image we use as the reference point.
optimized_metric (Module | Callable[[Tensor, Tensor], Tensor]) – The metric whose value you wish to minimize or maximize, which takes two tensors and returns a scalar.
reference_metric (Module | Callable[[Tensor, Tensor], Tensor]) – The metric whose value you wish to keep fixed, which takes two tensors and returns a scalar.
minmax (Literal['min', 'max']) – Whether you wish to minimize or maximize optimized_metric.
metric_tradeoff_lambda (float | None (default: None)) – Lambda to multiply by reference_metric loss and add to optimized_metric loss. If None, we pick a value so the two initial losses are approximately equal in magnitude.
penalty_function (Callable[[Tensor], Tensor] (default: <function penalize_range at 0x7f41412b2160>)) – A function applied to the metamer during optimization, that returns a scalar penalty to be minimized. By penalizing certain properties of the image, like pixels values outside an allowed range, we can constrain those image properties. See Regularization penalty in the documentation for details and examples.
penalty_lambda (float (default: 0.1)) – Weight of the penalty term. Must be non-negative.

References

Methods

`get_progress`(iteration[, iteration_selection])	Return dictionary summarizing synthesis progress at `iteration`.
`load`(file_path[, map_location, ...])	Load all relevant stuff from a .pt file.
`objective_function`([mad_image, image])	Compute the MADCompetition synthesis loss.
`save`(file_path)	Save all relevant variables in .pt file.
`setup`([initial_noise, optimizer, ...])	Initialize the MAD image, optimizer, and scheduler.
`synthesize`([max_iter, store_progress, ...])	Synthesize a MAD image.
`to`(args, *kwargs)	Move and/or casts the parameters and buffers.

Attributes

`gradient_norm`	Optimization gradient's L2 norm over iterations.
`image`	The reference image for this MAD Competition.
`initial_image`	Initial image for MAD Competition.
`losses`	Optimization loss over iterations.
`mad_image`	Maximally-differentiating image, the parameter we are optimizing.
`metric_tradeoff_lambda`	Tradeoff between the two metrics in synthesis loss.
`minmax`	Whether we are minimizing or maximizing `optimized_metric`.
`optimized_metric`	The metric whose value we are minimizing or maximizing.
`optimized_metric_loss`	`optimized_metric` loss over iterations.
`optimizer`	Torch optimizer object which updates the synthesis target.
`penalties`	Penalty function output over iterations.
`penalty_function`	Callable which penalizes additional properties of the synthesized image.
`penalty_lambda`	Magnitude of the regularization weight.
`pixel_change_norm`	L2 norm change in pixel values over iterations.
`reference_metric`	The metric whose value we are keeping constant.
`reference_metric_loss`	`reference_metric` loss over iterations.
`saved_mad_image`	`mad_image`, cached over time for later examination.
`scheduler`	Learning rate scheduler which adjusts optimizer learning rate.
`store_progress`	How often we are caching progress.

get_progress(iteration, iteration_selection='round')[source]#

Return dictionary summarizing synthesis progress at iteration.

This returns a dictionary containing info from losses, pixel_change_norm, gradient_norm, penalties, and saved_mad_image corresponding to iteration. If synthesis was run with store_progress=False (and so we did not cache anything in saved_mad_image), then that key will be missing. If synthesis was run with store_progress>1, we will grab the corresponding tensor from saved_mad_image, with behavior determined by iteration_selection.

The returned dictionary will additionally contain the keys:

"iteration": the (0-indexed positive) synthesis iteration that the values for losses, pixel_change_norm, penalties and gradient_norm come from.
If self.store_progress, "store_progress_iteration": the (0-indexed positive) synthesis iteration that the value for saved_mad_image comes from.

Note that for the most recent iteration (iteration=-1 or iteration=None or iteration==len(self.losses)-1), we do not have values for pixel_change_norm or gradient_norm, since in this case we are showing the loss and value for the current MAD image.

Parameters:

iteration (int) – Synthesis iteration to summarize. If None, grab the most recent. Negative values are allowed.
iteration_selection (Literal['floor', 'ceiling', 'round'] (default: 'round')) –
How to select the relevant iteration from saved_mad_image when the request iteration wasn’t stored.

When synthesis was run with store_progress=n (where n>1), MAD images are only saved every n iterations. If you request an iteration where a MAD image wasn’t saved, this determines which available iteration is used instead:
- "floor": use the closest saved iteration before the requested one.
- "ceiling": use the closest saved iteration after the requested one.
- "round": use the closest saved iteration.

Return type:

dict

Returns:

progress_info – Dictionary summarizing synthesis progress.

Raises:

IndexError – If iteration takes an illegal value.

Warns:

UserWarning – If the iteration used for saved_mad_image is not the same as the argument iteration (because e.g., you set iteration=3 but self.store_progress=2).

load(file_path, map_location=None, raise_on_checks=True, tensor_equality_atol=1e-08, tensor_equality_rtol=1e-05, **pickle_load_args)[source]#

Load all relevant stuff from a .pt file.

This must be called by a MADCompetition object initialized just like the saved object.

Note this operates in place and so doesn’t return anything.

Changed in version 1.2: load behavior changed in a backwards-incompatible manner in order to compatible with breaking changes in torch 2.6.

Changed in version 2.0.0: Adds raise_on_checks argument.

Parameters:

file_path (str) – The path to load the synthesis object from.
map_location (str | None (default: None)) – Argument to pass to torch.load as map_location. If you save stuff that was being run on a GPU and are loading onto a CPU, you’ll need this to make sure everything lines up properly. This should be structured like the str you would pass to torch.device.
raise_on_checks (bool (default: True)) – During load, we perform several checks to ensure that the saved object was initialized in the same way as the loading object. This is to ensure that the model, image, etc. are all the same and avoid unpleasant surprises. If True, we raise a ValueError if any of these checks fail. If False, we instead raise a LoadWarning. The intended use here is if you’re loading something that was saved with an older version of plenoptic and you’re sure that you’re doing everything correctly. Note that different devices or dtypes will always result in a ValueError. See raise_on_checks on the “Reproducibility and Compatibility” page of the documentation for more info. Additionally, note that, if the MADCompetition object itself has changed, we cannot ensure that methods are the same – proceed at your own risk.
tensor_equality_atol (float (default: 1e-08)) – Absolute tolerance to use when checking for tensor equality during load, passed to torch.allclose. It may be necessary to increase if you are saving and loading on two machines with torch built by different cuda versions. Be careful when changing this! See torch.finfo for more details about floating point precision of different data types (especially, eps); if you have to increase this by more than 1 or 2 decades, then you are probably not dealing with a numerical issue.
tensor_equality_rtol (float (default: 1e-05)) – Relative tolerance to use when checking for tensor equality during load, passed to torch.allclose. It may be necessary to increase if you are saving and loading on two machines with torch built by different cuda versions. Be careful when changing this! See torch.finfo for more details about floating point precision of different data types (especially, eps); if you have to increase this by more than 1 or 2 decades, then you are probably not dealing with a numerical issue.
**pickle_load_args (Any) – Any additional kwargs will be added to pickle_module.load via torch.load, see that function’s docstring for details.

Raises:

ValueError – If setup or synthesize has been called before this call to load.
ValueError – If the object saved at file_path is not a MADCompetition object.
ValueError – If the saved and loading MADCompetition objects have a different value for any of image, penalty_lambda, metric_tradeoff_lambda, or minmax.
ValueError – If the behavior of optimized_metric or reference_metric is different between the saved and loading objects.

Warns:

UserWarning – If setup will need to be called after load, to finish initializing optimizer or scheduler.

See also

examine_saved_synthesis: Examine metadata from saved object: pytorch and plenoptic versions, name of the synthesis object, shapes of tensors, etc.

Examples

>>> import plenoptic as po
>>> img = po.data.einstein()
>>> def ds_ssim(x, y):
...     return 1 - po.metric.ssim(x, y)
>>> mad = po.MADCompetition(
...     img, po.metric.mse, ds_ssim, "min", metric_tradeoff_lambda=10
... )
>>> mad.synthesize(max_iter=5, store_progress=True)
>>> mad.save("mad.pt")
>>> mad_copy = po.MADCompetition(
...     img, po.metric.mse, ds_ssim, "min", metric_tradeoff_lambda=10
... )
>>> mad_copy.load("mad.pt")

objective_function(mad_image=None, image=None)[source]#

Compute the MADCompetition synthesis loss.

This computes:

\[\begin{split}t L_1(x, \hat{x}) &+ \lambda_1 [L_2(x, x+\epsilon) - L_2(x, \hat{x})]^2 \\ &+ \lambda_2 \mathcal{B}(\hat{x})\end{split}\]

where \(t\) is 1 if minmax is 'min' and -1 if it’s 'max', \(L_1\) is optimized_metric, \(L_2\) is reference_metric, \(x\) is image, \(\hat{x}\) is mad_image, \(\epsilon\) is the initial noise, \(\mathcal{B}\) is the penalty function, \(\lambda_1\) is metric_tradeoff_lambda and \(\lambda_2\) is penalty_lambda.

If setup or synthesize has not been called to initialize the MAD image, then this will return an empty tensor.

Parameters:

mad_image (Tensor | None (default: None)) – Proposed mad_image, \(\hat{x}\) in the above equation. If None, use self.mad_image.
image (Tensor | None (default: None)) – Proposed image, \(x\) in the above equation. If None, use self.image.

Return type:

Tensor

Returns:

loss – 1-element tensor containing the loss on this step.

save(file_path)[source]#

Save all relevant variables in .pt file.

Note that if store_progress is True, this will probably be very large.

See load docstring for an example of use.

Parameters:: file_path (str) – The path to save the MADCompetition object to.

setup(initial_noise=None, optimizer=None, optimizer_kwargs=None, scheduler=None, scheduler_kwargs=None)[source]#

Initialize the MAD image, optimizer, and scheduler.

Can only be called once. If load() has been called, initial_noise must be None.

Parameters:

initial_noise (float | None (default: None)) – mad_image is initialized to self.image + initial_noise * torch.randn_like(self.image), so this gives the standard deviation of the Gaussian noise. If None, we use a value of 0.1.
optimizer (Optimizer | None (default: None)) – The un-initialized optimizer object to use. If None, we use Adam.
optimizer_kwargs (dict | None (default: None)) – The keyword arguments to pass to the optimizer on initialization. If None, we use {"lr": .01} and, if optimizer is None, {"amsgrad": True}.
scheduler (LRScheduler | None (default: None)) – The un-initialized learning rate scheduler object to use. If None, we don’t use one.
scheduler_kwargs (dict | None (default: None)) – The keyword arguments to pass to the scheduler on initialization.

Raises:

ValueError – If you try to set initial_noise after calling load.
ValueError – If setup is called more than once or after synthesize.

Examples

Set initial noise:

>>> import plenoptic as po
>>> img = po.data.einstein()
>>> mad = po.MADCompetition(
...     img,
...     lambda x, y: 1 - po.metric.ssim(x, y),
...     po.metric.mse,
...     "min",
...     metric_tradeoff_lambda=0.1,
... )
>>> mad.setup(1)
>>> mad.synthesize(10)

Set optimizer:

>>> import plenoptic as po
>>> img = po.data.einstein()
>>> mad = po.MADCompetition(
...     img,
...     lambda x, y: 1 - po.metric.ssim(x, y),
...     po.metric.mse,
...     "min",
...     metric_tradeoff_lambda=0.1,
... )
>>> mad.setup(optimizer=torch.optim.SGD, optimizer_kwargs={"lr": 0.01})
>>> mad.synthesize(10)

Use with save/load. Only the optimizer object is necessary, its kwargs and the initial noise are handled by load.

>>> import plenoptic as po
>>> img = po.data.einstein()
>>> mad = po.MADCompetition(
...     img,
...     lambda x, y: 1 - po.metric.ssim(x, y),
...     po.metric.mse,
...     "min",
...     metric_tradeoff_lambda=0.1,
... )
>>> mad.setup(1, optimizer=torch.optim.SGD, optimizer_kwargs={"lr": 0.01})
>>> mad.synthesize(10)
>>> mad.save("mad_setup.pt")
>>> mad = po.MADCompetition(
...     img,
...     lambda x, y: 1 - po.metric.ssim(x, y),
...     po.metric.mse,
...     "min",
...     metric_tradeoff_lambda=0.1,
... )
>>> mad.load("mad_setup.pt")
>>> mad.setup(optimizer=torch.optim.SGD)
>>> mad.synthesize(10)

synthesize(max_iter=100, store_progress=False, stop_criterion=0.0001, stop_iters_to_check=50)[source]#

Synthesize a MAD image.

Update the pixels of initial_image to maximize or minimize (depending on the value of minmax) the value of optimized_metric(image, mad_image) while keeping the value of reference_metric(image, mad_image) constant.

We run this until either we reach max_iter or the loss changes less than stop_criterion over the past stop_iters_to_check iterations, whichever comes first.

Parameters:

max_iter (int (default: 100)) – The maximum number of iterations to run before we end synthesis (unless we hit the stop criterion).
store_progress (bool | int (default: False)) – Whether we should store the MAD image in progress during synthesis. If False, we don’t save anything. If True, we save every iteration. If an int, we save every store_progress iterations (note then that 0 is the same as False and 1 the same as True).
stop_criterion (float (default: 0.0001)) – If the loss over the past stop_iters_to_check has changed less than stop_criterion, we terminate synthesis.
stop_iters_to_check (int (default: 50)) – How many iterations back to check in order to see if the loss has stopped decreasing (for stop_criterion).

Raises:

ValueError – If we find a NaN during optimization.

to(*args, **kwargs)[source]#

Move and/or casts the parameters and buffers.

This can be called as

to(device=None, dtype=None, non_blocking=False)

to(dtype, non_blocking=False)

to(tensor, non_blocking=False)

Its signature is similar to torch.Tensor.to, but only accepts floating point desired dtype. In addition, this method will only cast the floating point parameters and buffers to dtype (if given). The integral parameters and buffers will be moved device, if that is given, but with dtypes unchanged. When on_blocking` is set, it tries to convert/move asynchronously with respect to the host if possible, e.g., moving CPU Tensors with pinned memory to CUDA devices.

See torch.nn.Module.to for examples.

Note

This method modifies the module in-place.

Parameters:

device (torch.device) – The desired device of the parameters and buffers in this module.
dtype (torch.dtype) – The desired floating point type of the floating point parameters and buffers in this module.
tensor (torch.Tensor) – Tensor whose dtype and device are the desired dtype and device for all parameters and buffers in this module.

property gradient_norm: Tensor#: Optimization gradient’s L2 norm over iterations.

property image: Tensor#: The reference image for this MAD Competition.

property initial_image: Tensor#

Initial image for MAD Competition.

This is the image whose distance to image, the reference, we are maximizing/minimizing for optimized_metric, while keeping constant for reference_metric.

property losses: Tensor#

Optimization loss over iterations.

Will have length=num_iter+1, where num_iter is the number of iterations of synthesis run so far.

This tensor always lives on the CPU.

property mad_image: Tensor#: Maximally-differentiating image, the parameter we are optimizing.

property metric_tradeoff_lambda: float#: Tradeoff between the two metrics in synthesis loss.

property minmax: Literal['min', 'max']#: Whether we are minimizing or maximizing optimized_metric.

property optimized_metric: Module | Callable[[Tensor, Tensor], Tensor]#: The metric whose value we are minimizing or maximizing.

property optimized_metric_loss: Tensor#

optimized_metric loss over iterations.

That is, the value of optimized_metric(image, mad_image). Ideally, this is either very different from optimized_metric(image, initial_image).

This tensor always lives on the CPU, regardless of the device of the MADCompetition object.

property optimizer: Optimizer#: Torch optimizer object which updates the synthesis target.

property penalties: Tensor#

Penalty function output over iterations.

Will have length=num_iter+1, where num_iter is the number of iterations of synthesis run so far.

This tensor always lives on the CPU.

property penalty_function: Callable[[Tensor], Tensor]#: Callable which penalizes additional properties of the synthesized image.

property penalty_lambda: float#: Magnitude of the regularization weight.

property pixel_change_norm: Tensor#: L2 norm change in pixel values over iterations.

property reference_metric: Module | Callable[[Tensor, Tensor], Tensor]#: The metric whose value we are keeping constant.

property reference_metric_loss: Tensor#

reference_metric loss over iterations.

That is, the value of reference_metric(image, mad_image). Ideally, this is equal to reference_metric(image, initial_image).

This tensor always lives on the CPU, regardless of the device of the MADCompetition object.

property saved_mad_image: Tensor#

mad_image, cached over time for later examination.

How often the MAD image is cached is determined by the store_progress argument to the synthesize function.

The last entry will always be the current mad_image.

If store_progress==1, then this corresponds directly to losses: losses[i] is the error for saved_mad_image[i]

This tensor always lives on the CPU, regardless of the device of the MADCompetition object.

property scheduler: LRScheduler | None#: Learning rate scheduler which adjusts optimizer learning rate.

property store_progress: bool | int#

How often we are caching progress.

If False, we don’t save anything. If True, we save every iteration. If an int, we save every store_progress iterations (note then that 0 is the same as False and 1 the same as True).