Reproducibility and Compatibility#

Reprodubility#

Best Practices#

plenoptic’s synthesis methods are high-dimensional non-linear optimization problems. Thus, when trying to make your results reproducible, you should follow the guidelines below, which plenoptic uses for its tests and documentation:

Set the seed for the random number generator. In plenoptic’s synthesis methods, this largely affects the initialization of the synthesis method (e.g., with random noise for plenoptic.Metamer). We provide a convenience function for this, set_seed, which sets both the pytorch and numpy seeds.
Use torch.float64 dtype (torch defaults to torch.float32). torch.set_default_dtype might be helpful for this. See torch.dtype for more information.
Note the versions of plenoptic and pytorch when performing your analysis. We do not believe the versions of other packages are likely to affect reproducibility, but breaking changes in both plenoptic and pytorch have broken reproducibility.
If you really need to guarantee reproducibility and you used a GPU, note all relevant information, especially the CUDA and driver versions. See plenoptic issue #368 for a discussion here.

Small differences in the output of computations resulting from any of the latter three points above can result in small differences in the gradient that accumulate over the course of synthesis, so that it is easier to guarantee reproducibility for procedures that take fewer iterations than those that take more.

However, even when following all of the above steps, as discussed below, we cannot guarantee perfect reproducibility.

Limits#

plenoptic includes several results reproduced from the literature and aims to facilitate reproducible research. However, we are limited by our dependencies and PyTorch, in particular, comes with the caveat that “Completely reproducible results are not guaranteed across PyTorch releases, individual commits, or different platforms. Furthermore, results may not be reproducible between CPU and GPU executions, even when using identical seeds” (quote from the v1.12 documentation).

This means that you should note the plenoptic version and the pytorch version your synthesis used in order to guarantee reproducibility (some versions of pytorch will give consistent results with each other, but it’s not guaranteed and hard to predict). We do not believe reproducibility depends on the python version or any other packages.

However, in general, the CPU and GPU will always give different results, and you may also end up with different outputs on GPU devices with different CUDA or driver versions.

We reproduce several results from the literature and validate these as part of our tests. We are therefore aware of the following changes that broke reproducibility:

PyTorch 1.8 and 1.9 give the same results, but 1.10 changes results in changes, probably due to the difference in how the sub-gradient for torch.min and torch.max are computed (see this PR).
PyTorch 1.12 breaks reproducibility with 1.10 and 1.11, unclear why (see this issue).

Compatibility#

While we try to maintain compatibility between plenoptic versions, plenoptic is under active development and so the API of its objects may change. Similar to the comments on reproducibility above, we cannot guarantee that you will be able to load a plenoptic object saved with a different version of plenoptic or pytorch. The following notes known breaking changes and how, if possible, to load your object anyway.

Note that you should always be able to load in the saved object using pytorch directly, like so:

If your object was saved in plenoptic 1.1 or earlier:

import torch
plen_object = torch.load("saved_plen_object.pt", weights_only=False)

Note that setting weights_only=False allows arbitrary code execution and thus is unsafe — only run the above code when you know the contents of and trust the saved file! (See related issue for more details.)

If your object was saved in plenoptic 1.2 or later, weights_only=False is unnecessary:

import torch
plen_object = torch.load("saved_plen_object.pt")

The object you have loaded is a dictionary with strings as keys. Once you’ve loaded your object, you can extract the outcome of your synthesis (e.g., by looking at the "metamer" key). Post an issue if you need help!

Breaking change to load in plenoptic 1.2#

Prior to plenoptic 1.2, we were saving python functions and pytorch optimization objects. This was incompatible with a breaking change made to torch.load in pytorch version 2.6. Unfortunately, there is no way to make an object saved prior to 1.2 compatible with later plenoptic releases; see Compatibility for how to extract the outputs of your synthesis.

FutureWarning in load in plenoptic 1.3.1#

A small change was made to the Metamer and MADCompetition APIs in plenoptic 1.3.1. You will be able to load Metamer and MADCompetition objects saved with version 1.2 and 1.3 for some time, but doing so will raise a FutureWarning and this compatibility will eventually be removed.

In order to make an object compatible with future releases, you can either load it in with plenoptic 1.3.1 and re-save it, or do the following:

import torch
old_save = torch.load("old_save.pt")
old_save["_current_loss"] = None
torch.save(old_save, "old_save.pt")
# and then load as normal

FutureWarning in load in plenoptic 1.4#

Loading Metamer and MADCompetition objects saved before plenoptic 1.4 will result in a FutureWarning due to two changes:

Metamer objects’ save method was updated allow saving of more general loss functions (e.g., portilla_simoncelli_loss_factory).
Metamer and MADCompetition APIs were changed, replacing the range penalty with a more generic penalty_function. See Regularization penalty for more details on how to use this new capability.

You will be able to load Metamer and MADCompetition objects saved with version 1.2 through 1.3.1 for some time, but doing so will raise a FutureWarning and this compatibility will eventually be removed.

In order to make an object compatible with future releases, you can load it in with plenoptic 1.4 and re-save it.

Penalties attribute#

Accompanying the new penalty_function attribute is the new attribute penalties, which tracks the penalty function output over synthesis iterations (analogous to how losses tracks the value of the objective function over synthesis iterations). Because this data was not tracked before this release, we are unable to produce it when loading an old object. Instead, penalties will be set to an appropriately-sized tensor of torch.nan to signify the missing values.

If you called synthesize with the optional store_progress argument, the metamer-in-progress was cached over synthesis with some frequency and so you can manually compute the corresponding penalty values. To do so:

# initialize your Metamer object
...
met.load("old_metamer_object.pt")

import torch
# saved_metamer is always on cpu, so if your metamer is on the GPU, you'll need to move it there
saved_mets = met.saved_metamer.to(met.image.device)
penalties = torch.func.vmap(met.penalty_function)(saved_mets)
assert len(saved_mets) == len(penalties)

Warning

Note that, if store_progress>1, then the synthesis procedure did not cache the metamer-in-progress on every iteration and so len(penalties) < len(met.losses).

(The above all holds for both Metamer and MADCompetition.)

`raise_on_checks`#

When loading synthesis objects, plenoptic is fairly conservative: we do not save callable objects (as that can allow for arbitrary code execution during load, which is unsafe), and thus users are required to initialize the object before loading in the saved version. Therefore, while loading, we run a variety of checks to ensure that the saved object is the same as the one doing the loading. This is to avoid a situation where e.g., a Metamer object was initialized model=po.models.OnOff but the saved object used model=po.models.Gaussian. If the user attempted to continue synthesis at that point, things would get very confused, and, even if they were just examining the completed synthesis, it’s unclear what exactly correct behavior should be.

However, it is possible that we’re too conservative, and are raising errors where they don’t belong. This is especially likely across plenoptic versions or if you do something simple like change the name of an object.

Starting in plenoptic 2.0, all synthesis object load methods have the argument raise_on_checks. If this is set to True (the default), then failing any of our checks will result in an error. If it’s set to False, then they will instead result in a warning (plenoptic.io.LoadWarning) being raised. It is recommended that you always run your code with raise_on_checks=True first, only set it to False if you are certain that the saved object is the same as the loading one, and carefully inspect all LoadWarning raised. See the example in the docstring of LoadWarning for a usage example.

Note that loading with raise_on_checks=False will still result in errors if the dtype or device are different. The proper way to handle this is to call synth_object.to(dtype) before loading or synth_object.load(file, map_location=device), respectively. These will ensure that the object is properly moved to the correct data type or device.

Finally, note that we are unable to ensure that the behavior of synthesis object’s methods is unchanged. Thus, you should be especially careful if the synthesis object itself appears to have changed – proceed at your own risk!