Under development
This currently contains examples of the earlier MAD synthesis, but we have yet to reproduce it using plenoptic.
Run this notebook yourself!
Download the executed notebook: Original_MAD.ipynb!
Run it in your browser: Original_MAD.ipynb!
Reproducing Wang and Simoncelli, 2008 (MAD Competition)#
Goal here is to reproduce original MAD Competition results, as generated using the matlab code originally provided by Zhou Wang and then modified by the authors. MAD Competition is a synthesis method for efficiently computing two models, by generating sets of images that minimize/maximize one model’s loss while holding the other’s constant. For more details, see the MAD Competition Usage and MAD Competition Conceptual Introduction notebooks.
import plenoptic as po
%load_ext autoreload
%autoreload 2
# Download some data we'll need for this notebook
import contextlib
import os
# the contextlib.redirect_stderr here is so that we don't print out the progressbar.
# If you would like to see it, remove this line.
with contextlib.redirect_stderr(open(os.devnull, "w")):
po.data.fetch_data("MAD_results.tar.gz")
po.data.fetch_data("ssim_images.tar.gz")
SSIM#
Before we discuss MAD Competition, let’s look a little at SSIM, since that’s the metric used in the original paper, and which we’ll be using here. Important to remember that SSIM is a similarity metric, so higher is better, and thus a value of 1 means the images are identical and it’s bounded between 0 and 1.
We have tests to show that this matches the output of the MATLAB code, won’t show here.
img1 = po.data.einstein()
img2 = po.data.curie()
noisy = po.process.add_noise(img1, [2, 4, 8])
We can see that increasing the noise level decreases the SSIM value, but not linearly
po.metric.ssim(img1, noisy)
/home/jenkins/agent/workspace/CCN_neurorse_plenoptic_main/lib/python3.12/site-packages/plenoptic/process/metric.py:92: UserWarning: Image range falls outside [0, 1]. SSIM output may not make sense.
warnings.warn(
tensor([[0.0029],
[0.0013],
[0.0003]])
And that our noise level does match the MSE
po.metric.mse(img1, noisy)
tensor([[2.0000],
[4.0000],
[8.0000]])
MAD Competition#
The following figure shows the results of MAD Competition synthesis using the original MATLAB code. It shows the original image in the top left. We then added some Gaussian noise (with a specified standard error) to get the image right below it. The four images to the right of that are the MAD-synthesized images. The first two have the same mean-squared error (MSE) as the first image (and each other), but the best and worst SSIM value (SSIM is a similarity metric, so higher is better), while the second two have the same SSIM as the first image, but the best and worst MSE. By comparing these images, we can get a sense for what MSE and SSIM consider important for image quality.
# We need to download some additional data for this portion of the notebook.
fig, results = po.external.plot_MAD_results("samp6", [128], vrange="row1", zoom=3)
There’s lots of info here, on the outputs of the MATLAB synthesis. We will later add stuff to investigate this using plenoptic.
results
{'L128': {'FIX_MSE': 127.99999999999999,
'FIX_SSIM': 0.8183184633106257,
'mse_fixmse_maxssim': array([128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128.]),
'ssim_fixmse_maxssim': array([0.82669306, 0.83641599, 0.84768936, 0.86021352, 0.87332037,
0.8861153 , 0.89794336, 0.90864194, 0.91828312, 0.9270046 ,
0.93495769, 0.94226293, 0.94896492, 0.95506452, 0.9605342 ,
0.96533558, 0.96945409, 0.97290935, 0.97575609, 0.97807185,
0.97994388, 0.98145704, 0.98268627, 0.98369369, 0.98452855,
0.98522884, 0.98582351, 0.98633451, 0.98677852, 0.98716831,
0.98751374, 0.98782249, 0.98810062, 0.98835294, 0.98858334,
0.98879493, 0.98899028, 0.98917148, 0.98934029, 0.98949816,
0.9896463 , 0.98978576, 0.98991742, 0.99004203, 0.99016023,
0.99027259, 0.99037961, 0.9904817 , 0.99057924, 0.99067257,
0.99076198, 0.99084774, 0.99093007, 0.9910092 , 0.99108531,
0.99115858, 0.99122917, 0.99129722, 0.99136287, 0.99142623,
0.99148744, 0.99154658, 0.99160377, 0.99165909, 0.99171263,
0.99176448, 0.9918147 , 0.99186338, 0.99191058, 0.99195637,
0.99200081, 0.99204396, 0.99208587, 0.99212661, 0.99216622,
0.99220476, 0.99224226, 0.99227877, 0.99231435, 0.99234902,
0.99238284, 0.99241583, 0.99244804, 0.99247949, 0.99251023,
0.99254029, 0.99256969, 0.99259847, 0.99262665, 0.99265426,
0.99268134, 0.99270789, 0.99273395, 0.99275955, 0.99278469,
0.99280941, 0.99283372, 0.99285765, 0.9928812 , 0.99290441]),
'mse_fixmse_minssim': array([128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128., 128., 128., 128., 128., 128., 128., 128., 128., 128., 128.,
128.]),
'ssim_fixmse_minssim': array([0.81069721, 0.80415136, 0.79827382, 0.7927989 , 0.78758476,
0.7825861 , 0.77782948, 0.77338629, 0.76933555, 0.7657286 ,
0.76258054, 0.75987991, 0.75759382, 0.75567185, 0.75405628,
0.7526936 , 0.75154034, 0.75056265, 0.74973342, 0.74902986,
0.74843227, 0.74792359, 0.7474892 , 0.74711674, 0.74679588,
0.74651809, 0.74627635, 0.74606493, 0.74587912, 0.74571508,
0.74556965, 0.74544023, 0.74532467, 0.74522116, 0.74512817,
0.74504441, 0.74496878, 0.74490032, 0.74483822, 0.74478177,
0.74473035, 0.74468342, 0.74464051, 0.74460123, 0.7445652 ,
0.74453212, 0.74450172, 0.74447375, 0.74444799, 0.74442426,
0.74440239, 0.74438221, 0.7443636 , 0.74434643, 0.74433057,
0.74431594, 0.74430244, 0.74428997, 0.74427846, 0.74426784,
0.74425803, 0.74424898, 0.74424062, 0.74423291, 0.74422579,
0.74421923, 0.74421317, 0.74420758, 0.74420242, 0.74419767,
0.74419328, 0.74418924, 0.74418551, 0.74418208, 0.74417891,
0.74417599, 0.74417331, 0.74417083, 0.74416855, 0.74416646,
0.74416453, 0.74416275, 0.74416112, 0.74415963, 0.74415825,
0.74415699, 0.74415583, 0.74415477, 0.7441538 , 0.74415291,
0.7441521 , 0.74415135, 0.74415068, 0.74415006, 0.7441495 ,
0.74414899, 0.74414853, 0.74414811, 0.74414773, 0.74414739]),
'maxssim': 0.99290440833461,
'minssim': 0.7441473913547447,
'mse_fixssim_minmse': array([127.62569966, 127.25907758, 126.89830629, 126.5432421 ,
126.19375253, 125.84970824, 125.51098323, 125.17745477,
124.84900331, 124.5255124 , 124.2068686 , 123.89296145,
123.58368333, 123.27892946, 122.97859775, 122.68258882,
122.39080584, 122.10315454, 121.81954311, 121.53988213,
121.26408454, 120.99206555, 120.72374258, 120.45903523,
120.19786522, 119.94015629, 119.68583422, 119.43482671,
119.18706338, 118.94247568, 118.70099686, 118.46256195,
118.22710767, 117.99457239, 117.76489611, 117.53802042,
117.31388843, 117.09244474, 116.87363542, 116.65740794,
116.44371116, 116.23249528, 116.02371179, 115.81731348,
115.61325436, 115.41148964, 115.21197571, 115.01467011,
114.81953145, 114.62651948, 114.43559495, 114.24671966,
114.0598564 , 113.87496892, 113.69202191, 113.510981 ,
113.33181268, 113.15448433, 112.97896417, 112.80522123,
112.63322535, 112.46294714, 112.29435797, 112.12742994,
111.96213588, 111.79844929, 111.63634436, 111.47579594,
111.31677951, 111.15927119, 111.00324769, 110.8486863 ,
110.6955649 , 110.54386192, 110.39355633, 110.24462762,
110.09705579, 109.95082136, 109.8059053 , 109.66228908,
109.5199546 , 109.37888422, 109.23906072, 109.10046731,
108.96308761, 108.82690563, 108.69190575, 108.55807275,
108.42539176, 108.29384826, 108.16342808, 108.03411739,
107.90590266, 107.77877069, 107.6527086 , 107.52770378,
107.40374392, 107.280817 , 107.15891125, 107.03801519]),
'ssim_fixssim_minmse': array([0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846,
0.81831846, 0.81831846, 0.81831846, 0.81831846, 0.81831846]),
'mse_fixssim_maxmse': array([ 131.81989257, 136.34348117, 141.63201819, 147.93891278,
155.38338115, 164.2072263 , 174.58969259, 180.77038681,
187.35687873, 194.48181227, 198.57370304, 202.66885863,
206.95343129, 211.44231618, 216.13691245, 221.02634466,
226.11581241, 231.45406809, 237.02262995, 242.83028251,
248.81360619, 255.03345526, 261.46183743, 268.20589445,
271.92855589, 275.50581231, 279.16591822, 282.90724991,
286.73714745, 290.64830841, 294.64536986, 298.72977854,
302.9070491 , 307.17661949, 311.52367919, 315.9615864 ,
320.49015071, 325.12141654, 329.86378322, 334.71438487,
339.66399361, 344.73644098, 349.93210914, 355.25219363,
360.69572122, 366.26837608, 371.96846258, 377.80329199,
383.77938626, 389.88638347, 396.14353339, 402.54835547,
409.12344679, 415.86473831, 422.7769448 , 429.86538679,
437.11955142, 444.55628708, 452.18579893, 460.02165701,
468.0649593 , 476.29650757, 484.7347087 , 493.40456576,
502.30552879, 511.44021594, 520.83114704, 530.48547572,
540.40541267, 550.59925317, 561.07367344, 571.86215993,
582.96345493, 594.39243813, 606.16232589, 618.28830266,
630.7534013 , 643.57218989, 656.77008653, 670.38608448,
684.42229927, 698.90648368, 713.75502045, 729.09525608,
744.89191175, 761.03686543, 777.65684378, 794.83179371,
812.49479186, 830.66898008, 849.46972268, 868.84766347,
888.75751215, 909.25712128, 930.36668338, 952.1656941 ,
974.65873922, 997.87604088, 1021.77859313, 1046.42121422]),
'ssim_fixssim_maxmse': array([0.81838315, 0.81837394, 0.81840433, 0.81841667, 0.8184811 ,
0.81854265, 0.81865555, 0.8185667 , 0.81860168, 0.8186602 ,
0.81849321, 0.81849264, 0.81849454, 0.81849667, 0.81850134,
0.81850698, 0.81852163, 0.8185241 , 0.81853312, 0.81854646,
0.81856169, 0.81858209, 0.81861676, 0.8186252 , 0.8184758 ,
0.81848028, 0.81848157, 0.81848229, 0.8184804 , 0.81848067,
0.81848019, 0.81847912, 0.81847739, 0.81847648, 0.81848131,
0.81848611, 0.81848742, 0.81848633, 0.81848348, 0.81848151,
0.81848324, 0.81848096, 0.81847737, 0.81847369, 0.81847083,
0.81846829, 0.81846601, 0.81846364, 0.81846065, 0.81845897,
0.81845727, 0.8184584 , 0.81845481, 0.81845096, 0.81844714,
0.81844336, 0.81844286, 0.81844313, 0.8184418 , 0.81843787,
0.81843385, 0.81843244, 0.81843006, 0.81842643, 0.81842301,
0.81842012, 0.8184164 , 0.81841261, 0.81841113, 0.81841123,
0.81841167, 0.81840845, 0.81840571, 0.81840289, 0.81840006,
0.81839648, 0.81839794, 0.81840175, 0.81840656, 0.81840632,
0.8184053 , 0.8184015 , 0.81841222, 0.8184118 , 0.81841594,
0.81844041, 0.81845594, 0.81846008, 0.81847317, 0.81848535,
0.81848382, 0.81848619, 0.81849768, 0.81851117, 0.81852665,
0.81853106, 0.81853637, 0.81854015, 0.81854653, 0.81855083]),
'minmse': 107.03801518643529,
'maxmse': 1046.4212142210524,
'noise_level': 128,
'original_image': 'samp6'}}