IHMO the study has been done very well. They've even eliminated potential differences coming from colorspace conversion and chroma subsampling. If anything, this study favors WebP that can't do full-resolution color.
Right, I probably came across not quite as intended; sorry about that. I didn't mean to say the study is hosed (which I can't say because I didn't read it!), the remark about objective metrics was a more general one. Yes, SSIM is one of the better metrics available, and obviously it's useful for certain things.
However it still doesn't accurately represent preceived quality; time and time again certain psy optimizations for instance are shown to reduce SSIM while perceived quality goes up. Conversely, a codec can optimize for SSIM alone and score well on a test that ranks by SSIM; but the perceived quality could be better. For this reason e.g. x264 has an option to disable psy optimizations that hurt the objective metrics.
There are other things that can hurt metrics a lot while doing nothing perceptible, though these are rather like bugs or errors in testing methodology, and as such would likely be noticed in a well conducted study. See also: https://wiki.xiph.org/Notes_on_testing_theora#General_points...
From what I can tell, for Y-SSIM and IW-SSIM they appear to be encoding color images but only measuring the luma component, which is invalid and renders the results meaningless. Unless they're actually encoding greyscale images for those sections without mentioning it?
Study author here. For Y-SSIM and IW-SSIM we convert the images to grayscale (MATLAB 'rgb2gray') before running the scoring algorithms. We aren't running the algorithms on luma planes. See published code and methodology.
How are you measuring filesize? Looking at compression_test.py, it looks like you encode a color image and use that filesize. But chroma takes bits to encode, and if you're only measuring greyscale (which is basically luma depending on what formula rgb2gray uses) then you're penalizing better chroma quality.
If you use a greyscale/luma-only metric, you have to encode greyscale from the start, not convert a color image after encoding. Otherwise you're measuring chroma bit cost, but not chroma distortion. Thus, invalid.
Actually, is PSNR-HVS-M greyscale-only as well? In that case, only the RGB-SSIM results would be valid...
And actually, thinking about it even more, converting to greyscale at the tail end just for Y-SSIM and IW-SSIM is fundamentally flawed (if it doesn't result in exactly the same bits as the luma plane), as now you're measuring distortion from an image that the codec was never asked to encode!
In test of objective metrics SSIM was one of the best (0.8 spearman correlation with human scores) http://www.ponomarenko.info/tid2008.htm
IHMO the study has been done very well. They've even eliminated potential differences coming from colorspace conversion and chroma subsampling. If anything, this study favors WebP that can't do full-resolution color.