SSIM

SSIM (structural similarity index) compares a reference and a distorted image window by window, combining differences in local brightness, contrast and structure (correlation) instead of raw per-pixel error. It was published by Wang, Bovik, Sheikh and Simoncelli in 2004. A score of 1 means identical; lower is worse. MS-SSIM (multi-scale SSIM) repeats the comparison at several downscaled resolutions. Both track perception better than PSNR, but they still reward smoothing and mostly ignore colour, so a higher SSIM doesn’t mean an encode looks better.

What SSIM is good for, and what it isn’t

SSIM is cheap and appears in almost every codec paper, so it is useful for codec development, regression tests and as a second sanity check on a pipeline next to PSNR.

It is still a weak predictor of how an encode looks:

• It rewards smoothing. A flat area scores higher than texture or grain that looks right but doesn’t match the source pixel for pixel, which is exactly the trade psychovisual tuning makes. Encoder tunes for SSIM (x264 and x265 --tune ssim, SVT-AV1 --tune 2) drop psychovisual optimisations to score higher, and x264 warns that --ssim results “will be invalid” with psy on.
• It is mostly luma. Many implementations only measure luma, and FFmpeg’s combined score weights planes by pixel count, so colour shifts and chroma artifacts barely register.

On Cloudinary’s CID22 image dataset, SSIM’s and MS-SSIM’s rank correlation with viewer scores is 0.76, against 0.88 for SSIMULACRA2. For comparing encoders or settings, use SSIMULACRA2, Butteraugli, XPSNR or CVVDP, and look at the video; see Compare encoding quality.

Measure SSIM with FFmpeg

ffmpeg -i distorted.mkv -i reference.mkv -lavfi ssim=stats_file=ssim.log -f null -

The summary prints SSIM for each plane (Y, U, V) and All, with each value also given in dB in parentheses. ssim.log has the values for every frame.

The input rules are the same as for PSNR: distorted first, reference second; equal dimensions (scale the encode back to the reference size with scale=W:H:flags=bicubic if needed); the same frame count, since FFmpeg silently repeats the last frame of a shorter input; and setpts=PTS-STARTPTS on both inputs only if one starts at a non-zero timestamp.

Read the numbers

• Scores crowd towards 1, so a visible difference can be a change in the third decimal place. The dB value, -10 × log10(1 − SSIM), spreads out the top of the scale: 0.99 is 20 dB and 0.999 is 30 dB.
• All weights the planes by pixel count, so with 4:2:0 luma makes up two-thirds of it. Quote Y if you need a figure comparable with tools that measure luma only.
• SSIM numbers from different tools aren’t interchangeable. libvmaf’s float_ssim, for example, downsamples the image first as the original authors suggest, and FFmpeg’s filter does not.

MS-SSIM

FFmpeg has no MS-SSIM filter, but libvmaf computes it as an extra feature (a VMAF score is calculated alongside it):

ffmpeg -i distorted.mkv -i reference.mkv -lavfi libvmaf=feature=name=float_ms_ssim:log_fmt=json:log_path=msssim.json -f null -

The per-frame and pooled float_ms_ssim values are in the JSON log. MS-SSIM handles viewing distance and resolution changes better than single-scale SSIM, but it shares SSIM’s luma-only view and its preference for smooth output.