{ "schemaVersion": 1, "contentVersion": "ca42e61ede0a24f789d7287a0001244ad22f4abf", "id": "docs/guides/compare-encoding-quality", "collection": "docs", "title": "Compare encoding quality with metrics and visual checks", "description": "How to compare encoders or settings fairly, with one reference, several quality levels, perceptual metrics summarised by their worst frames, and a careful look at those frames.", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/", "apiUrl": "https://encode.wiki/api/v1/documents/docs/guides/compare-encoding-quality.json", "markdownUrl": "https://encode.wiki/api/v1/markdown/docs/guides/compare-encoding-quality.md", "status": "current", "tags": [ "metrics", "comparison", "ffmpeg" ], "license": "CC BY-SA 4.0", "dates": { "lastVerified": "2026-10-08T00:00:00.000Z" }, "authors": [], "sources": [ { "title": "FFmpeg filter documentation (psnr, ssim, xpsnr, libvmaf, framesync)", "url": "https://ffmpeg.org/ffmpeg-filters.html", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "FFVship v5.1.2 usage", "url": "https://codeberg.org/Line-fr/Vship/src/tag/v5.1.2/doc/FFVship.md", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "SSIMULACRA 2 README", "url": "https://github.com/cloudinary/ssimulacra2/blob/main/README.md", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "Netflix VMAF FAQ (pooling and resolution)", "url": "https://github.com/Netflix/vmaf/blob/v3.2.1/resource/doc/faq.md", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "bjontegaard, BD-rate implementations in Python", "url": "https://github.com/FAU-LMS/bjontegaard", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "FFmpeg codec options (export_side_data film_grain)", "url": "https://ffmpeg.org/ffmpeg-codecs.html#Codec-Options", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "SVT-AV1 v4.2.0 parameters (film-grain-denoise)", "url": "https://gitlab.com/AOMediaCodec/SVT-AV1/-/blob/v4.2.0/Docs/Parameters.md", "accessedAt": "2026-10-08T00:00:00.000Z" }, { "title": "FFVship v5.1.2 argument parser (--verbose colour report)", "url": "https://codeberg.org/Line-fr/Vship/src/tag/v5.1.2/src/ffvship_utility/ffmpegmain.hpp", "accessedAt": "2026-10-08T00:00:00.000Z" } ], "headings": [ { "depth": 2, "text": "Make one reference", "anchor": "make-one-reference" }, { "depth": 2, "text": "Encode several quality levels", "anchor": "encode-several-quality-levels" }, { "depth": 2, "text": "Check that the frames line up", "anchor": "check-that-the-frames-line-up" }, { "depth": 2, "text": "Pick the metrics", "anchor": "pick-the-metrics" }, { "depth": 2, "text": "Summarise with the worst frames", "anchor": "summarise-with-the-worst-frames" }, { "depth": 2, "text": "Look at the worst sections", "anchor": "look-at-the-worst-sections" }, { "depth": 2, "text": "Keep it reproducible", "anchor": "keep-it-reproducible" } ], "sections": [ { "id": "_preamble", "title": "Page introduction", "depth": 0, "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/_preamble.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/_preamble.md", "checksum": "bcadea2dede928fc8ce767a4e0c77e6de8cf41ba3ac256bbf3050baa4a595862", "bytes": 286 }, { "id": "make-one-reference", "title": "Make one reference", "depth": 2, "anchor": "make-one-reference", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#make-one-reference", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/make-one-reference.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/make-one-reference.md", "checksum": "0c3467d2dc4ebb2f075da44e7494f9d336b1e1d8d367b4bbe917d616f69ed538", "bytes": 460 }, { "id": "encode-several-quality-levels", "title": "Encode several quality levels", "depth": 2, "anchor": "encode-several-quality-levels", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#encode-several-quality-levels", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/encode-several-quality-levels.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/encode-several-quality-levels.md", "checksum": "d65da3efd91d762057d73baacfac47d178a4c3972f085d9081406a870f880dff", "bytes": 1457 }, { "id": "check-that-the-frames-line-up", "title": "Check that the frames line up", "depth": 2, "anchor": "check-that-the-frames-line-up", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#check-that-the-frames-line-up", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/check-that-the-frames-line-up.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/check-that-the-frames-line-up.md", "checksum": "a9380c4e117f8444cb62272da4f521997bf18aeb7fae04710bbeb0ed810dc815", "bytes": 762 }, { "id": "pick-the-metrics", "title": "Pick the metrics", "depth": 2, "anchor": "pick-the-metrics", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#pick-the-metrics", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/pick-the-metrics.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/pick-the-metrics.md", "checksum": "5b7aea1f9903470f3c89533e27609aba230e21c61480bbe47d7cc5a61b28e2ab", "bytes": 1534 }, { "id": "summarise-with-the-worst-frames", "title": "Summarise with the worst frames", "depth": 2, "anchor": "summarise-with-the-worst-frames", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#summarise-with-the-worst-frames", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/summarise-with-the-worst-frames.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/summarise-with-the-worst-frames.md", "checksum": "cdd37ce72833db4c2d7a1777b3c437fb5f722c1a21b08211a873864ec1c1fda1", "bytes": 621 }, { "id": "look-at-the-worst-sections", "title": "Look at the worst sections", "depth": 2, "anchor": "look-at-the-worst-sections", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#look-at-the-worst-sections", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/look-at-the-worst-sections.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/look-at-the-worst-sections.md", "checksum": "afd13d2e50f2af2c14dd335f95db8f498696fa4a04f905f69688403ca1d1a30b", "bytes": 618 }, { "id": "keep-it-reproducible", "title": "Keep it reproducible", "depth": 2, "anchor": "keep-it-reproducible", "canonicalUrl": "https://encode.wiki/guides/compare-encoding-quality/#keep-it-reproducible", "apiUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/keep-it-reproducible.json", "markdownUrl": "https://encode.wiki/api/v1/sections/docs/guides/compare-encoding-quality/keep-it-reproducible.md", "checksum": "3196a5b7eba2ac231ef4d67e0323ce8bda8c8a464d23eb1e576df4b8bae2821b", "bytes": 324 } ], "outboundReferences": [ "/reference/metrics/butteraugli/", "/reference/metrics/cvvdp/", "/reference/metrics/psnr/", "/reference/metrics/ssim/", "/reference/metrics/ssimulacra2/", "/reference/metrics/vmaf/", "/reference/metrics/xpsnr/", "/tools/utilities/yuview/", "/tools/video-players/", "https://github.com/FAU-LMS/bjontegaard" ], "checksum": "fdbca95c9ddfb83bff544f189e7c94473a678b7ba8aeaad9b1b4b82097aa84f9", "contentFormat": "markdown", "markdown": "Metrics narrow down settings, find bad frames and let you compare many encodes; your eyes make the\nfinal call. A fair comparison needs one reference, aligned frames, several quality levels, metrics\nthat track perception, a summary that shows the worst parts, and a look at those parts.\n\n## Make one reference\n\nEncode every candidate from the same file with the same preprocessing, and measure against that\nfile, not the original release. Use a few 10–30 second clips that stress encoders: grain, dark\ngradients, foliage, fast motion, line art. Cutting them to lossless FFV1 gives every encoder and\nmetric identical frames (`-map 0:v:0` keeps just the video):\n\n```sh\nffmpeg -ss 00:12:30 -i source.mkv -t 20 -map 0:v:0 -c:v ffv1 reference.mkv\n```\n\n## Encode several quality levels\n\nA setting can win at one bitrate and lose at another, so encode each candidate at four or more\nlevels and compare **rate–quality curves** (size or bitrate against score):\n\n```sh\nfor crf in 20 25 30 35; do\n ffmpeg -i reference.mkv -c:v libx264 -preset slow -crf $crf x264-crf$crf.mkv\ndone\n```\n\nCRF values aren't comparable between encoders or even presets, so compare at equal size, not equal\nCRF. **BD-rate** condenses two curves into the average bitrate difference at equal quality; the\n[`bjontegaard`](https://github.com/FAU-LMS/bjontegaard) Python package computes it:\n\n```python\nimport bjontegaard as bd\n\nprint(bd.bd_rate(rate_a, score_a, rate_b, score_b, method=\"akima\"))\n```\n\n−10 means candidate B needs 10% fewer bits for the same score. It expects higher-is-better scores,\nso negate Butteraugli distances, and it only covers the quality range where the curves overlap.\n\n**Score encodes without film-grain synthesis.** Synthesised grain never matches the source pixel\nfor pixel, so full-reference metrics punish it even when it looks better. For AV1 in FFmpeg, put\n`-export_side_data film_grain` before the encode's `-i` to decode it without the grain, and judge\nthe grain by eye. If the encoder also denoised the source (SVT-AV1 `--film-grain-denoise 1`,\naomenc's `--denoise-noise-level`), the decode is still a clean picture measured against a grainy\nreference, and the scores stay low however good it looks.\n\n## Check that the frames line up\n\n```sh\nffprobe -v error -select_streams v:0 -count_frames -show_entries stream=width,height,nb_read_frames -of csv=p=0 x264-crf30.mkv\n```\n\n- Frame counts must match. FFmpeg's metric filters silently repeat the last frame of a shorter\n input, and a one-frame offset ruins every score. If everything scores terribly, check alignment\n first; [PSNR](/reference/metrics/psnr/) is a quick sanity check.\n- Scale a downscaled encode back to the reference resolution (bicubic) before measuring.\n- SSIMULACRA2, Butteraugli and CVVDP convert to RGB using the files' colour tags, so missing or\n wrong matrix, range or transfer tags give wrong scores. FFVship guesses missing tags;\n `FFVship --verbose` prints what it used for each file.\n\n## Pick the metrics\n\nUse more than one. They fail in different ways, and where they disagree is worth a look.\n\n| Metric | Role | How to run it |\n| ----------------------------------------------------- | ------------------------------------------ | ----------------------------------- |\n| [SSIMULACRA2](/reference/metrics/ssimulacra2/) | Main perceptual score | FFVship or Vship (GPU), vszip (CPU) |\n| [Butteraugli](/reference/metrics/butteraugli/) 3-norm | High-fidelity and near-transparent encodes | FFVship or Vship |\n| [XPSNR](/reference/metrics/xpsnr/) | Fast, motion-aware first pass | FFmpeg `xpsnr` (reference first) |\n| [CVVDP](/reference/metrics/cvvdp/) | Temporal, colour and display-aware; HDR | `cvvdp`, FFVship |\n| [VMAF](/reference/metrics/vmaf/) NEG or v1 | Optional streaming-style score | FFmpeg `libvmaf` |\n| PSNR, [SSIM](/reference/metrics/ssim/) | Sanity checks only; they reward blur | FFmpeg `psnr`, `ssim` |\n\n```sh\nFFVship --source reference.mkv --encoded x264-crf30.mkv --json x264-crf30-ssimu2.json\nFFVship --source reference.mkv --encoded x264-crf30.mkv -m Butteraugli --json x264-crf30-butter.json\nffmpeg -i reference.mkv -i x264-crf30.mkv -lavfi xpsnr=stats_file=x264-crf30-xpsnr.log -f null -\n```\n\n## Summarise with the worst frames\n\nA short ugly section is what viewers notice, and a mean hides it. For each encode keep the\n**mean**, a **low percentile** (the 5th for SSIMULACRA2, the 95th for Butteraugli, where lower is\nbetter) and the **worst frames** by number. FFVship prints the mean and percentiles; this lists the\nten worst frames from its SSIMULACRA2 JSON:\n\n```sh\njq -c 'to_entries | sort_by(.value[0]) | .[:10] | map(.key)' x264-crf30-ssimu2.json\n```\n\nPlot curves for the low percentile as well as the mean. A setting that lifts the mean but lowers\nthe percentile is trading consistency for average quality.\n\n## Look at the worst sections\n\nWatch the scenes around the worst frames at normal speed, at the size people will watch them. Then\nstep frame by frame and flip between the reference and each candidate at the same frame; flipping\nshows differences that side-by-side views hide. A [video player](/tools/video-players/) with frame\nstepping or [YUView](/tools/utilities/yuview/) works, or extract a frame:\n\n```sh\nffmpeg -i x264-crf30.mkv -vf \"select=eq(n\\,1234)\" -frames:v 1 x264-crf30-1234.png\n```\n\nLook for banding, blocking, ringing, smeared texture, lost grain, colour shifts and flicker, and\ncompare blind if you can.\n\n## Keep it reproducible\n\nRecord encoder versions and commands, the source and clip timestamps, and each metric tool with\nits settings (VMAF model, Butteraugli norm and intensity target, CVVDP display model). Otherwise\nnobody, including you later, can tell whether a difference came from the encoder or from the\nmeasurement.\n" }