{ "schemaVersion": 1, "contentVersion": "ca42e61ede0a24f789d7287a0001244ad22f4abf", "id": "articles/turbo-metrics-performance", "collection": "articles", "title": "Fast Computation of SSIMULACRA2 on GPUs: A Performance Evaluation", "description": "Turbo-metrics offers GPU acceleration for popular video quality metrics. Let's put it to the test!", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/", "apiUrl": "https://encode.wiki/api/v1/documents/articles/turbo-metrics-performance.json", "markdownUrl": "https://encode.wiki/api/v1/markdown/articles/turbo-metrics-performance.md", "status": "current", "tags": [ "video", "metrics", "benchmarks", "migrated" ], "license": "CC BY-SA 4.0", "dates": { "publishedAt": "2024-11-21T00:00:00.000Z", "lastVerified": "2026-09-07T00:00:00.000Z" }, "authors": [ "bolu" ], "sources": [ { "title": "TurboMetrics — CUDA implementation and scoring differences", "url": "https://github.com/Gui-Yom/turbo-metrics/tree/master/crates/ssimulacra2-cuda" }, { "title": "Cloudinary — SSIMULACRA2 reference metric", "url": "https://github.com/cloudinary/ssimulacra2" }, { "title": "NVIDIA Video Codec SDK 13.0 — NVDEC application note", "url": "https://docs.nvidia.com/video-technologies/video-codec-sdk/13.0/nvdec-application-note/index.html" }, { "title": "ITU-R BT.500 — subjective television picture-quality assessment", "url": "https://www.itu.int/rec/R-REC-BT.500" }, { "title": "Original Codec Wiki article", "url": "https://github.com/av1-community-contributors/codec-wiki/blob/37c3317fd058b23c71a87679bed5619a98f77752/blog/2024-11-21-turbo-metrics-evaluation.mdx" } ], "provenance": { "originUrl": "https://github.com/av1-community-contributors/codec-wiki/blob/37c3317fd058b23c71a87679bed5619a98f77752/blog/2024-11-21-turbo-metrics-evaluation.mdx", "originCommit": "37c3317fd058b23c71a87679bed5619a98f77752", "migratedAt": "2026-08-21T00:00:00.000Z", "modified": true, "originPath": "blog/2024-11-21-turbo-metrics-evaluation.mdx" }, "headings": [ { "depth": 1, "text": "Introduction", "anchor": "introduction" }, { "depth": 2, "text": "Basics", "anchor": "basics" }, { "depth": 3, "text": "SSIMULACRA2", "anchor": "ssimulacra2" }, { "depth": 4, "text": "In-depth: SSIMULACRA2 computation", "anchor": "in-depth-ssimulacra2-computation" }, { "depth": 4, "text": "SSIMULACRA2 subsampling", "anchor": "ssimulacra2-subsampling" }, { "depth": 3, "text": "TurboMetrics", "anchor": "turbometrics" }, { "depth": 4, "text": "In-depth: Implementation differences from the reference", "anchor": "in-depth-implementation-differences-from-the-reference" }, { "depth": 2, "text": "Performance evaluation", "anchor": "performance-evaluation" }, { "depth": 3, "text": "Setup", "anchor": "setup" }, { "depth": 3, "text": "Results and discussion", "anchor": "results-and-discussion" }, { "depth": 3, "text": "Considerations about video decoding performance", "anchor": "considerations-about-video-decoding-performance" }, { "depth": 3, "text": "Conclusions", "anchor": "conclusions" }, { "depth": 3, "text": "Further discussion: Possible improvements", "anchor": "further-discussion-possible-improvements" }, { "depth": 2, "text": "Short discussion on scoring error", "anchor": "short-discussion-on-scoring-error" }, { "depth": 2, "text": "Overall conclusions and final thoughts", "anchor": "overall-conclusions-and-final-thoughts" }, { "depth": 2, "text": "Future plans recorded in November 2024", "anchor": "future-plans-recorded-in-november-2024" }, { "depth": 4, "text": "Special thanks", "anchor": "special-thanks" } ], "sections": [ { "id": "_preamble", "title": "Page introduction", "depth": 0, "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/_preamble.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/_preamble.md", "checksum": "5cd70135d95ce87ef60315dd13cf5208f8e0c69c331604c1709cf871a6b4ca93", "bytes": 436 }, { "id": "introduction", "title": "Introduction", "depth": 1, "anchor": "introduction", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#introduction", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/introduction.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/introduction.md", "checksum": "e90c7c63b443e109b80c753937cfae4acfaf748e207a2e88c589ed1c0da84a55", "bytes": 28347 }, { "id": "basics", "title": "Basics", "depth": 2, "anchor": "basics", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#basics", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/basics.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/basics.md", "checksum": "8608b0bfe3a0d974d3a22552c3a0eea56b0cb0c03dbef2c230bdb9ed05f566d4", "bytes": 8024 }, { "id": "ssimulacra2", "title": "SSIMULACRA2", "depth": 3, "anchor": "ssimulacra2", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#ssimulacra2", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/ssimulacra2.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/ssimulacra2.md", "checksum": "3f285f2a861a2748e05d734189fa0ee25d70250021cbdee7af618275bcc7802c", "bytes": 4627 }, { "id": "in-depth-ssimulacra2-computation", "title": "In-depth: SSIMULACRA2 computation", "depth": 4, "anchor": "in-depth-ssimulacra2-computation", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#in-depth-ssimulacra2-computation", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/in-depth-ssimulacra2-computation.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/in-depth-ssimulacra2-computation.md", "checksum": "50a70020edb1df614f7fb21e0bf07d5b9c647a241d5c6157a592c27d4726b2f4", "bytes": 1183 }, { "id": "ssimulacra2-subsampling", "title": "SSIMULACRA2 subsampling", "depth": 4, "anchor": "ssimulacra2-subsampling", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#ssimulacra2-subsampling", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/ssimulacra2-subsampling.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/ssimulacra2-subsampling.md", "checksum": "0b6375ffa3358ffb3a6e35b9a6550cd926fca82f73009768e6d29d84133c3e16", "bytes": 907 }, { "id": "turbometrics", "title": "TurboMetrics", "depth": 3, "anchor": "turbometrics", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#turbometrics", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/turbometrics.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/turbometrics.md", "checksum": "e6135cfd2ca3a4965d5320e4c0e89f179e07e06fb8ffe2cc47728900eba0eb45", "bytes": 3275 }, { "id": "in-depth-implementation-differences-from-the-reference", "title": "In-depth: Implementation differences from the reference", "depth": 4, "anchor": "in-depth-implementation-differences-from-the-reference", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#in-depth-implementation-differences-from-the-reference", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/in-depth-implementation-differences-from-the-reference.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/in-depth-implementation-differences-from-the-reference.md", "checksum": "e8fddcde3545c4a5ca1cd7b7865239b00be104582da729d9cede337479d365e8", "bytes": 1239 }, { "id": "performance-evaluation", "title": "Performance evaluation", "depth": 2, "anchor": "performance-evaluation", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#performance-evaluation", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/performance-evaluation.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/performance-evaluation.md", "checksum": "bfb781ddb13c2c42f56b7c1438847acd200f2cd9ffc832c0ba10082b6107b2d8", "bytes": 11550 }, { "id": "setup", "title": "Setup", "depth": 3, "anchor": "setup", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#setup", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/setup.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/setup.md", "checksum": "799cfb0061460648e67d2c8d52d4dde48026ced1c4f52aaeb901eabb938d4825", "bytes": 1885 }, { "id": "results-and-discussion", "title": "Results and discussion", "depth": 3, "anchor": "results-and-discussion", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#results-and-discussion", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/results-and-discussion.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/results-and-discussion.md", "checksum": "de57a7a538afc8a25e5efbaa89025a080803957245d8c364527039b1e99584c9", "bytes": 3585 }, { "id": "considerations-about-video-decoding-performance", "title": "Considerations about video decoding performance", "depth": 3, "anchor": "considerations-about-video-decoding-performance", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#considerations-about-video-decoding-performance", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/considerations-about-video-decoding-performance.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/considerations-about-video-decoding-performance.md", "checksum": "e7b5403370c3e8acba1433fdc849bee0f6e64b94fe65333fc0da891dff990ced", "bytes": 3229 }, { "id": "conclusions", "title": "Conclusions", "depth": 3, "anchor": "conclusions", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#conclusions", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/conclusions.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/conclusions.md", "checksum": "6923fcbbf6f4bb0fea9d925d38c4c642e5fa7f7a761b84943a4458ca768ffccf", "bytes": 632 }, { "id": "further-discussion-possible-improvements", "title": "Further discussion: Possible improvements", "depth": 3, "anchor": "further-discussion-possible-improvements", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#further-discussion-possible-improvements", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/further-discussion-possible-improvements.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/further-discussion-possible-improvements.md", "checksum": "9e10a3c777246990bc2d92d86c7cb726d563fc27c52c866f114e6b3c78262b0a", "bytes": 2188 }, { "id": "short-discussion-on-scoring-error", "title": "Short discussion on scoring error", "depth": 2, "anchor": "short-discussion-on-scoring-error", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#short-discussion-on-scoring-error", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/short-discussion-on-scoring-error.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/short-discussion-on-scoring-error.md", "checksum": "cb8b23a4dd8b48cadbc397a4ed83c984806a64187f80936c6e1baa488f7760dd", "bytes": 3615 }, { "id": "overall-conclusions-and-final-thoughts", "title": "Overall conclusions and final thoughts", "depth": 2, "anchor": "overall-conclusions-and-final-thoughts", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#overall-conclusions-and-final-thoughts", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/overall-conclusions-and-final-thoughts.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/overall-conclusions-and-final-thoughts.md", "checksum": "2ff242cacafef1d17b09096ddaf01f06052dc5fb71c09e102ad983a1bdf730c7", "bytes": 1908 }, { "id": "future-plans-recorded-in-november-2024", "title": "Future plans recorded in November 2024", "depth": 2, "anchor": "future-plans-recorded-in-november-2024", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#future-plans-recorded-in-november-2024", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/future-plans-recorded-in-november-2024.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/future-plans-recorded-in-november-2024.md", "checksum": "c862e3f55a9f5cd0b47c78ff6eecbb7da16685aa31c7242dc56a923e5f415811", "bytes": 2533 }, { "id": "special-thanks", "title": "Special thanks", "depth": 4, "anchor": "special-thanks", "canonicalUrl": "https://encode.wiki/articles/turbo-metrics-performance/#special-thanks", "apiUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/special-thanks.json", "markdownUrl": "https://encode.wiki/api/v1/sections/articles/turbo-metrics-performance/special-thanks.md", "checksum": "a4454477df12ebf6b11978e58a03ccf6273d3b14e4e706c03e806a2391f707d0", "bytes": 298 } ], "outboundReferences": [ "/guides/compare-encoding-quality/", "/img/blog_turbo-metrics_11-2024/ssimu2_cpu-gpu_comp.svg", "/img/blog_turbo-metrics_11-2024/ssimu2_cpu-gpu_dec.svg", "/reference/metrics/ssimulacra2/", "https://discord.gg/bbQD5MjDr3", "https://docs.nvidia.com/video-technologies/video-codec-sdk/13.0/nvdec-application-note/index.html#nvdec-performance", "https://docs.rs/crate/ssimulacra2_rs/latest", "https://en.wikipedia.org/wiki/Structural_similarity_index_measure", "https://github.com/Gui-Yom", "https://github.com/Gui-Yom/turbo-metrics", "https://github.com/Gui-Yom/turbo-metrics/blob/master/crates/ssimulacra2-cuda/README.md#computing-ssimulacra2", "https://github.com/Gui-Yom/turbo-metrics/blob/master/crates/ssimulacra2-cuda/README.md#differences-from-the-reference-implementation", "https://github.com/Gui-Yom/turbo-metrics/blob/master/crates/ssimulacra2-cuda/README.md#how-to-do-better-", "https://github.com/Gui-Yom/turbo-metrics/tree/master/crates/ssimulacra2-cuda", "https://github.com/Netflix/vmaf/blob/master/libvmaf/README.md", "https://github.com/cloudinary/ssimulacra2", "https://github.com/dnjulek/vapoursynth-zip", "https://github.com/fraunhoferhhi/xpsnr" ], "checksum": "0ddb5d5e7700f119988a48247e40f14f62d4061e7032eed180ec0029244de0a2", "contentFormat": "markdown", "markdown": "> **Note — Editorial correction — 7 September 2026.**\n>\n> This is the author's 2024 timing experiment, preserved with its hardware, tables, graphs, and preliminary scoring observations. It establishes measured throughput on those workloads, not perceptual validity or equivalence to reference SSIMULACRA2. No validation rerun has been performed. Corrections below withdraw the general stride and quality-assessment recommendations.\n# Introduction\n\nIn this blog entry, we will evaluate the performance of [TurboMetrics](https://github.com/Gui-Yom/turbo-metrics), a software which provides a GPU-accelerated implementation of SSIMULACRA2, and promises to save a lot of time in its computation.\n\nVideo encoding is a time-consuming task by itself. When it comes to assessing the resulting encodes, objective quality metrics are often used, providing advantages over subjective measurements, e.g., faster evaluation processes. Nevertheless, the computation of some of these metrics is still somewhat slow. This is the case for [SSIMULACRA2](/reference/metrics/ssimulacra2/), one of the most popular objective metrics for image and video assessment.\n\n## Basics\n\nBefore we start with the evaluation, we will first cover some fundamental (and not-so-fundamental) concepts.\n\n### SSIMULACRA2\n\nSSIMULACRA2, which is sometimes abbreviated as \"ssimu2\", is a perceptual metric for images, based on the concept of the multi-scale structural similarity index measure ([MS-SSIM](https://en.wikipedia.org/wiki/Structural_similarity_index_measure)). In the words of its developer, it is \"[based on MS-SSIM], computed in a perceptually relevant color space, adding two other (asymmetric) error maps, and aggregating using two different norms\". A total of 54 error maps are computed over multiple downscalings (from 1:1 to 1:32) of the image to assess (_distorted_) and its source image (_reference_). These error maps are then added using a weighted sum to produce the final SSIMULACRA2 score. The weight of each error map is tuned based on a large set of subjective scores from different image benchmarks.\n\nSSIMULACRA2 scores are in the range [-inf, 100]. They are reported to correlate to subjective visual quality scores as follows:\n* 30 = low quality\n* 50 = medium quality\n* 70 = high quality\n* 90 = very high quality; likely impossible to distinguish from the original when viewed at 1:1 from a normal viewing distance\n\n**Editorial correction:** there is no validated universal video-transparency threshold of 80 established here. Image-score interpretations do not automatically transfer to pooled video scores; motion can mask some defects and reveal others. The reference project describes image measurements, not a guarantee for every viewer or video. [SSIMULACRA2 reference](https://github.com/cloudinary/ssimulacra2).\n\nThe reference implementation for SSIMULACRA2 can be found [here](https://github.com/cloudinary/ssimulacra2). Other popular implementations include [the Rust implementation](https://docs.rs/crate/ssimulacra2_rs/latest), and [the Zig implementation](https://github.com/dnjulek/vapoursynth-zip).\n\nSSIMULACRA2 can be applied independently to corresponding video frames. Means, percentiles, and per-frame traces summarize those image measurements, but they do not model temporal artifacts. A low standard deviation also does not guarantee consistently good quality. This article does not establish that SSIMULACRA2 universally outperforms PSNR, SSIM, or VMAF for video.\n\nAs stated previously, SSIMULACRA2 computes 54 error maps per image assessed. This entails a high amount of computation, especially for videos, which usually contain tens of thousands of frames (images). That explains its rather slow execution time. Speeding up these computations would be very much desired, if possible.\n\n---\n#### In-depth: SSIMULACRA2 computation\n\nThe summary of the algorithm for SSIMULACRA2 is the following, as [reported here](https://github.com/Gui-Yom/turbo-metrics/blob/master/crates/ssimulacra2-cuda/README.md#computing-ssimulacra2):\n1. Get the frame pair (reference and distorted).\n2. Convert the frames to linear RGB (using 32-bit floating-point).\n3. For each scale (1:1 to 1:32, downsampling by 2 every step: 6 different scales):\n 1. Downscale by the scale number (if needed).\n 2. Convert the frames to XYB color space.\n 3. Blur the following pictures, using a recursive Gaussian blur: (reference * reference), (distorted * distorted), (reference * distorted), (reference), (distorted).\n 4. Compute 1-SSIM, artifact and detail\\_loss error scores from the 5 blurred images and the original frame pair; this yields 3 error maps.\n 5. The error maps are reduced to a single number using the 1-norm and 4-norm.\n4. From this, we get 6 scores for each scale (6) and color component (3), totaling 108 scores.\n5. The scores are added using a weighted sum.\n6. The final value is processed through a non-linear function and clipped to render a score between 100 and -infinite.\n\n#### SSIMULACRA2 subsampling\n\nGiven that the metric is quite slow to compute, it is often computed over a subset (subsample) of the video frames, instead of over all of them. For example, a _stride_ of 3 could be used, to compute SSIMULACRA2 scores for 1 frame out of every 3 in the video, skipping the computation for the other 2 frames. This is expected to speedup the computation by a factor of ×3.\n\nSkipping frames changes the sampled population. Its error can grow or shrink with stride depending on the content and phase; it is not necessarily monotonic. Prime strides such as 3 or 5 do not guarantee unbiased coverage of temporal layers, periodic motion, scene cuts, or short defects. Compare several offsets with a full-frame run on representative material before adopting subsampling, and retain full-frame checks for brief failures. The timing results below measure speed, not sampling accuracy.\n\n### TurboMetrics\n\n[TurboMetrics](https://github.com/Gui-Yom/turbo-metrics) is a WIP software suite developed by the very talented [Gui-Yom/LimelioN](https://github.com/Gui-Yom). It is \"a collection of video related libraries and tools oriented at performance and hardware acceleration\". It comes as a collection of Rust libraries/crates, and some include command-line applications to interact with them. There are currently no public releases of the suite in the dedicated \"Releases\" page of its GitHub repository, but you can download the code and compile it yourself. The latest tagged version of the software, at the moment of writing, is v0.2.2.\n\nTurboMetrics focuses mainly on **NVIDIA GPUs** using CUDA. However, the developer is actively working on adding support for other hardware (e.g., AMD and Intel GPUs). Nevertheless, I only have experience with NVIDIA GPUs, so I cannot tell how the progress is going on that front.\n\nAmong the tools, there is a working SSIMULACRA2 implementation that uses CUDA to leverage GPU acceleration, ssimulacra2-cuda. We will focus exclusively on it from now on. We will use this tool to compute SSIMULACRA2 scores for certain video samples. However, it is worth noting that the tool is also able to compute other metrics, such as SSIM and PSNR. It can even compute multiple metrics on the same rum.\n\nssimulacra2-cuda is [stated to be](https://github.com/Gui-Yom/turbo-metrics/tree/master/crates/ssimulacra2-cuda) \"close to the original implementation, and with close results\". Also, the developer has said that \"[r]ight now, [it] can compute SSIMULACRA2 orders of magnitude faster than the ssimulacra2\\_rs implementation\". Therefore, we can expect the tool to compute SSIMULACRA2 scores for videos much faster than any CPU implementation out there, but provide scores that are not quite the same as the ones those CPU implementations would give, including the reference one. In this blog entry, we will test that second claim, concerning the speed of TurboMetrics in computing SSIMULACRA2.\n\n---\n#### In-depth: Implementation differences from the reference\n\nAs [reported here](https://github.com/Gui-Yom/turbo-metrics/blob/master/crates/ssimulacra2-cuda/README.md#differences-from-the-reference-implementation), the GPU implementation of SSIMULACRA2 provided in TurboMetrics differs from the reference implementation in the following ways that can affect the accuracy of the reported scores:\n* The conversion from YUV to linear RGB might not be yielding the same results as other tools.\n* Explicit Fused Multiply-Add (FMA) operations are used when possible, as the GPU can leverage them for higher performance.\n* The order of operations might not be the same, as some calculations have been rearranged.\n* Floating-point operations might not yield the same results, especially with the approximated math functions used in the GPU (like `powf`).\n\nAdditionally, the differences in scores are amplified by the following features of the SSIMULACRA2 computation:\n* The original weights were computed by fitting the error scores to Mean Opinion Score (MOS), and deviation in the error scores is amplified by them.\n* The final non-linear function is rather steep (it is a cubic function, x³), and, thus, score variations are amplified again.\n\n## Performance evaluation\n\n### Setup\n\nIn this experiment, we will evaluate TurboMetrics in terms of performance; that is, the amount of ssimu2 scores computed per unit of time. For that, an episode of an animated series (~25 minutes), encoded with H.264, has been chosen as sample video. The selected clip has 1080p resolution (1920×1080), 8 bits per sample, and YUV 4:2:0 chroma subsampling. These properties affect the metric workload, while codec, encoding settings, I/O, and decoder paths can also affect end-to-end throughput. The experiment does not isolate GPU kernel speed.\n\nThe video source and example distorted encode have been compared using **ssimulacra2\\_rs**, **vszip**, and **TurboMetrics**. ssimulacra2\\_rs and vszip are both CPU implementations of SSIMULACRA2, the former written in Rust and the latter in Zig (but invoked from VapourSynth/Python scripts). It has been reported that the Zig implementation is considerably faster than the Rust one, and we can check that out as well!\n\nThe CPU implementations have been executed on an Intel i5-11400 CPU (base frequency of 2.60 GHz, turbo frequency of 4.40 GHz), using 6 CPU cores (`-f 6` parameter in ssimulacra2\\_rs, `-t 6` parameter in ssimulacravszip.py). TurboMetrics has been executed both on an NVIDIA GTX 1060 GPU and an NVIDIA RTX 4060 GPU. The three programs have been executed for the same video pair with **varying frame strides: 1, 3, 5, 7, 11, 13, 17, 19, and 23** (not skipping frames, plus every prime number lower than 24 frames / 1 second, excluding 2).\n\nFor the CPU and RTX 4060 executions, the distorted video was a custom AV1 encode of the reference video. For the GTX 1060, as it does not have AV1 decoding hardware support, the H.264 reference video was compared to itself **This is a different decode workload from the H.264/AV1 pair used elsewhere, so the GTX and RTX timings are not a controlled GPU-only comparison.**\n\n### Results and discussion\n\nThe exact results of this experiment are shown in the following table:\n\n| Stride | ssimulacra2\\_rs FPS | vszip FPS | GTX 1060 GPU FPS | RTX 4060 GPU FPS |\n|:-:|:-:|:-:|:-:|:-:|\n| 1 | 3.37 | 14.63 | 46.70 | 85.64 |\n| 3 | 9.55 | 37.67 | 133.86 | 245.73 |\n| 5 | 15.15 | 49.34 | 164.34 | 350.01 |\n| 7 | 20.45 | 56.37 | 197.17 | 400.94 |\n| 11 | 30.23 | 64.46 | 222.98 | 458.61 |\n| 13 | 34.54 | 66.65 | 239.17 | 488.83 |\n| 17 | 40.51 | 69.23 | 249.58 | 520.30 |\n| 19 | 42.82 | 69.99 | 258.71 | 525.44 |\n| 23 | 45.43 | 88.60 | 264.85 | 546.14 |\n\nHere they are in graph form, for better visualization:\n\n![CPU vs GPUs decoded FPS](/img/blog_turbo-metrics_11-2024/ssimu2_cpu-gpu_dec.svg)\n\nOn a superficial level, we can already make some observations:\n* SSIMULACRA2 computation with ssimulacra2\\_rs is quite slow: 3.37 frames per second (without skipping frames) when using 6 threads. That is like 0.56 frames per second per thread. I know the CPU used is no beast by any measure, but beast CPUs are quite expensive and not that common (that is something to keep in mind). Those processing speeds are comparable to some of the slower-ish presets of SVT-AV1.\n - vszip is, in fact, considerably faster: Between ×4.34 (full video computation) and ×1.63 (stride of 19) with the same amount of threads. However, it seems that vszip does not scale as well as the other implementations (except for that weird performance boost with a stride of 23).\n* GPU computation of SSIMULACRA2 scores is impressive, considering the CPU performances:\n - Compared to ssimulacra2\\_rs: Speedups for the GTX 1060 (an 8-year-old GPU) are between ×14.01 (stride of 3) and ×5.83 (stride of 23). Speedups for the RTX 4060 are between ×25.73 (stride of 3) and ×12.02 (stride of 23).\n - Compared to vszip: Speedups for the GTX 1060 are between ×3.70 (stride of 19) and ×2.99 (stride of 23). Speedups for the RTX 4060 are between ×7.51 (stride of 17) and ×5.85 (stride of 1 / full video computation).\n - It is amazing seeing how GPUs not only achieve real-time SSIMULACRA2 computation of entire videos, but more than ×1.94 and ×3.56 real-time processing, even (GTX and RTX, respectively).\n\nNevertheless, the most interesting observation is made when we use computation strides bigger than 1: The performance does _not_ scale linearly with the stride, as it would be expected. This happens with both the CPU and GPUs, although it is more noticeable in the case of the GPUs due to their higher performance.\n\nWhy is that? Well, there is actually a good explanation: The reported FPS are _decoded_ FPS. That is, the speed in which the software is decoding and processing all the video frames. All the frames in the video bitstream are decoded and reconstructed as images, but the SSIMULACRA2 score is only computed for some of them. SSIMULACRA2 computation affects decoding speed in that the decoding of each frame only starts after the processing of the previous has finished (kind of, this is not be entirely true for multithreaded CPU implementations, but we can ignore that). The faster the SSIMULACRA2 processing is done, the sooner the next frame starts decoding. And there is no faster processing than not doing any processing at all, that is, skipping the frame! However, **the decoding itself is a process that has to be done regardless**, and the time it takes should not be disregarded. Although decoding as a process is considerably faster than SSIMULACRA2 computation, it starts becoming the bottleneck when you both compute SSIMULACRA2 so fast, and skip computing frames.\n\n### Considerations about video decoding performance\n\n[NVIDIA's Video Codec SDK 13.0 application note](https://docs.nvidia.com/video-technologies/video-codec-sdk/13.0/nvdec-application-note/index.html#nvdec-performance) gives these indicative per-NVDEC figures for 1080p YUV 4:2:0 decoding:\n* Pascal architecture decodes H.264 at up to 694 frames per second.\n* Ada architecture decodes H.264 at up to 903 frames per second.\n* Ada architecture decodes AV1 at up to 1,018 frames per second.\n\nBeing proprietary NVIDIA technology, served as a black box through their API, we cannot be sure how the NVDEC engines are implemented and what exactly are their limitations under scenarios such as these (i.e., concurrent decoding of multiple video streams of same or different codecs). We can only trust the information NVIDIA provides. NVDEC engine count and capability are product-specific; the statement that all GeForce products have one engine is not a current hardware rule. The ratios below are the author's comparison against indicative SDK figures, not measurements of decoder utilization.\nIn our case, considering two videos are being decoded at the same time:\n* The GTX 1060 decodes up to 529.7 H.264 (2×264.85) frames per second when using a stride of 23 frames, which corresponds to 75% of its reported peak decoding performance for H.264.\n* The RTX 4060 decodes up to 546.14 H.264 + 546.14 AV1 (1092.28 total) frames per second when using a stride of 23 frames, which corresponds to 61.85% and 54.34% of its reported peak decoding performance for H.264 and AV1 respectively. Notice how, even though the NVDEC hardware resources are reportedly being shared between the two decodes, both are able to reach more than 50% peak performance **at the same time**.\n\nWe can just focus on _processed_ FPS, that is, only not-skipped frames processed per second, or SSIMULACRA2 scores computed per second, to see how the processing performance degrades with increasing strides.\n\n| Stride | ssimulacra2\\_rs FPS | vszip FPS | GTX 1060 GPU FPS | RTX 4060 GPU FPS |\n|:-:|:-:|:-:|:-:|:-:|\n| 1 | 3.37 | 14.63 | 46.70 | 85.64 |\n| 3 | 3.18 | 12.56 | 44.62 | 81.91 |\n| 5 | 3.03 | 9.87 | 32.87 | 70.00 |\n| 7 | 2.92 | 8.05 | 28.17 | 57.28 |\n| 11 | 2.75 | 5.86 | 20.27 | 41.69 |\n| 13 | 2.66 | 5.13 | 18.40 | 37.60 |\n| 17 | 2.38 | 4.07 | 14.68 | 30.61 |\n| 19 | 2.25 | 3.68 | 13.62 | 27.65 |\n| 23 | 1.98 | 3.85 | 11.52 | 23.75 |\n\nHere they are in graph form, for better visualization:\n\n![CPU vs GPUs processed FPS](/img/blog_turbo-metrics_11-2024/ssimu2_cpu-gpu_comp.svg)\n\nIt can be seen how the actual SSIMULACRA2 processing performance quickly decreases, as more and more relative time gets dedicated to decoding the videos.\n\n**Editorial correction:** these measurements do not justify a GPU purchase recommendation or a claim that increasing stride has insignificant scoring error. Decoder throughput depends on the GPU model, clocks, stream characteristics, and software pipeline; generation alone does not guarantee identical end-to-end performance. Profile the intended workload before deciding whether decoding, conversion, transfer, or metric computation is the bottleneck. Sampling error and implementation error require separate validation.\n\n### Conclusions\n\nAs a quick summary of the experiment, we can highlight:\n* **GPU processing greatly accelerates SSIMULACRA2 computation**, with modern GPUs enabling speeds **faster than real-time** on the tested RTX 4060 workload (~85 fps), while the GTX 1060's different H.264 self-comparison workload reached ~46 fps.\n* As SSIMULACRA2 computations accelerate, **video decoding speed becomes a relevant factor, to the point of being the main bottleneck when the computation stride is large enough**. It is not worthwhile to increase the computation stride too much, as you will quickly get diminishing returns in performance.\n\n---\n\n### Further discussion: Possible improvements\n\nOne may think that a possible solution to the decoding speed bottleneck, which does not require waiting until the next-generation GPUs implement faster decoding hardware, would be to **make the software decode only the frames that are going to be processed**. Nevertheless, that approach comes with significant caveats: Inter-predicted frames require their reference dependencies to be decoded, but not necessarily every frame in the entire GOP. Savings from selective decoding depend on the dependency graph, random-access points, and decoder API; this experiment did not measure such a pipeline. (Aside from the possible accuracy/representativeness concerns associated to using such big strides.) Additionally, when leveraging GPU programming, the actual hardware is often treated like a black box, and the available APIs may limit considerably the control the user has over it.\n\nAnother, more useful approach one may think of, would be to exploit **computation overlapping**: To keep decoding frames while the SSIMULACRA2 scores are being computed, storing the decoded frames that are to be processed in a buffer. This seems to be partially implemented in TurboMetrics already, but only to some extent.\n\nWithout any modification to the SSIMULACRA2 computations, the overlap in computation and decoding would still have limited effect for low strides (where SSIMULACRA2 computation is still the bottleneck), as every time the decoded frame buffer fills completely, the decoder would have to wait for it to have free space again. In theory, the SSIMULACRA2 computations could be sped up further (maybe requiring faster but more imprecise approximations). And, ideally, TurboMetrics should aim to approach the theoretical decode FPS limit in the computation of the SSIMULACRA2 scores, thus perfectly overlapping computing and decoding. In such scenario, skipping the computation of any frames would not make sense anymore, as no speed advantage would be obtained from it. However, this is still far from being a reality, as a speedup of at least ×10.31 is needed for an RTX 4060 to match its decoding speeds with a computation stride of 1.\n\n## Short discussion on scoring error\n\nMy original idea for this blog entry was to evaluate TurboMetrics both in terms of performance and scoring error. By \"scoring error\" I mean \"the difference between the scores a CPU reference provides, and the ones TurboMetrics provides\". However, once the first draft for that original version of this blog entry was completed, I realized I was not satisfied with the methodology followed and conclusions extracted from that second experiment. Therefore, I am putting off that evaluation for a second blog entry, that I will write once I make a new, bigger, and better, evaluation of the error.\n\nThe main concern about the error the GPU computation incurs on SSIMULACRA2 scores is that SSIMULACRA2 scores have been tuned mathematically to correlate with visual quality, while the GPU implementation's scores _try to mimic_ those scores, but have not undergone any kind of mathematical tuning or assessment themselves. The GPU scores could be seen as \"SSIMULACRA2 + random error\", and that \"random error\" might break the desired correlation, thus reducing the reliability of the metric.\n\nNevertheless, it is worth noting that SSIMULACRA2 is not by any means a kind of \"perfect metric\" or \"absolute truth\" about video quality assessment. **SSIMULACRA2 still is unreliable in some situations, and it does not always present a perfect correlation with visual observations**. You may find here and there specific examples of SSIMULACRA2 scores that do not accurately reflect the (perceived) quality of a compressed image or video. Besides, we should keep in mind that quality is a subjective matter. In that regard, and as an example, people who do not care much about film grain might give denoised videos higher quality scores than those that SSIMULACRA2 gives. (Other metrics, such as VMAF, are more tolerant to denoising.) Calling the output an “alternative metric” does not validate it. A changed implementation needs agreement checks against the reference with identical preprocessing, and a changed perceptual model needs testing against independent subjective data. The preliminary findings below are reasons to investigate, not proof of either perceptual validity or universal invalidity.\n\nJust to give a preview on the error evaluation thus far, I will share the following: The preliminary results I obtained from that now-scraped evaluation seem to indicate that:\n* The scores provided by the GPU implementation of SSIMULACRA2 are **statistically different** to the real scores (provided by the CPU implementations).\n* **No linear correlation** between the real scores and the GPU scores has been found.\n* TurboMetrics **might tend to under-score**, compared to the real scores (i.e., the results for the frames tend to be lower than those a CPU implementation gives). This has yet to be assessed over a larger sample size to prove if it is really true, though.\n\nI am looking for help with the error evaluation for the follow-up blog entry. Mainly, I need a considerably larger sample of SSIMULACRA2 scores for videos, computed both in CPU and GPU. I myself do not have enough material (encoded videos and/or the SSIMULACRA2 scores for all their frames) yet to do a sufficiently good evaluation, and it would take me quite some time to get it. Thus, I am looking for anyone who has saved the SSIMULACRA2 scores for all the frames of any video they had assessed, and is able to re-assess those same videos using TurboMetrics. If you think you can help, and are willing to, please contact me through the [AV1 for Dummies Discord server](https://discord.gg/bbQD5MjDr3).\n\n## Overall conclusions and final thoughts\n\nIn this blog entry, we have tested the GPU implementation of SSIMULACRA2 available through TurboMetrics, in terms of performance (i.e., scores computed per unit of time). The main conclusions are the following:\n* All in all, TurboMetrics provides **impressive acceleration** of the SSIMULACRA2 computation.\n* **The performance of TurboMetrics does not scale linearly with the computational stride/frame skipping**. When increasing the stride, a point of diminishing returns is reached rather soon.\n - This is caused by the time taken to decode the video frames, which is non-negligible, and becomes the bottleneck when computing SSIMULACRA2 so fast and only for some of the frames.\n\nHardware acceleration of computationally-intensive tasks is an important topic nowadays in several fields, especially considering how powerful current-day GPUs are; and I would hope for the teams developing these quality assessment metrics to consider it as an existing option. So far, only [XPSNR](https://github.com/fraunhoferhhi/xpsnr) has seemed to be concerned about computational complexity from the beginning, and the developers' response was not to consider hardware acceleration, but rather to simplify the design of their metric.\n\n**Editorial correction to the recommendation:** the reported speedups justify considering TurboMetrics for further evaluation, but this article does not establish it as a substitute for reference SSIMULACRA2 quality scores. Do not transfer reference score thresholds to it or assume stride 3 or 5 is harmless. First match color conversion and frame alignment, quantify score and ranking differences on representative distortions, and validate sampling separately. Use [quality-comparison guidance](/guides/compare-encoding-quality/) and the [reference implementation](https://github.com/cloudinary/ssimulacra2) for a reproducible baseline.\n\n## Future plans recorded in November 2024\n\nTurboMetrics' development is still ongoing, and I know Gui-Yom has tons of ideas for it. To begin with, I know the following ideas are being worked on:\n* Support for GPUs of other vendors (AMD, Intel).\n* Support for FFmpeg piping, to enable working with videos of any codec.\n* GPU acceleration of XPSNR.\n\nAll these features would improve TurboMetrics significantly, while also opening the door for more tests to be performed. Additionally, as it has been mentioned in this blog entry, the SSIMULACRA2 computation can still be optimized further, including overlapping computation and decoding. Gui-Yom knows this, and plans to tackle that issue eventually.\n\nA list of features to improve TurboMetrics' computation of SSIMULACRA2, written by the main developer, can be found [here](https://github.com/Gui-Yom/turbo-metrics/blob/master/crates/ssimulacra2-cuda/README.md#how-to-do-better-).\n\nAs I said earlier, I also plan to write a follow-up blog entry evaluating the scoring error incurred in the GPU implementation, to test the reliability of the scores provided by TurboMetrics. Additionally, it may be insightful to evaluate the error for GPU implementations of other video metrics, such as [VMAF CUDA](https://github.com/Netflix/vmaf/blob/master/libvmaf/README.md), and see if we find any similarities. I do not recall reading any analysis on the matter for other metrics, either. Nevertheless, I cannot give a date yet for when that follow-up blog entry might come out. I reckon it will take quite some time to perform the required experimentation, as well as processing and analyzing the results.\n\nAlso, this blog entry is heavily missing information on 4K video, which I believe to be of high interest. The main reason for it is that most 4K content is served using HEVC, and TurboMetrics still does not support that codec. When support for it gets added, experiments with 4K video should be conducted.\n\nIf you like TurboMetrics and have knowledge about programming, please consider [contributing to it](https://github.com/Gui-Yom/turbo-metrics) to the best of your ability. Together, we can make it even more awesome than it already is!\n\nAnd, lastly, thank you very much for reading!\n\n#### Special thanks\n\n* **Gui-Yom**, for developing TurboMetrics, and also for proofreading and green-lighting this blog entry.\n* **Trix**, for linking me to the Python script to easily compute SSIMULACRA2 using vszip (ssimulacravszip.py), and for helping me to get it to work the way I needed.\n\n{}\n" }