VideoToolbox

VideoToolbox is Apple’s low-level framework for hardware video encoding and decoding on macOS and iOS. On Apple Silicon (and Macs with a T2 chip) the work runs on the media engine, which makes it fast and very efficient. FFmpeg exposes h264_videotoolbox, hevc_videotoolbox and prores_videotoolbox; HandBrake offers the same engines as its VideoToolbox H.264, H.265 and H.265 10-bit encoders.

Hardware or software encoder?

As with every hardware encoder, you get speed and low power use at the cost of compression efficiency: x264 and x265 at slow presets give better quality at the same bitrate. VideoToolbox is a good fit for fast exports, recordings and previews, and for hardware ProRes. For archival or size-critical H.264 and HEVC encodes, use the software encoders. More in NVENC: hardware or software.

Codec support

Codec	FFmpeg encoder	Notes
H.264	h264_videotoolbox	8-bit 4:2:0.
HEVC	hevc_videotoolbox	8-bit and 10-bit (-profile:v main10 -pix_fmt p010le), alpha channel input, 4:2:2 10-bit on hardware that supports it.
ProRes	prores_videotoolbox	Profiles proxy, lt, standard, hq, 4444, xq. Hardware ProRes needs a chip with a ProRes engine: M1 Pro, Max and Ultra, and every M2 or later chip.
AV1	none	FFmpeg has no AV1 VideoToolbox encoder. Apple chips from the M3 and A17 Pro decode AV1 in hardware, but that is decode only.

Quality-oriented encode

On Apple Silicon Macs, -q:v sets a constant-quality target from 0 to 100, higher is better (FFmpeg enables it only in native arm64 macOS builds):

ffmpeg -i input.mkv -c:v hevc_videotoolbox -q:v 60 -tag:v hvc1 -c:a copy output.mp4

-tag:v hvc1 makes QuickTime, Safari and iOS accept HEVC in MP4 or MOV; without it the file carries the hev1 tag that Apple players do not recognise. The 0–100 scale is not CRF, and a good value depends on the source, so encode a short clip and look at it. 10-bit HEVC:

ffmpeg -i input.mkv -c:v hevc_videotoolbox -q:v 60 -profile:v main10 -pix_fmt p010le -tag:v hvc1 -c:a copy output.mp4

On Intel Macs -q:v is not available (FFmpeg stops with “-q:v qscale not available for encoder. Use -b:v bitrate instead”), so set a bitrate:

ffmpeg -i input.mkv -c:v h264_videotoolbox -b:v 8M -c:a copy output.mp4

Full-GPU pipeline

ffmpeg -hwaccel videotoolbox -hwaccel_output_format videotoolbox_vld -i input.mkv -vf scale_vt=1920:1080 -c:v hevc_videotoolbox -q:v 60 -tag:v hvc1 output.mp4

Decode, scale and encode stay on the media engine. scale_vt is the only scaler in this path; any other filter needs hwdownload,format=nv12 first.

Options that matter

Option	What it does
-q:v	Quality 0–100, Apple Silicon only.
-b:v	Bitrate target, used when no quality is set. -constant_bit_rate 1 asks for true CBR (macOS 13 or newer).
-profile:v	H.264 baseline, main, high, constrained_high; HEVC main, main10, main42210; ProRes names above.
-allow_sw 1, -require_sw 1	Permit or force Apple’s software encoder. Off by default, so a missing hardware encoder is an error, not a silent fallback.
-realtime 1	Hint for live capture. Lowers latency, may cost efficiency.
-prio_speed 1	Prefer speed over quality.
-power_efficient 1	Ask for the lower-power path when available.
-spatial_aq 1	Spatial adaptive quantization where supported.
-alpha_quality	Quality of the alpha channel in HEVC with alpha (0–1).

HandBrake

HandBrake can drive VideoToolbox as well. In HandBrakeCLI the encoders are vt_h264, vt_h265, vt_h265_10bit and vt_prores:

HandBrakeCLI -i input.mkv -o output.mp4 --encoder vt_h265 --quality 60

HandBrake’s own scaler is sharper than the VideoToolbox one, so prefer it for resizing.

Pitfalls

• Constant quality does not work on Intel Macs or with an x86_64 FFmpeg running under Rosetta; use a bitrate there.
• Missing -tag:v hvc1 is the usual reason an HEVC file will not play in Apple software.
• The hardware encoder cannot be tuned like x264/x265: there are no presets, and most encoder internals are not exposed.