DEAD / distributed-encode
Poorly coded distributed encoding using av1an, python-rq, Redis, and optionally S3.
DEAD (“Distributed Encoding AV1, Dude!”), the distributed-encode project by Kosaka, splits a video into scenes the way Av1an does, then hands the chunks to Av1an workers on several machines. It is for people with more than one encoding box who want them to work on the same file.
How it works
Section titled “How it works”A client process detects scenes with Av1an and puts groups of scenes into a Redis queue as jobs (via python-rq). Worker processes on each machine take jobs, encode them with Av1an, and return the results, either through a shared network folder that every machine mounts at the same path or through an S3-compatible bucket. The client then joins the chunks with mkvmerge.
Requirements:
- Python 3.12 or newer and a reachable Redis server.
- On every worker, a patched Av1an from the author’s Av1an-verbosity fork, which reports per-chunk progress; stock Av1an does not work.
- Run it from the repository (or a distribution package; an Arch PKGBUILD is included), not from a virtual environment or pipx, because Av1an breaks inside one.
Audio is dropped, so mux it back afterwards with mkvmerge. A VapourSynth script can be used for filtering with --vs-input (and --vs-scene-input for scene detection).
With a shared folder mounted at /mnt/dead on all machines, start a worker on each encoding machine and then the client, from the root of the repository (an installed package provides the same commands as dead):
python src/distributed_encode.py server --name worker1 --basedir /mnt/dead --output-dir worker1_outputpython src/distributed_encode.py client --input /mnt/dead/file.mkv --av1an-options "-e svt-av1" --encoder-options "--preset 4 --crf 30" --basedir /mnt/dead --output-dir client_outputRedis defaults to 127.0.0.1:6379; point remote machines at it with --host, --port and --password. The --s3-* options replace the shared folder. Only expose Redis on a trusted network.