An up-to-date objective AV1 benchmark
Posted at , last edit:
!☕ Last April I rented a large virtual machine in the Amazon Web Services cloud to run some AV1 benchmarks. They test the aom, SVT-AV1 and rav1e AV1 encoders against the x264 H.264/AVC encoder. The versions used are very recent development versions, pulled directly from source code repositories. Benchmark information and results follow !
- Source content:
crowd_runsample, available from Xiph.org :: Derf’s Test Media Collection, 1920x1080 50 Hz , YUV, 8-bits per pixel, progressive, 500 frames, 10 seconds (readme)
- Benchmark type: objective metric
- Encoding mode: single pass, constant quality mode
- Machine: AWS EC2 c5.9xlarge spot instance (Intel Xeon Scalable Cascade Lake, 36 vCPUs, 72 GiB RAM, all-core turbo frequency 3.4 GHz, with AVX-512)
- Operating System: Ubuntu Server 20.04 LTS (HVM), SSD Volume Type, 64-bit x86
- Encoder front-end: ffmpeg
- Default encoding parameters: 2x2 tiles, multithread enabled
- Quality metric: VMAF with default model file
- Encoding date: April 12 to April 15, 2021
- Instance usage time: 73.9 hours (includes restarting some rav1e encodings)
- Instance cost information: region US East (Ohio), spot price US$ 0.342 per hour
- Total cost: US$ 29.81 (includes data transfer, data storage, extra t2.micro instance and taxes)
- There are no timing results on these benchmarks. Sorry, there will be next time.
- Many rav1e benchmarks with speed 0 are missing because the spot instance was shut down in the middle of the execution. Again, this will be left for a next time.
- There was a pause in commits in the SVT-AV1 repository, so its commit date is from February instead of April.
The list of versions of custom built components used, with commit ID and commit date,
follow. The Rust compiler version 1.51.0 was installed with
rustup. All other components
required by ffmpeg, including libx264 were installed from the Ubuntu package repository.
|component||commit ID||commit date|
The following list of encoder configurations were benchmarked (the full command lines of all encodings are available in the encoding script’s state file:
|encoder||preset parameter||quality parameter|
|aom||none (use defaults)||none (use defaults)|
|rav1e||none (use defaults)||none (use defaults)|
|SVT-AV1||none (use defaults)||none (use defaults)|
|x264||none (use defaults)||none (use defaults)|
Before the actual benchmarks, this section shows some results that relate the encoders parameters and actual results in bit rate and VMAF score. These are useful to better situate each encoders (abstract) parameter values with real world metrics. You may skip this section and go straight to the benchmarks if you prefer.
Benchmarks separated per encoder
These are actual benchmarks, containing the encoding curves per preset. Individual symbols in the chart represent one of the preset x quality combination tested and the curves are interpolated from them. Encoding without any extra parameters is indicated as the “default” symbol. Zoomed in charts are plotted with the same data, but with limited VMAF x bitrate range to the highest quality region (VMAF above 85).
Consolidated multiple encoder benchmark
Finally, here follows the consolidated chart, with selected curves from each encoder. The x264 “slower” preset curve is used as the benchmark: the chart separates encoding worse and better than it.
In this benchmark, most work revolved around creating the benchmark scripts and researching a cheap and powerful enough cloud server to run the benchmarks in reasonable time. For the next steps, some tweaks in the encoder configurations might be useful for removing encoding modes with too low quality and adding modes with very high quality. Other quality metrics might also be used and different encoding modes too. Different videos and source types are a must.
The scripts used to run the benchmarks and generate the charts are available on gitlab. The generated metadata for all the encoded files, including codec and container information and encoder settings is available on this site in file mediainfo.json. The data files for the charts and the complete VMAF log file for all the encoded frames is available in the benchmarks directory.