Files
wuffs/doc/benchmarks.md
T
Nigel Tao f4e4381a3b Update dod/benchmarks.md.
Note that these benchmarks were measured on Debian Testing, which uses
different clang and gcc versions than the Ubuntu Trusty system used for
benchstat numbers reported in recent git commits.
2017-11-09 20:37:16 +11:00

138 lines
4.5 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Benchmarks
*Preliminary* `puffs bench -mimic` summarized throughput numbers for various
codecs are below. Higher is better.
"Mimic" tests check that Puffs' output mimics (i.e. exactly matches) other
libraries' output. "Mimic" benchmarks give the numbers for those other
libraries, as shipped with the OS. These were measured on a Debian Testing
system as of November 2017, which means these compiler versions:
- clang/llvm version 5.0.0
- gcc version 7.2.0
and these mimic library versions:
- libgif 5.1.4
- zlib 1.2.8
Unless otherwise stated, these benchmarks were run on an Intel x86_64
Broadwell.
## Reproducing
The benchmark programs aim to be runnable "out of the box" without any
configuration or installation. For example, to run the `std/flate` benchmarks:
# TODO: confirm that we will live at github.com/google/puffs
git clone https://github.com/google/puffs.git
cd puffs/test/c/std
gcc -O3 flate.c
./a.out -bench
rm a.out
A comment near the top of that .c file says how to run the mimic benchmarks.
The output of those benchmark programs is compatible with the
[benchstat](https://godoc.org/golang.org/x/perf/cmd/benchstat) tool. For
example, that tool can calculate confidence intervals based on multiple
benchmark runs, or calculate p-values when comparing numbers before and after a
code change. To install it, first install Go, then run `go get
golang.org/x/perf/cmd/benchstat`.
## puffs bench
As mentioned above, individual benchmark programs can be run manually. However,
the canonical way to run the benchmarks (across multiple compilers and multiple
packages like GIF and PNG) is to use the `puffs` command line tool, as it will
also re-generate (transpile) the C code whenever you edit the \*.puffs code.
Running `go install -v github.com/google/puffs/cmd/...` will install the Puffs
tools. After that, you can say
puffs bench
or
puffs bench -mimic std/flate
or
puffs bench -ccompilers=gcc -reps=3 -focus=Benchmarkpuffs_gif_lzw std/gif
## CPU Scaling
CPU power management can inject noise in benchmark times. On a Linux system,
power management can be controlled with:
# Query.
cpupower --cpu all frequency-info --policy
# Turn on.
sudo cpupower frequency-set --governor powersave
# Turn off.
sudo cpupower frequency-set --governor performance
# Flate (including gzip and zlib)
The 1k, 10k, etc. numbers are approximately how many bytes there in the decoded
output.
name speed
puffs_adler32_10k/clang 2.42GB/s ± 3%
puffs_adler32_100k/clang 2.42GB/s ± 3%
puffs_flate_decode_1k/clang 133MB/s ± 3%
puffs_flate_decode_10k/clang 199MB/s ± 3%
puffs_flate_decode_100k/clang 229MB/s ± 4%
puffs_zlib_decode_10k/clang 185MB/s ± 1%
puffs_zlib_decode_100k/clang 210MB/s ± 3%
puffs_adler32_10k/gcc 3.22GB/s ± 1%
puffs_adler32_100k/gcc 3.23GB/s ± 0%
puffs_flate_decode_1k/gcc 152MB/s ± 1%
puffs_flate_decode_10k/gcc 250MB/s ± 1%
puffs_flate_decode_100k/gcc 298MB/s ± 1%
puffs_zlib_decode_10k/gcc 238MB/s ± 2%
puffs_zlib_decode_100k/gcc 270MB/s ± 2%
mimic_adler32_10k 3.00GB/s ± 1%
mimic_adler32_100k 2.91GB/s ± 2%
mimic_flate_decode_1k 211MB/s ± 1%
mimic_flate_decode_10k 270MB/s ± 2%
mimic_flate_decode_100k 285MB/s ± 1%
mimic_zlib_decode_10k 250MB/s ± 2%
mimic_zlib_decode_100k 294MB/s ± 2%
# GIF
The 1k, 10k, etc. numbers are approximately how many bytes of pixel data there
are in the decoded image. For example, the `test/testdata/harvesters.*` images
are 1165 × 859 (approximately 1000k pixels) and a GIF image (a paletted image)
is 1 byte per pixel.
name speed
puffs_gif_decode_1k/clang 346MB/s ± 1%
puffs_gif_decode_10k/clang 137MB/s ± 0%
puffs_gif_decode_100k/clang 118MB/s ± 0%
puffs_gif_decode_1000k/clang 120MB/s ± 0%
puffs_gif_decode_1k/gcc 399MB/s ± 1%
puffs_gif_decode_10k/gcc 141MB/s ± 0%
puffs_gif_decode_100k/gcc 128MB/s ± 0%
puffs_gif_decode_1000k/gcc 131MB/s ± 0%
mimic_gif_decode_1k 147MB/s ± 0%
mimic_gif_decode_10k 90.7MB/s ± 0%
mimic_gif_decode_100k 95.4MB/s ± 0%
mimic_gif_decode_1000k 97.8MB/s ± 0%
---
Updated on November 2017.