From 7b18295688f14a83a09e7b137175436a07834971 Mon Sep 17 00:00:00 2001 From: Nigel Tao Date: Fri, 20 Oct 2017 09:53:21 +1100 Subject: [PATCH] Update doc/benchmarks.md. --- doc/benchmarks.md | 63 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 18 deletions(-) diff --git a/doc/benchmarks.md b/doc/benchmarks.md index 4f75115f..7fadc3c2 100644 --- a/doc/benchmarks.md +++ b/doc/benchmarks.md @@ -7,31 +7,58 @@ codec are below. Higher is better. libraries' output. "Mimic" benchmarks give the numbers for those other libraries, as shipped with the OS, measured here on Ubunty 14.04 LTS "Trusty". +Unless otherwise stated, these benchmarks were run on an Intel x86_64 +Broadwell. + + +## Reproducing + +The benchmark programs aim to be runnable "out of the box" without any +configuration or installation. For example, to run the `std/flate` benchmarks: + + git clone etc + cd puffs/test/c/std + gcc -O3 flate.c + ./a.out -bench + rm a.out + +A comment near the top of that .c file says how to run the mimic benchmarks. + +The output of those benchmark programs is compatible with the +[benchstat](https://godoc.org/golang.org/x/perf/cmd/benchstat) tool. For +example, that tool can calculate confidence intervals based on multiple +benchmark runs, or calculate p-values when comparing numbers before and after a +code change. To install it, first install Go, then run `go get +golang.org/x/perf/cmd/benchstat`. + + +## GIF + The 1k, 10k, etc. numbers are approximately how many bytes of pixel data there is in the decoded image. For example, the `test/testdata/harvesters.*` images are 1165 × 859 (approximately 1000k pixels) and a GIF image (a paletted image) is 1 byte per pixel. - name speed - gif_puffs_decode_1k/clang 389MB/s ± 2% - gif_puffs_decode_10k/clang 137MB/s ± 0% - gif_puffs_decode_100k/clang 121MB/s ± 0% - gif_puffs_decode_1000k/clang 124MB/s ± 0% + name speed + puffs_gif_decode_1k/clang 335MB/s ± 2% + puffs_gif_decode_10k/clang 124MB/s ± 1% + puffs_gif_decode_100k/clang 113MB/s ± 1% + puffs_gif_decode_1000k/clang 115MB/s ± 1% - gif_mimic_decode_1k/clang 158MB/s ± 1% - gif_mimic_decode_10k/clang 94.4MB/s ± 0% - gif_mimic_decode_100k/clang 100MB/s ± 0% - gif_mimic_decode_1000k/clang 102MB/s ± 0% + mimic_gif_decode_1k/clang 154MB/s ± 1% + mimic_gif_decode_10k/clang 91.1MB/s ± 0% + mimic_gif_decode_100k/clang 97.1MB/s ± 1% + mimic_gif_decode_1000k/clang 98.7MB/s ± 2% - gif_puffs_decode_1k/gcc 406MB/s ± 1% - gif_puffs_decode_10k/gcc 158MB/s ± 0% - gif_puffs_decode_100k/gcc 138MB/s ± 0% - gif_puffs_decode_1000k/gcc 142MB/s ± 0% + puffs_gif_decode_1k/gcc 395MB/s ± 1% + puffs_gif_decode_10k/gcc 161MB/s ± 2% + puffs_gif_decode_100k/gcc 134MB/s ± 2% + puffs_gif_decode_1000k/gcc 138MB/s ± 1% - gif_mimic_decode_1k/gcc 158MB/s ± 0% - gif_mimic_decode_10k/gcc 94.4MB/s ± 0% - gif_mimic_decode_100k/gcc 100MB/s ± 0% - gif_mimic_decode_1000k/gcc 102MB/s ± 0% + mimic_gif_decode_1k/gcc 154MB/s ± 1% + mimic_gif_decode_10k/gcc 90.8MB/s ± 2% + mimic_gif_decode_100k/gcc 97.7MB/s ± 0% + mimic_gif_decode_1000k/gcc 98.7MB/s ± 0% The mimic numbers measure the pre-compiled library that shipped with the OS, so it is unsurprising that they don't depend on the C compiler (clang or gcc) used @@ -40,4 +67,4 @@ to run the test harness. --- -Updated on August 2017. +Updated on October 2017.