Commit Graph
82 Commits
Author SHA1 Message Date
Nigel Tao 1910ca31cf Run the mimic checksum benchmarks for longer. 2017-11-08 12:46:57 +11:00
Nigel Tao 13731f989f Add a reader1.unread_u8 method.
Also re-enable a number of std/flate tests and benchmarks.

It's not an optimization per se, but as a data point, here are some
benchmark numbers between 36a93fa and this commit. 36a93fa is "36a93fa
Add a TODO about impure functions", which is one commit before "f968950
Read bits up front in decode_huffman_fast", and f968950 was the commit
that made decode_huffman_fast read more than one byte at a time.

name                             old speed      new speed       delta

puffs_adler32_10k/clang          2.45GB/s ± 0%   2.45GB/s ± 0%     ~     (p=0.684 n=10+10)
puffs_adler32_100k/clang         2.45GB/s ± 0%   2.46GB/s ± 0%   +0.07%  (p=0.019 n=10+10)
puffs_flate_decode_1k/clang       124MB/s ± 0%    132MB/s ± 0%   +6.06%  (p=0.000 n=10+10)
puffs_flate_decode_10k/clang      145MB/s ± 0%    174MB/s ± 0%  +20.19%  (p=0.000 n=9+10)
puffs_flate_decode_100k/clang     168MB/s ± 0%    208MB/s ± 0%  +23.31%  (p=0.000 n=10+10)
puffs_zlib_decode_10k/clang       136MB/s ± 0%    162MB/s ± 0%  +18.94%  (p=0.000 n=10+9)
puffs_zlib_decode_100k/clang      158MB/s ± 0%    192MB/s ± 0%  +21.58%  (p=0.000 n=10+9)

puffs_adler32_10k/gcc            3.26GB/s ± 0%   3.26GB/s ± 0%     ~     (p=0.631 n=10+10)
puffs_adler32_100k/gcc           3.27GB/s ± 0%   3.27GB/s ± 0%     ~     (p=0.237 n=10+8)
puffs_flate_decode_1k/gcc        95.3MB/s ± 0%  105.5MB/s ± 0%  +10.67%  (p=0.000 n=10+10)
puffs_flate_decode_10k/gcc        110MB/s ± 0%    140MB/s ± 0%  +27.32%  (p=0.000 n=9+9)
puffs_flate_decode_100k/gcc       121MB/s ± 0%    150MB/s ± 0%  +24.02%  (p=0.000 n=8+10)
puffs_zlib_decode_10k/gcc         112MB/s ± 0%    120MB/s ± 0%   +6.96%  (p=0.000 n=10+9)
puffs_zlib_decode_100k/gcc        123MB/s ± 0%    134MB/s ± 0%   +8.93%  (p=0.000 n=9+8)
2017-11-05 15:59:02 +11:00
Nigel Tao f6e072fe77 Add some more "inconsistent n_bits" checks.
This uncovered a bug in the decode_huffman_fast function. A follow up commit
will actually fix the bug.

For what it's worth, some benchmark numbers are below, but they're
possibly bogus, as we might be decoding incorrect (and truncated) output
for some inputs. Everything since "f968950 Read bits up front in
decode_huffman_fast" landed possibly has bogus numbers too.

name                           old speed     new speed     delta
puffs_flate_decode_100k/clang  194MB/s ± 0%  204MB/s ± 1%   +5.02%  (p=0.008 n=5+5)
puffs_flate_decode_100k/gcc    180MB/s ± 3%  202MB/s ± 1%  +12.09%  (p=0.008 n=5+5)
2017-11-05 09:23:49 +11:00
Nigel Tao cf61ca11c3 Run the adler benchmarks for longer.
A future commit could set each benchmark's iteration count (call it N)
dynamically, but for now, the iteration counts are hard-coded. The rough
guiding principle is those N iterations should take more or less one
second, on my modern desktop CPU.

Prior to this commit, the adler benchmarks ran for the same N as the
zlib benchmarks. Each zlib decode involves exactly one adler checksum
computation, so both having the same N makes it very easy to calculate a
rough estimate of what fraction of zlib decoding is spent in adler code.
But in having the same N, the adler benchmarks run for a tiny fraction
of a second (and would run for an even tinier fraction if we further
optimize the adler code, e.g. with SIMD).

This commit raises that N for the adler benchmarks (but not for the zlib
benchmarks), which should hopefully make the adler numbers a little more
reliable. It will make that very easy calculation slightly less easy,
but it'd still be trivial.

The benchmark numbers do change. I'm not entirely sure why, but keep in
mind that this commit only changes the benchmark harness code, not the
actual std/flate code running in that harness.

name                           old speed      new speed      delta

puffs_adler32_10k/clang        2.45GB/s ± 0%  2.40GB/s ± 1%  -2.00%  (p=0.008 n=5+5)
puffs_adler32_100k/clang       2.44GB/s ± 0%  2.40GB/s ± 1%  -1.67%  (p=0.008 n=5+5)
puffs_flate_decode_1k/clang     133MB/s ± 0%   131MB/s ± 1%  -1.37%  (p=0.008 n=5+5)
puffs_flate_decode_10k/clang    174MB/s ± 0%   171MB/s ± 1%  -1.66%  (p=0.008 n=5+5)
puffs_flate_decode_100k/clang   205MB/s ± 0%   202MB/s ± 0%  -1.28%  (p=0.016 n=5+4)
puffs_zlib_decode_10k/clang     162MB/s ± 0%   159MB/s ± 1%  -1.94%  (p=0.008 n=5+5)
puffs_zlib_decode_100k/clang    188MB/s ± 0%   185MB/s ± 1%  -1.60%  (p=0.008 n=5+5)

puffs_adler32_10k/gcc          3.25GB/s ± 0%  3.26GB/s ± 0%  +0.25%  (p=0.016 n=4+5)
puffs_adler32_100k/gcc         3.25GB/s ± 0%  3.27GB/s ± 0%  +0.39%  (p=0.008 n=5+5)
puffs_flate_decode_1k/gcc       132MB/s ± 0%   132MB/s ± 0%    ~     (p=0.151 n=5+5)
puffs_flate_decode_10k/gcc      170MB/s ± 0%   170MB/s ± 0%    ~     (p=0.738 n=5+5)
puffs_flate_decode_100k/gcc     207MB/s ± 0%   207MB/s ± 0%    ~     (p=0.841 n=5+5)
puffs_zlib_decode_10k/gcc       161MB/s ± 1%   162MB/s ± 0%    ~     (p=0.841 n=5+5)
puffs_zlib_decode_100k/gcc      194MB/s ± 0%   194MB/s ± 0%    ~     (p=0.841 n=5+5)
2017-11-05 07:16:04 +11:00
Nigel Tao 1487da7af8 Add a -focus flag. 2017-11-03 17:54:29 +11:00
Nigel Tao aec5715f12 Declare proc_filename exactly once. 2017-11-03 15:09:22 +11:00
Nigel Tao d7c910fd1b Avoid "array decl needs an explicit size". 2017-11-03 12:11:21 +11:00
Nigel Tao e3822f9951 Re-introduce puffs_base__limit1.
It's not immediately obvious why this data structure performs differently, but
on net, the benchmark numbers seem better. It also won't require any
PUFFS_USE_NO_OP_PERFORMANCE_HACKS or "redundant but mysteriously performant"
use_limit fields.

name                             old speed      new speed      delta

puffs_adler32_10k/clang          2.46GB/s ± 0%  2.43GB/s ± 3%     ~     (p=0.513 n=5+10)
puffs_adler32_100k/clang         2.46GB/s ± 0%  2.44GB/s ± 2%     ~     (p=0.241 n=5+10)
puffs_flate_decode_1k/clang       122MB/s ± 1%   119MB/s ± 2%   -1.86%  (p=0.013 n=5+10)
puffs_flate_decode_10k/clang      147MB/s ± 0%   142MB/s ± 2%   -3.47%  (p=0.001 n=5+10)
puffs_flate_decode_100k/clang     161MB/s ± 0%   161MB/s ± 4%     ~     (p=0.635 n=4+10)
puffs_zlib_decode_10k/clang       139MB/s ± 0%   135MB/s ± 1%   -2.69%  (p=0.001 n=5+10)
puffs_zlib_decode_100k/clang      151MB/s ± 0%   152MB/s ± 3%     ~     (p=1.000 n=5+10)

puffs_adler32_10k/gcc            3.21GB/s ± 1%  3.18GB/s ± 2%     ~     (p=0.310 n=5+10)
puffs_adler32_100k/gcc           3.22GB/s ± 0%  3.19GB/s ± 2%     ~     (p=0.075 n=5+10)
puffs_flate_decode_1k/gcc         121MB/s ± 4%   126MB/s ± 1%   +4.54%  (p=0.002 n=5+8)
puffs_flate_decode_10k/gcc        144MB/s ± 1%   141MB/s ± 1%   -2.52%  (p=0.001 n=5+9)
puffs_flate_decode_100k/gcc       154MB/s ± 1%   158MB/s ± 0%   +2.39%  (p=0.001 n=5+9)
puffs_zlib_decode_10k/gcc         137MB/s ± 2%   135MB/s ± 1%     ~     (p=0.055 n=5+10)
puffs_zlib_decode_100k/gcc        146MB/s ± 2%   150MB/s ± 0%   +2.87%  (p=0.001 n=5+10)

puffs_gif_lzw_decode_20k/clang    132MB/s ± 0%   125MB/s ± 0%   -5.24%  (p=0.001 n=5+10)
puffs_gif_lzw_decode_100k/clang   176MB/s ± 0%   164MB/s ± 0%   -6.82%  (p=0.001 n=5+9)
puffs_gif_decode_1k/clang         363MB/s ± 0%   339MB/s ± 1%   -6.58%  (p=0.002 n=4+10)
puffs_gif_decode_10k/clang        138MB/s ± 0%   133MB/s ± 1%   -3.77%  (p=0.001 n=5+10)
puffs_gif_decode_100k/clang       125MB/s ± 0%   119MB/s ± 0%   -4.79%  (p=0.001 n=5+10)
puffs_gif_decode_1000k/clang      127MB/s ± 0%   121MB/s ± 0%   -5.04%  (p=0.001 n=5+10)

puffs_gif_lzw_decode_20k/gcc      141MB/s ± 0%   140MB/s ± 2%     ~     (p=1.000 n=5+10)
puffs_gif_lzw_decode_100k/gcc     182MB/s ± 0%   180MB/s ± 2%   -1.56%  (p=0.001 n=5+10)
puffs_gif_decode_1k/gcc           363MB/s ± 0%   409MB/s ± 3%  +12.71%  (p=0.001 n=5+10)
puffs_gif_decode_10k/gcc          127MB/s ± 0%   161MB/s ± 3%  +26.15%  (p=0.001 n=5+10)
puffs_gif_decode_100k/gcc         112MB/s ± 0%   136MB/s ± 2%  +21.95%  (p=0.001 n=5+10)
puffs_gif_decode_1000k/gcc        114MB/s ± 0%   140MB/s ± 2%  +22.78%  (p=0.001 n=5+10)
2017-11-03 10:09:57 +11:00
Nigel Tao aa396e7992 Change cgen's limit from a count to a pointer. 2017-10-27 10:28:35 +11:00
Nigel Tao e0836d73a8 Migrate tests to use new limit mechanism. 2017-10-27 07:49:08 +11:00
Nigel Tao 21bbbd422f Generate code for "mark in.dst".
The generated code is ugly and the language design is not finalized, but
the flate_backref_crosses_blocks test (testing a correctness bug) now
passes. We can fix the ugliness in subsequent commits.
2017-10-22 14:44:53 +11:00
Nigel Tao bcc7171b5a Add a flate_backref_crosses_blocks test. 2017-10-22 12:54:57 +11:00
Nigel Tao 27a33ff27c Add a test/testdata/artificial directory. 2017-10-22 11:41:06 +11:00
Nigel Tao f29c4e278f Follow the pkg__type__method convention for base. 2017-10-21 15:59:08 +11:00
Nigel Tao 6815375de5 Generate double '_'s for pkg__type__method names. 2017-10-21 15:10:25 +11:00
Nigel Tao 2639f8f46a Rename std/flate's decoder to flate_decoder. 2017-10-21 13:05:19 +11:00
Nigel Tao 1c75edb32e Calculate the zlib checksum. 2017-10-21 12:38:23 +11:00
Nigel Tao 74f3b76b09 Rename the adler struct to adler32. 2017-10-19 20:49:20 +11:00
Nigel Tao f9cea5c17c Fix the adler.update implementation. 2017-10-19 20:45:43 +11:00
Nigel Tao e7d01e7b2a Fix flate decoding across multiple I/O calls. 2017-10-14 16:27:53 +11:00
Nigel Tao 42933d4557 Refactor some tests. 2017-10-14 15:00:31 +11:00
Nigel Tao fca0cab724 Add write and read limits to some tests. 2017-10-14 14:52:54 +11:00
Nigel Tao d4372cb5b8 Change license to Apache 2.
Also remove explicit PATENTS file. The new LICENSE file already
discusses patents.

Note that the Puffs project has not yet been publicly released.
2017-10-12 11:17:41 +11:00
Nigel Tao ac405d485b Rename some internal test functions. 2017-10-07 15:57:18 +11:00
Nigel Tao d275b379a3 Remove the idea of limited reads and writes. 2017-10-07 15:32:00 +11:00
Nigel Tao 010c4db54f Allow explicit returns of built-in status codes. 2017-10-06 17:34:58 +11:00
Nigel Tao 2e883fcc23 Drop the "?" from "copy_history32?".
This makes the caller, not the implementation, responsible for detecting
short reads and writes. This (the lack of "?") is consistent with the
slice.copy_from method.

name                           old speed     new speed     delta
puffs_flate_decode_1k/clang    117MB/s ± 6%  120MB/s ± 0%   +2.31%  (p=0.016 n=5+5)
puffs_flate_decode_10k/clang   153MB/s ± 1%  155MB/s ± 0%   +1.16%  (p=0.008 n=5+5)
puffs_flate_decode_100k/clang  168MB/s ± 1%  177MB/s ± 0%   +5.41%  (p=0.008 n=5+5)
puffs_flate_decode_1k/gcc      127MB/s ± 1%  143MB/s ± 0%  +12.15%  (p=0.008 n=5+5)
puffs_flate_decode_10k/gcc     145MB/s ± 0%  180MB/s ± 1%  +24.00%  (p=0.008 n=5+5)
puffs_flate_decode_100k/gcc    163MB/s ± 0%  212MB/s ± 1%  +29.90%  (p=0.008 n=5+5)
2017-10-06 17:02:09 +11:00
Nigel Tao cf6b1be5b5 Re-organize flate Huffman decoder table bits.
Benchmarks are mixed, but it seems conceptually cleaner to arrange the
bits in the order that they're examined.

name                           old speed     new speed     delta
puffs_flate_decode_1k/clang    125MB/s ± 0%  123MB/s ± 1%  -1.91%  (p=0.008 n=5+5)
puffs_flate_decode_10k/clang   163MB/s ± 1%  157MB/s ± 0%  -3.55%  (p=0.008 n=5+5)
puffs_flate_decode_100k/clang  175MB/s ± 0%  173MB/s ± 0%  -0.75%  (p=0.016 n=4+5)
puffs_flate_decode_1k/gcc      127MB/s ± 0%  130MB/s ± 0%  +2.14%  (p=0.008 n=5+5)
puffs_flate_decode_10k/gcc     142MB/s ± 1%  149MB/s ± 0%  +4.99%  (p=0.016 n=5+4)
puffs_flate_decode_100k/gcc    162MB/s ± 2%  168MB/s ± 0%  +3.66%  (p=0.016 n=5+4)
2017-10-06 13:07:49 +11:00
Nigel Tao e8eff8f5e4 Test that flate history copy doesn't go too far. 2017-10-02 14:39:07 +11:00
Nigel Tao a28dba7fd9 Rename C's foo_constructor to foo_initializer. 2017-09-30 13:01:26 +10:00
Nigel Tao 552fcde35a Remove C destructor functions; they're no-ops. 2017-09-30 12:57:55 +10:00
Nigel Tao 04e8a73d10 Update std/flate's decoder.history_index. 2017-09-30 11:56:30 +10:00
Nigel Tao 6a48794033 Add a flate history test. 2017-09-23 12:39:02 +10:00
Nigel Tao b3710e583f Add a zlib decoder to std/flate. 2017-08-05 16:34:03 +10:00
Nigel Tao 0a45eee9d0 Fix some comment grammar. 2017-07-29 17:24:01 +10:00
Nigel Tao 7cf7829c9e Implement std/flate Huffman table redirects. 2017-07-29 16:25:00 +10:00
Nigel Tao bfbcae4365 Decode std/flate fixed Huffman tables. 2017-07-28 13:28:31 +10:00
Nigel Tao cbcf1b6025 Remove test_puffs_flate_work_in_progress. 2017-07-28 11:33:48 +10:00
Nigel Tao d2aca16f3c Add a copy_history32 method. 2017-07-28 11:25:03 +10:00
Nigel Tao e772352036 Decode the lcode and dcode code lengths. 2017-07-22 16:45:54 +10:00
Nigel Tao f7fb991619 Implement init_huffman. 2017-07-21 15:55:49 +10:00
Nigel Tao 4b7a350a68 Rename the code_length_code_lengths field. 2017-07-21 10:09:44 +10:00
Nigel Tao fe7e6ed10e Handle suspending inside a skip32 call. 2017-07-16 19:28:09 +10:00
Nigel Tao fa18eec3e9 Fix returning limited vs short read. 2017-07-16 19:26:58 +10:00
Nigel Tao bd6d2ea3a1 Handle decoding a GIF in many short reads. 2017-07-16 19:25:49 +10:00
Nigel Tao 20f37a6dab Decode flate code length code lengths. 2017-07-16 12:51:44 +10:00
Nigel Tao 7b7dca4767 Start a flate decoder.decode_dynamic method. 2017-07-14 16:36:51 +10:00
Nigel Tao 85610b5df2 Add a throughput_counter concept.
Example benchstat summary of "puffs bench -mimic" output below, with
some blank lines added for clarity.

For example, gzip and zlib both add a checksum (crc32 and adler32
respectively) to raw flate. The headline mimic_{gzip,zlib}_decode_100k
numbers are 386µs and 347µs per op. Subtracting 73.2µs and 34.3µs for
the checksum times from the headline times give the same number: 313µs.

This 313µs differs from 358µs for raw flate (mimic_flate_decode_100k)
because zlib takes two different code paths for wrapped vs raw flate. In
the zlib source code, this is inflate.c vs infback.c respectively.

----
name                             time/op

mimic_adler32_10k/clang            3.74µs ± 1%
mimic_adler32_100k/clang           34.3µs ± 1%
mimic_crc32_10k/clang              8.10µs ± 1%
mimic_crc32_100k/clang             73.2µs ± 0%

mimic_flate_decode_1k/clang        4.31µs ± 1%
mimic_flate_decode_10k/clang       42.1µs ± 1%
mimic_flate_decode_100k/clang       358µs ± 1%
mimic_gzip_decode_10k/clang        49.8µs ± 1%
mimic_gzip_decode_100k/clang        386µs ± 0%
mimic_zlib_decode_10k/clang        45.3µs ± 0%
mimic_zlib_decode_100k/clang        347µs ± 1%

mimic_adler32_10k/gcc              3.72µs ± 1%
mimic_adler32_100k/gcc             34.3µs ± 2%
mimic_crc32_10k/gcc                8.06µs ± 0%
mimic_crc32_100k/gcc               73.1µs ± 0%

mimic_flate_decode_1k/gcc          4.26µs ± 0%
mimic_flate_decode_10k/gcc         42.0µs ± 0%
mimic_flate_decode_100k/gcc         356µs ± 0%
mimic_gzip_decode_10k/gcc          50.4µs ± 4%
mimic_gzip_decode_100k/gcc          390µs ± 2%
mimic_zlib_decode_10k/gcc          45.6µs ± 1%
mimic_zlib_decode_100k/gcc          348µs ± 1%

puffs_gif_lzw_decode_20k/clang      151µs ± 0%
puffs_gif_lzw_decode_100k/clang     566µs ± 0%
puffs_gif_decode_1k/clang          2.74µs ± 1%
puffs_gif_decode_10k/clang         73.8µs ± 0%
puffs_gif_decode_100k/clang        1.13ms ± 0%
puffs_gif_decode_1000k/clang       7.99ms ± 0%

mimic_gif_decode_1k/clang          6.44µs ± 0%
mimic_gif_decode_10k/clang          107µs ± 0%
mimic_gif_decode_100k/clang        1.38ms ± 0%
mimic_gif_decode_1000k/clang       9.79ms ± 0%

puffs_gif_lzw_decode_20k/gcc        127µs ± 0%
puffs_gif_lzw_decode_100k/gcc       501µs ± 0%
puffs_gif_decode_1k/gcc            2.45µs ± 1%
puffs_gif_decode_10k/gcc           65.1µs ± 0%
puffs_gif_decode_100k/gcc          1.01ms ± 0%
puffs_gif_decode_1000k/gcc         7.16ms ± 1%

mimic_gif_decode_1k/gcc            6.66µs ± 2%
mimic_gif_decode_10k/gcc            108µs ± 1%
mimic_gif_decode_100k/gcc          1.39ms ± 0%
mimic_gif_decode_1000k/gcc         9.93ms ± 1%

name                             speed

mimic_adler32_10k/clang          2.96GB/s ± 1%
mimic_adler32_100k/clang         2.91GB/s ± 1%
mimic_crc32_10k/clang            1.37GB/s ± 1%
mimic_crc32_100k/clang           1.37GB/s ± 0%

mimic_flate_decode_1k/clang       219MB/s ± 1%
mimic_flate_decode_10k/clang      263MB/s ± 1%
mimic_flate_decode_100k/clang     280MB/s ± 1%
mimic_gzip_decode_10k/clang       222MB/s ± 1%
mimic_gzip_decode_100k/clang      259MB/s ± 0%
mimic_zlib_decode_10k/clang       244MB/s ± 0%
mimic_zlib_decode_100k/clang      288MB/s ± 1%

mimic_adler32_10k/gcc            2.97GB/s ± 1%
mimic_adler32_100k/gcc           2.92GB/s ± 2%
mimic_crc32_10k/gcc              1.37GB/s ± 0%
mimic_crc32_100k/gcc             1.37GB/s ± 0%

mimic_flate_decode_1k/gcc         221MB/s ± 0%
mimic_flate_decode_10k/gcc        264MB/s ± 0%
mimic_flate_decode_100k/gcc       281MB/s ± 0%
mimic_gzip_decode_10k/gcc         219MB/s ± 4%
mimic_gzip_decode_100k/gcc        256MB/s ± 2%
mimic_zlib_decode_10k/gcc         243MB/s ± 1%
mimic_zlib_decode_100k/gcc        287MB/s ± 1%

puffs_gif_lzw_decode_20k/clang    128MB/s ± 0%
puffs_gif_lzw_decode_100k/clang   177MB/s ± 0%
puffs_gif_decode_1k/clang         373MB/s ± 1%
puffs_gif_decode_10k/clang        137MB/s ± 0%
puffs_gif_decode_100k/clang       122MB/s ± 0%
puffs_gif_decode_1000k/clang      125MB/s ± 0%

mimic_gif_decode_1k/clang         159MB/s ± 0%
mimic_gif_decode_10k/clang       94.2MB/s ± 0%
mimic_gif_decode_100k/clang       100MB/s ± 0%
mimic_gif_decode_1000k/clang      102MB/s ± 0%

puffs_gif_lzw_decode_20k/gcc      151MB/s ± 0%
puffs_gif_lzw_decode_100k/gcc     200MB/s ± 0%
puffs_gif_decode_1k/gcc           418MB/s ± 1%
puffs_gif_decode_10k/gcc          155MB/s ± 0%
puffs_gif_decode_100k/gcc         137MB/s ± 0%
puffs_gif_decode_1000k/gcc        140MB/s ± 1%

mimic_gif_decode_1k/gcc           154MB/s ± 2%
mimic_gif_decode_10k/gcc         93.0MB/s ± 1%
mimic_gif_decode_100k/gcc        98.9MB/s ± 0%
mimic_gif_decode_1000k/gcc        101MB/s ± 1%
----
2017-07-14 13:09:45 +10:00
Nigel Tao 8be6dc3394 Add adler32 and crc32 benchmarks. 2017-07-14 13:01:08 +10:00
Nigel Tao 03572374dd Add zlib (not just raw flate) benchmark. 2017-07-14 12:00:00 +10:00