While the std/adler32 benchmarks got better, the std/zlib and std/png benchmarks got worse. This commit will soon be followed by a rollback, but it is committed anyway so that we can refer to these numbers in the git log. Binary size, before: 4056 gen/lib/c/clang-9-dynamic/wuffs-std-adler32.lo 4104 gen/lib/c/clang-9-static/wuffs-std-adler32.o 4624 gen/lib/c/gcc-dynamic/wuffs-std-adler32.lo 4680 gen/lib/c/gcc-static/wuffs-std-adler32.o After: 4952 gen/lib/c/clang-9-dynamic/wuffs-std-adler32.lo 5016 gen/lib/c/clang-9-static/wuffs-std-adler32.o 5512 gen/lib/c/gcc-dynamic/wuffs-std-adler32.lo 5576 gen/lib/c/gcc-static/wuffs-std-adler32.o name old speed new speed delta wuffs_adler32_10k/clang9 11.5GB/s ± 1% 15.0GB/s ± 5% +30.73% (p=0.000 n=9+10) wuffs_adler32_100k/clang9 13.5GB/s ± 1% 15.6GB/s ±14% +16.06% (p=0.001 n=9+10) wuffs_adler32_10k/gcc10 11.3GB/s ± 2% 15.6GB/s ± 1% +38.27% (p=0.000 n=9+9) wuffs_adler32_100k/gcc10 11.3GB/s ± 1% 19.4GB/s ± 1% +71.20% (p=0.000 n=7+9) wuffs_zlib_decode_10k/clang9 223MB/s ± 1% 216MB/s ± 2% -3.45% (p=0.000 n=10+10) wuffs_zlib_decode_100k/clang9 285MB/s ± 1% 284MB/s ± 1% ~ (p=0.243 n=9+10) wuffs_zlib_decode_10k/gcc10 232MB/s ± 1% 216MB/s ± 1% -6.70% (p=0.000 n=10+9) wuffs_zlib_decode_100k/gcc10 291MB/s ± 1% 280MB/s ± 1% -3.70% (p=0.000 n=9+8) wuffs_png_decode_image_19k_8bpp/clang9 152MB/s ± 1% 143MB/s ± 3% -5.89% (p=0.000 n=8+9) wuffs_png_decode_image_40k_24bpp/clang9 171MB/s ± 1% 173MB/s ± 1% ~ (p=0.094 n=9+9) wuffs_png_decode_image_77k_8bpp/clang9 541MB/s ± 1% 509MB/s ± 1% -5.77% (p=0.000 n=8+9) wuffs_png_decode_image_552k_32bpp_ignore_checksum/clang9 479MB/s ± 1% 464MB/s ± 3% -3.20% (p=0.000 n=8+10) wuffs_png_decode_image_552k_32bpp_verify_checksum/clang9 461MB/s ± 1% 452MB/s ± 1% -2.02% (p=0.000 n=8+10) wuffs_png_decode_image_4002k_24bpp/clang9 177MB/s ± 1% 175MB/s ± 2% -1.14% (p=0.004 n=10+10) wuffs_png_decode_image_19k_8bpp/gcc10 159MB/s ± 1% 153MB/s ± 1% -3.54% (p=0.000 n=8+10) wuffs_png_decode_image_40k_24bpp/gcc10 188MB/s ± 1% 187MB/s ± 1% -0.47% (p=0.021 n=10+9) wuffs_png_decode_image_77k_8bpp/gcc10 557MB/s ± 0% 546MB/s ± 4% -1.94% (p=0.000 n=8+10) wuffs_png_decode_image_552k_32bpp_ignore_checksum/gcc10 510MB/s ± 0% 493MB/s ± 0% -3.33% (p=0.000 n=10+9) wuffs_png_decode_image_552k_32bpp_verify_checksum/gcc10 486MB/s ± 1% 478MB/s ± 0% -1.77% (p=0.000 n=8+9) wuffs_png_decode_image_4002k_24bpp/gcc10 189MB/s ± 1% 187MB/s ± 1% -1.26% (p=0.000 n=10+10)
Adler-32
Adler-32 is a checksum algorithm that hashes byte sequences to 32 bit values. It is named after its inventor, Mark Adler, who also co-invented the Gzip and Zlib compressed file formats. Amongst other differences, Gzip uses CRC-32 as its checksum and Zlib uses Adler-32.
The algorithm, described in RFC 1950,
is simple. Conceptually, there are two unsigned integers s1 and s2 of
infinite precision, initialized to 0 and 1. These two accumulators are
updated for every input byte src[i]. At the end of the loop, s1 is 1 plus
the sum of all source bytes and s2 is the sum of all (intermediate and final)
s1 values:
var s1 = 1;
var s2 = 0;
for_each i in_the_range_of src {
s1 = s1 + src[i];
s2 = s2 + s1;
}
return ((s2 % 65521) << 16) | (s1 % 65521);
The final uint32_t hash value is composed of two 16-bit values: (s1 % 65521) in the low 16 bits and (s2 % 65521) in the high 16 bits. 65521 is
the largest prime number less than (1 << 16).
Infinite precision arithmetic requires arbitrarily large amounts of memory. In
practice, computing the Adler-32 hash instead uses a uint32_t typed s1 and
s2, modifying the algorithm to be concious of overflow inside the loop:
uint32_t s1 = 1;
uint32_t s2 = 0;
for_each i in_the_range_of src {
s1 = (s1 + src[i]) % 65521;
s2 = (s2 + s1) % 65521;
}
return (s2 << 16) | s1;
The loop can be split into two levels, so that the relatively expensive modulo operation can be hoisted out of the inner loop:
uint32_t s1 = 1;
uint32_t s2 = 0;
for_each_sub_slice s of_length_up_to M partitioning src {
for_each i in_the_range_of s {
s1 = s1 + s[i];
s2 = s2 + s1;
}
s1 = s1 % 65521;
s2 = s2 % 65521;
}
return (s2 << 16) | s1;
We just need to find the largest M such that the inner loop cannot overflow.
The worst case scenario is that s1 and s2 both start the inner loop at
65520 and every subsequent src[i] byte equals 0xFF. A simple
computation finds that the largest
non-overflowing M is 5552.
In a happy coincidence, 5552 is an exact multiple of 16, which often works well with loop unrolling and with SIMD alignment.
Comparison with CRC-32
Adler-32 is a very simple hashing algorithm. While its output is nominally a
uint32_t value, it isn't uniformly distributed across the entire uint32_t
range. The [65521, 65535] range of each 16-bit half of an Adler-32 checksum
is never touched.
While neither Adler-32 or CRC-32 are cryptographic hash functions, there is
still a stark difference in the patterns (or lack of) in their hash values of
the N-byte string consisting entirely of zeroes, as this Go
program shows:
N Adler-32 CRC-32 Input
0 0x00000001 0x00000000 ""
1 0x00010001 0xD202EF8D "\x00"
2 0x00020001 0x41D912FF "\x00\x00"
3 0x00030001 0xFF41D912 "\x00\x00\x00"
4 0x00040001 0x2144DF1C "\x00\x00\x00\x00"
5 0x00050001 0xC622F71D "\x00\x00\x00\x00\x00"
6 0x00060001 0xB1C2A1A3 "\x00\x00\x00\x00\x00\x00"
7 0x00070001 0x9D6CDF7E "\x00\x00\x00\x00\x00\x00\x00"
Adler-32 is a simpler algorithm than CRC-32. At the time Adler-32 was invented, it had noticeably higher throughput. With modern SIMD implementations, that performance difference has largely disappeared.
Worked Example
A worked example for calculating the Adler-32 hash of the three byte input
"Hi\n", starting from the initial state (s1 = 1) and (s2 = 0):
src[i] ((s2 << 16) | s1)
---- 0x00000001
0x48 0x00490049
0x69 0x00FB00B2
0x0A 0x01B700BC