On a mid-range x86_64 laptop: wuffs_adler32_10k/clang9 1.35GB/s ± 0% 10.78GB/s ± 1% +700.65% (p=0.008 n=5+5) wuffs_adler32_100k/clang9 1.35GB/s ± 0% 12.88GB/s ± 0% +856.09% (p=0.008 n=5+5) wuffs_adler32_10k/gcc10 1.85GB/s ± 0% 11.21GB/s ± 0% +506.34% (p=0.008 n=5+5) wuffs_adler32_100k/gcc10 1.84GB/s ± 0% 11.43GB/s ± 0% +521.51% (p=0.008 n=5+5) mimic_adler32_10k 1.76GB/s ± 0% 1.76GB/s ± 0% ~ (p=0.841 n=5+5) mimic_adler32_100k 1.72GB/s ± 0% 1.72GB/s ± 0% ~ (p=0.222 n=5+5) wuffs_zlib_decode_10k/clang9 124MB/s ± 0% 139MB/s ± 0% +12.29% (p=0.016 n=4+5) wuffs_zlib_decode_100k/clang9 148MB/s ± 0% 155MB/s ± 0% +4.21% (p=0.008 n=5+5) wuffs_zlib_decode_10k/gcc10 133MB/s ± 1% 141MB/s ± 0% +5.75% (p=0.008 n=5+5) wuffs_zlib_decode_100k/gcc10 159MB/s ± 0% 172MB/s ± 0% +7.77% (p=0.016 n=5+4) mimic_zlib_decode_10k 133MB/s ± 0% 133MB/s ± 0% +0.38% (p=0.008 n=5+5) mimic_zlib_decode_100k 160MB/s ± 0% 160MB/s ± 0% ~ (p=0.841 n=5+5) wuffs_png_decode_image_19k_8bpp/clang9 90.6MB/s ± 0% 94.7MB/s ± 0% +4.58% (p=0.016 n=5+4) wuffs_png_decode_image_40k_24bpp/clang9 129MB/s ± 0% 135MB/s ± 0% +4.49% (p=0.008 n=5+5) wuffs_png_decode_image_77k_8bpp/clang9 335MB/s ± 0% 351MB/s ± 0% +4.62% (p=0.008 n=5+5) wuffs_png_decode_image_552k_32bpp_ignore_checksum/clang9 322MB/s ± 0% 320MB/s ± 1% -0.71% (p=0.016 n=4+5) wuffs_png_decode_image_552k_32bpp_verify_checksum/clang9 256MB/s ± 0% 308MB/s ± 0% +20.24% (p=0.008 n=5+5) wuffs_png_decode_image_4002k_24bpp/clang9 131MB/s ± 0% 138MB/s ± 0% +5.24% (p=0.008 n=5+5) wuffs_png_decode_image_19k_8bpp/gcc10 95.1MB/s ± 0% 100.8MB/s ± 1% +6.08% (p=0.008 n=5+5) wuffs_png_decode_image_40k_24bpp/gcc10 131MB/s ± 0% 136MB/s ± 0% +3.65% (p=0.008 n=5+5) wuffs_png_decode_image_77k_8bpp/gcc10 350MB/s ± 1% 371MB/s ± 0% +6.06% (p=0.008 n=5+5) wuffs_png_decode_image_552k_32bpp_ignore_checksum/gcc10 327MB/s ± 0% 333MB/s ± 0% +1.66% (p=0.008 n=5+5) wuffs_png_decode_image_552k_32bpp_verify_checksum/gcc10 274MB/s ± 0% 318MB/s ± 0% +15.93% (p=0.008 n=5+5) wuffs_png_decode_image_4002k_24bpp/gcc10 132MB/s ± 0% 138MB/s ± 0% +4.73% (p=0.008 n=5+5) mimic_png_decode_image_19k_8bpp 58.1MB/s ± 0% 58.2MB/s ± 0% +0.27% (p=0.008 n=5+5) mimic_png_decode_image_40k_24bpp 73.0MB/s ± 1% 73.3MB/s ± 0% +0.45% (p=0.008 n=5+5) mimic_png_decode_image_77k_8bpp 177MB/s ± 0% 177MB/s ± 0% ~ (p=0.421 n=5+5) mimic_png_decode_image_552k_32bpp_ignore_checksum skipped mimic_png_decode_image_552k_32bpp_verify_checksum 146MB/s ± 0% 146MB/s ± 0% ~ (p=0.310 n=5+5) mimic_png_decode_image_4002k_24bpp 104MB/s ± 0% 104MB/s ± 0% ~ (p=0.056 n=5+5)
Adler-32
Adler-32 is a checksum algorithm that hashes byte sequences to 32 bit values. It is named after its inventor, Mark Adler, who also co-invented the Gzip and Zlib compressed file formats. Amongst other differences, Gzip uses CRC-32 as its checksum and Zlib uses Adler-32.
The algorithm, described in RFC 1950,
is simple. Conceptually, there are two unsigned integers s1 and s2 of
infinite precision, initialized to 0 and 1. These two accumulators are
updated for every input byte src[i]. At the end of the loop, s1 is 1 plus
the sum of all source bytes and s2 is the sum of all (intermediate and final)
s1 values:
var s1 = 1;
var s2 = 0;
for_each i in_the_range_of src {
s1 = s1 + src[i];
s2 = s2 + s1;
}
return ((s2 % 65521) << 16) | (s1 % 65521);
The final uint32_t hash value is composed of two 16-bit values: (s1 % 65521) in the low 16 bits and (s2 % 65521) in the high 16 bits. 65521 is
the largest prime number less than (1 << 16).
Infinite precision arithmetic requires arbitrarily large amounts of memory. In
practice, computing the Adler-32 hash instead uses a uint32_t typed s1 and
s2, modifying the algorithm to be concious of overflow inside the loop:
uint32_t s1 = 1;
uint32_t s2 = 0;
for_each i in_the_range_of src {
s1 = (s1 + src[i]) % 65521;
s2 = (s2 + s1) % 65521;
}
return (s2 << 16) | s1;
The loop can be split into two levels, so that the relatively expensive modulo operation can be hoisted out of the inner loop:
uint32_t s1 = 1;
uint32_t s2 = 0;
for_each_sub_slice s of_length_up_to M partitioning src {
for_each i in_the_range_of s {
s1 = s1 + s[i];
s2 = s2 + s1;
}
s1 = s1 % 65521;
s2 = s2 % 65521;
}
return (s2 << 16) | s1;
We just need to find the largest M such that the inner loop cannot overflow.
The worst case scenario is that s1 and s2 both start the inner loop at
65520 and every subsequent src[i] byte equals 0xFF. A simple
computation finds that the largest
non-overflowing M is 5552.
In a happy coincidence, 5552 is an exact multiple of 16, which often works well with loop unrolling and with SIMD alignment.
Comparison with CRC-32
Adler-32 is a very simple hashing algorithm. While its output is nominally a
uint32_t value, it isn't uniformly distributed across the entire uint32_t
range. The [65521, 65535] range of each 16-bit half of an Adler-32 checksum
is never touched.
While neither Adler-32 or CRC-32 are cryptographic hash functions, there is
still a stark difference in the patterns (or lack of) in their hash values of
the N-byte string consisting entirely of zeroes, as this Go
program shows:
N Adler-32 CRC-32 Input
0 0x00000001 0x00000000 ""
1 0x00010001 0xD202EF8D "\x00"
2 0x00020001 0x41D912FF "\x00\x00"
3 0x00030001 0xFF41D912 "\x00\x00\x00"
4 0x00040001 0x2144DF1C "\x00\x00\x00\x00"
5 0x00050001 0xC622F71D "\x00\x00\x00\x00\x00"
6 0x00060001 0xB1C2A1A3 "\x00\x00\x00\x00\x00\x00"
7 0x00070001 0x9D6CDF7E "\x00\x00\x00\x00\x00\x00\x00"
Adler-32 is a simpler algorithm than CRC-32. At the time Adler-32 was invented, it had noticeably higher throughput. With modern SIMD implementations, that performance difference has largely disappeared.
Worked Example
A worked example for calculating the Adler-32 hash of the three byte input
"Hi\n", starting from the initial state (s1 = 1) and (s2 = 0):
src[i] ((s2 << 16) | s1)
---- 0x00000001
0x48 0x00490049
0x69 0x00FB00B2
0x0A 0x01B700BC