On a Raspberry Pi 4 (32-bit armv7l) with -march=native and -mfpu=neon
("native" means "armv8-a+crc+simd"):
wuffs_adler32_10k/clang9 1.21GB/s ± 0% 6.58GB/s ± 1% +442.43% (p=0.008 n=5+5)
wuffs_adler32_100k/clang9 1.20GB/s ± 0% 6.58GB/s ± 0% +449.37% (p=0.008 n=5+5)
wuffs_adler32_10k/gcc8 1.24GB/s ± 0% 4.31GB/s ± 0% +248.87% (p=0.008 n=5+5)
wuffs_adler32_100k/gcc8 1.22GB/s ± 0% 4.29GB/s ± 0% +251.09% (p=0.008 n=5+5)
mimic_adler32_10k 1.40GB/s ± 0% 1.40GB/s ± 0% -0.04% (p=0.008 n=5+5)
mimic_adler32_100k 1.35GB/s ± 0% 1.38GB/s ± 0% +2.77% (p=0.008 n=5+5)
wuffs_zlib_decode_10k/clang9 85.3MB/s ± 0% 91.1MB/s ± 0% +6.83% (p=0.008 n=5+5)
wuffs_zlib_decode_100k/clang9 107MB/s ± 0% 113MB/s ± 0% +5.20% (p=0.008 n=5+5)
wuffs_zlib_decode_10k/gcc8 86.1MB/s ± 0% 91.0MB/s ± 0% +5.69% (p=0.008 n=5+5)
wuffs_zlib_decode_100k/gcc8 104MB/s ± 0% 111MB/s ± 0% +7.08% (p=0.016 n=5+4)
mimic_zlib_decode_10k 108MB/s ± 0% 108MB/s ± 0% ~ (p=0.548 n=5+5)
mimic_zlib_decode_100k 142MB/s ± 0% 141MB/s ± 1% ~ (p=0.310 n=5+5)
wuffs_png_decode_image_19k_8bpp/clang9 69.9MB/s ± 0% 74.3MB/s ± 0% +6.37% (p=0.016 n=5+4)
wuffs_png_decode_image_40k_24bpp/clang9 99.4MB/s ± 0% 104.3MB/s ± 0% +4.92% (p=0.008 n=5+5)
wuffs_png_decode_image_77k_8bpp/clang9 266MB/s ± 0% 280MB/s ± 0% +5.21% (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_ignore_checksum/clang9 165MB/s ± 0% 169MB/s ± 0% +2.45% (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_verify_checksum/clang9 146MB/s ± 0% 163MB/s ± 0% +11.86% (p=0.008 n=5+5)
wuffs_png_decode_image_4002k_24bpp/clang9 93.2MB/s ± 0% 97.7MB/s ± 0% +4.78% (p=0.008 n=5+5)
wuffs_png_decode_image_19k_8bpp/gcc8 70.3MB/s ± 0% 74.3MB/s ± 0% +5.70% (p=0.008 n=5+5)
wuffs_png_decode_image_40k_24bpp/gcc8 101MB/s ± 0% 107MB/s ± 0% +5.67% (p=0.008 n=5+5)
wuffs_png_decode_image_77k_8bpp/gcc8 247MB/s ± 0% 259MB/s ± 0% +4.71% (p=0.008 n=5+5)
wuffs_png_decode_image_552k_32bpp_ignore_checksum/gcc8 162MB/s ± 0% 162MB/s ± 0% +0.09% (p=0.032 n=5+4)
wuffs_png_decode_image_552k_32bpp_verify_checksum/gcc8 140MB/s ± 0% 155MB/s ± 0% +10.67% (p=0.008 n=5+5)
wuffs_png_decode_image_4002k_24bpp/gcc8 94.1MB/s ± 0% 99.4MB/s ± 0% +5.57% (p=0.008 n=5+5)
mimic_png_decode_image_19k_8bpp 43.6MB/s ± 0% 43.3MB/s ± 1% ~ (p=0.167 n=5+5)
mimic_png_decode_image_40k_24bpp 54.6MB/s ± 0% 54.6MB/s ± 0% ~ (p=1.000 n=5+5)
mimic_png_decode_image_77k_8bpp 125MB/s ± 0% 123MB/s ± 0% -1.03% (p=0.016 n=5+4)
mimic_png_decode_image_552k_32bpp_ignore_checksum skipped
mimic_png_decode_image_552k_32bpp_verify_checksum 102MB/s ± 0% 101MB/s ± 0% -0.43% (p=0.008 n=5+5)
mimic_png_decode_image_4002k_24bpp 81.3MB/s ± 0% 81.8MB/s ± 0% +0.62% (p=0.008 n=5+5)
Adler-32
Adler-32 is a checksum algorithm that hashes byte sequences to 32 bit values. It is named after its inventor, Mark Adler, who also co-invented the Gzip and Zlib compressed file formats. Amongst other differences, Gzip uses CRC-32 as its checksum and Zlib uses Adler-32.
The algorithm, described in RFC 1950,
is simple. Conceptually, there are two unsigned integers s1 and s2 of
infinite precision, initialized to 0 and 1. These two accumulators are
updated for every input byte src[i]. At the end of the loop, s1 is 1 plus
the sum of all source bytes and s2 is the sum of all (intermediate and final)
s1 values:
var s1 = 1;
var s2 = 0;
for_each i in_the_range_of src {
s1 = s1 + src[i];
s2 = s2 + s1;
}
return ((s2 % 65521) << 16) | (s1 % 65521);
The final uint32_t hash value is composed of two 16-bit values: (s1 % 65521) in the low 16 bits and (s2 % 65521) in the high 16 bits. 65521 is
the largest prime number less than (1 << 16).
Infinite precision arithmetic requires arbitrarily large amounts of memory. In
practice, computing the Adler-32 hash instead uses a uint32_t typed s1 and
s2, modifying the algorithm to be concious of overflow inside the loop:
uint32_t s1 = 1;
uint32_t s2 = 0;
for_each i in_the_range_of src {
s1 = (s1 + src[i]) % 65521;
s2 = (s2 + s1) % 65521;
}
return (s2 << 16) | s1;
The loop can be split into two levels, so that the relatively expensive modulo operation can be hoisted out of the inner loop:
uint32_t s1 = 1;
uint32_t s2 = 0;
for_each_sub_slice s of_length_up_to M partitioning src {
for_each i in_the_range_of s {
s1 = s1 + s[i];
s2 = s2 + s1;
}
s1 = s1 % 65521;
s2 = s2 % 65521;
}
return (s2 << 16) | s1;
We just need to find the largest M such that the inner loop cannot overflow.
The worst case scenario is that s1 and s2 both start the inner loop at
65520 and every subsequent src[i] byte equals 0xFF. A simple
computation finds that the largest
non-overflowing M is 5552.
In a happy coincidence, 5552 is an exact multiple of 16, which often works well with loop unrolling and with SIMD alignment.
Comparison with CRC-32
Adler-32 is a very simple hashing algorithm. While its output is nominally a
uint32_t value, it isn't uniformly distributed across the entire uint32_t
range. The [65521, 65535] range of each 16-bit half of an Adler-32 checksum
is never touched.
While neither Adler-32 or CRC-32 are cryptographic hash functions, there is
still a stark difference in the patterns (or lack of) in their hash values of
the N-byte string consisting entirely of zeroes, as this Go
program shows:
N Adler-32 CRC-32 Input
0 0x00000001 0x00000000 ""
1 0x00010001 0xD202EF8D "\x00"
2 0x00020001 0x41D912FF "\x00\x00"
3 0x00030001 0xFF41D912 "\x00\x00\x00"
4 0x00040001 0x2144DF1C "\x00\x00\x00\x00"
5 0x00050001 0xC622F71D "\x00\x00\x00\x00\x00"
6 0x00060001 0xB1C2A1A3 "\x00\x00\x00\x00\x00\x00"
7 0x00070001 0x9D6CDF7E "\x00\x00\x00\x00\x00\x00\x00"
Adler-32 is a simpler algorithm than CRC-32. At the time Adler-32 was invented, it had noticeably higher throughput. With modern SIMD implementations, that performance difference has largely disappeared.
Worked Example
A worked example for calculating the Adler-32 hash of the three byte input
"Hi\n", starting from the initial state (s1 = 1) and (s2 = 0):
src[i] ((s2 << 16) | s1)
---- 0x00000001
0x48 0x00490049
0x69 0x00FB00B2
0x0A 0x01B700BC