[regression] Add initial regression test framework

The regression tests run nightly or on the `regression`
branch for convenience. The results get uploaded as the
artifacts of the job. If they change, check the diff
printed in the job. If all is well, download the new
results and commit them to the repo.

This code will only run on a UNIX like platform. It
could be made to run on Windows, but I don't think that
it is necessary. It also uses C99.

* data: This module defines the data to run tests on.
  It downloads data from a URL into a cache directory,
  checks it against a checksum, and unpacks it. It also
  provides helpers for accessing the data.
* config: This module defines the configs to run tests
  with. A config is a set of API parameters and a set of
  CLI flags.
* result: This module is a helper for method that defines
  the result type.
* method: This module defines the compression methods
  to test. It is what runs the regression test using the
  data and the config. It reports the total compressed
  size, or an error/skip.
* test: This is the test binary that runs the tests for
  every (data, config, method) tuple, and prints the
  results to the output file and stderr.
* results.csv: The results that the current commit is
  expected to produce.
This commit is contained in:
Nick Terrell
2018-11-29 14:33:04 -08:00
parent 787532b91a
commit 4aaa36f74b
13 changed files with 1608 additions and 1 deletions
+43
View File
@@ -0,0 +1,43 @@
Data, Config, Method, Total compressed size
silesia.tar, level -5, simple, 106176430
silesia.tar, level -3, simple, 98476550
silesia.tar, level -1, simple, 87206767
silesia.tar, level 0, simple, 66996953
silesia.tar, level 1, simple, 73658303
silesia.tar, level 3, simple, 66996953
silesia.tar, level 4, simple, 65996020
silesia.tar, level 5, simple, 64421326
silesia.tar, level 6, simple, 62388673
silesia.tar, level 7, simple, 61159525
silesia.tar, level 9, simple, 60214921
silesia.tar, level 13, simple, 58428642
silesia.tar, level 16, simple, 56363759
silesia.tar, level 19, simple, 53274173
silesia, level -5, cli file, 106202112
silesia, level -3, cli file, 98518660
silesia, level -1, cli file, 87226203
silesia, level 0, cli file, 67049190
silesia, level 1, cli file, 73676282
silesia, level 3, cli file, 67049190
silesia, level 4, cli file, 66090040
silesia, level 5, cli file, 64503721
silesia, level 6, cli file, 62446177
silesia, level 7, cli file, 61217029
silesia, level 9, cli file, 60282841
silesia, level 13, cli file, 58480658
silesia, level 16, cli file, 56414170
silesia, level 19, cli file, 53365292
silesia.tar, level -5, cli file, 106250113
silesia.tar, level -3, cli file, 98550747
silesia.tar, level -1, cli file, 87227322
silesia.tar, level 0, cli file, 67111168
silesia.tar, level 1, cli file, 73694374
silesia.tar, level 3, cli file, 67111168
silesia.tar, level 4, cli file, 66154079
silesia.tar, level 5, cli file, 64546998
silesia.tar, level 6, cli file, 62458454
silesia.tar, level 7, cli file, 61231085
silesia.tar, level 9, cli file, 60310313
silesia.tar, level 13, cli file, 58517476
silesia.tar, level 16, cli file, 56448694
silesia.tar, level 19, cli file, 53444920
1 Data Config Method Total compressed size
2 silesia.tar level -5 simple 106176430
3 silesia.tar level -3 simple 98476550
4 silesia.tar level -1 simple 87206767
5 silesia.tar level 0 simple 66996953
6 silesia.tar level 1 simple 73658303
7 silesia.tar level 3 simple 66996953
8 silesia.tar level 4 simple 65996020
9 silesia.tar level 5 simple 64421326
10 silesia.tar level 6 simple 62388673
11 silesia.tar level 7 simple 61159525
12 silesia.tar level 9 simple 60214921
13 silesia.tar level 13 simple 58428642
14 silesia.tar level 16 simple 56363759
15 silesia.tar level 19 simple 53274173
16 silesia level -5 cli file 106202112
17 silesia level -3 cli file 98518660
18 silesia level -1 cli file 87226203
19 silesia level 0 cli file 67049190
20 silesia level 1 cli file 73676282
21 silesia level 3 cli file 67049190
22 silesia level 4 cli file 66090040
23 silesia level 5 cli file 64503721
24 silesia level 6 cli file 62446177
25 silesia level 7 cli file 61217029
26 silesia level 9 cli file 60282841
27 silesia level 13 cli file 58480658
28 silesia level 16 cli file 56414170
29 silesia level 19 cli file 53365292
30 silesia.tar level -5 cli file 106250113
31 silesia.tar level -3 cli file 98550747
32 silesia.tar level -1 cli file 87227322
33 silesia.tar level 0 cli file 67111168
34 silesia.tar level 1 cli file 73694374
35 silesia.tar level 3 cli file 67111168
36 silesia.tar level 4 cli file 66154079
37 silesia.tar level 5 cli file 64546998
38 silesia.tar level 6 cli file 62458454
39 silesia.tar level 7 cli file 61231085
40 silesia.tar level 9 cli file 60310313
41 silesia.tar level 13 cli file 58517476
42 silesia.tar level 16 cli file 56448694
43 silesia.tar level 19 cli file 53444920