Commit Graph
194 Commits
Author SHA1 Message Date
W. Felix Handte 5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
2022-12-20 12:52:34 -05:00
W. Felix Handte 8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms'
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
2022-12-20 12:37:57 -05:00
Yann Collet 361b7bd1d0 Merge branch 'dev' into clean 2022-09-12 10:56:00 -07:00
Yann Collet c0b46738b4 streamline make clean list maintenance
When creating a new `Makefile` target to build,
it's also necessary to update the `clean` target,
which purpose is to remove built targets when they are present.

This process is simple, but it's also easy to forget :
since there is a large distance between the position in the `Makefile` where the new built target is added,
and the place where the list of files to `clean` is defined.
Moreover, the list of files becomes pretty long over time,
hence it's difficult to visually ensure that all built targets are present there,
or that no old target (no longer produced) is no longer in the list

This PR tries to improve this process by adding a CLEAN variable.
Now, when a new built target is added to the `Makefile`,
it should preceded by :
```
CLEAN += newTarget
newTarget:
<TAB> ...recipe...
```

This new requirement is somewhat similar to `.PHONY: newTarget` for non-built targets.

This new method offers the advantage of locality :
there is no separate place in the file to maintain a list of files to clean.
This makes maintenance of `make clean` easier.
2022-09-07 16:36:03 -07:00
Danielle Rozenblit bc7492cefa added zstreamtest_ubsan to make file + added ubsan zstreamtest job for CI tests 2022-09-07 11:54:17 -07:00
Danielle Rozenblit 69022ad886 null decompress buffer test and ubsan flag added 2022-09-06 14:34:55 -07:00
Nick Terrell 1fc42de86a [CI] Hook cli-tests up to CI
Add cli-tests to `make test`. This adds a `python3` dependency to `make
test`, but not `make check`. We could make this dependency optional by
skipping the tests if `python3` is not present.
2022-01-27 13:56:59 -08:00
Nick Terrell 8ea3d57de4 [build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`

Fixes #3006.
2022-01-18 15:11:29 -08:00
Alexander KanavinandAlexander Kanavin 1e514feec6 Makefile: sort all wildcard file list expansions
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
Yann ColletandGitHub 7868f38019 Merge pull request #2747 from Helflym/dev
Add AIX support in Makefile
2021-10-01 08:13:39 -07:00
Nick Terrell 8bf699aa59 [build] Add support for ASM files in Make + CMake
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
  Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
  same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
  `make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.

The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.

The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
2021-09-17 14:13:53 -07:00
Clément Chigot 399849e236 Makefile: add AIX support
For lib, AIX linker doesn't allow --soname.
2021-08-13 10:25:14 +02:00
senandGitHub 698f261b35 [1.5.0] Deprecate some functions (#2582)
* Add deprecated macro to zstd.h, mark certain functions as deprecated

* Remove ZSTD_compress.c dependencies on deprecated functions
2021-05-06 17:59:32 -04:00
Nick Terrell eb7e74ccb7 [tests] Set DEBUGLEVEL=2 by default
This allows us to quickly check for compile errors in debug log
messages, which are compiled out when `DEBUGLEVEL < 2`.
2021-05-05 13:29:06 -07:00
Nick Terrell 34aff7ea06 Bug fix & run overflow correction much more frequently in tests
* Fix overflow correction when `windowLog < cycleLog`. Previously, we
  got the correction wrong in this case, and our chain tables and binary
  trees would be corrupted. Now, we work as long as `maxDist` is a power
  of two, by adding `MAX(maxDist, cycleSize)` to our indices.
* When `ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY` is defined to non-zero
  run overflow correction as frequently as allowed without impacting
  compression ratio.
* Enable `ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY` in `fuzzer` and
  `zstreamtest` as well as all the OSS-Fuzz fuzzers. This has a 5-10%
  speed penalty at most, which seems reasonable.
2021-05-03 15:21:47 -07:00
senhuang42 a423305e7b Remove ZBUFF tests 2021-04-19 17:27:05 -04:00
Nick TerrellandNick Terrell a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
Nick Terrell 66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
Yann Collet 9648bf027b try to keep libzstd.a "as is" once created
to be compatible with scenarios such as
`make -j allmost`
2020-12-20 17:10:57 -08:00
Yann Collet 5de5c1d759 fixed fuzzer multithreading tests 2020-12-02 10:34:12 -08:00
Yann Collet db21d383b5 fixed fuzzer32 to support multithreading tests
though it still fails on test33:
`test 33: superblock uncompressible data, too many nocompress superblocks`
2020-12-02 09:13:55 -08:00
Yann Collet f69d8c027d removed fullbench-lib from tests/all
this build works fine on all my systems,
but since to fail on CI environment.
Unclear why there is a difference.
This build test is not relevant anyway.
2020-12-02 00:21:29 -08:00
Yann Collet 9f8b180d5d fixed API documentation 2020-12-02 00:15:07 -08:00
Yann Collet f8d0b46a9f streamline fuzzer
from fuzzer32
2020-12-01 23:44:16 -08:00
Yann Collet 37165f66b7 better usage of default build rules 2020-12-01 23:36:05 -08:00
Yann Collet 343a75d2ef simplified test makefile
removed gzstd target:
relevant tests are unused and broken anyway
2020-12-01 22:33:45 -08:00
Nick TerrellandGitHub 599ff58e08 Merge pull request #2339 from terrelln/zstdmt-stability
Fix zstdmt stability issues and clean up the zstdmt code
2020-10-27 19:43:13 -07:00
Yann Collet 91a8cb9559 fix DEBUGLEVEL redefinition from tests/ 2020-10-22 00:20:40 -07:00
Yann Collet 494f7169ed fix directory creation for Windows' libzstd 2020-10-22 00:15:31 -07:00
Yann Collet ca75da8fa3 fix test
DEBUGLEVEL redefinition
2020-10-21 23:51:13 -07:00
Nick Terrell c51a9e79b9 [zstdmt] Rip out the zstdmt API
This commit leaves only the functions used by zstd_compress.c. All other
functions have been removed from the API. The ZSTDMT unit tests in
fuzzer.c and zstreamtest.c have been rewritten to use the ZSTD API. And
the --mt zstreamtest tests have been ripped out.
2020-10-12 12:55:16 -07:00
Yann Collet 7ea2ae6649 added test linking user program to multi-threaded libzstd 2020-04-28 21:18:29 -07:00
Nick Terrell ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Bimba ShresthaandGitHub 6a4258a08a Removing symbols already in unit tests and adding some new unit tests for missing symbols (#1985)
* Removing symbols that are not being tested

* Removing symbols used in zstdcli, fileio, dibio and benchzstd

* Removing symbols used in zbuff and add test-zbuff to travis

* Removing remaining symbols and adding unit tests instead

* Removing symbols test entirely
2020-02-05 16:55:00 -08:00
Bimba ShresthaandYann Collet 8fe562a770 [automated_benchmarking] Make arguments optional and add --dict argument (#1968)
* Make arugments optional and add --dict argument

* Removing accidental print statement

* Change to more likely scenario for dictionary compression benchmark
2020-01-28 11:29:43 -08:00
Nick TerrellandNick Terrell 3ed0f65158 [cmake] Add playTests.sh as a test 2020-01-13 14:16:15 -08:00
Bimba ShresthaandYann Collet eb76f786bc [bench] Automated benchmarking script (#1906)
* Initial revised automated benchmarking script

* Updating nb_iterations and making loop infinite

* Allowing benchmarking params to be changed from cli

* Renaming old speed test

* Removing numpy dependency for cli

* Change filename and benchmakr on pr level

* Moving build outside loop and adding iterations param

* Moving benchmarking to seperate travis ci test

* Fixing typo and using unused variable

* Added mode labels and updated README accordingly

* Adding new mode 'current' that compraes facebook:dev against current hash

* Typo

* Reverting previous accidental diff

* Typo

* Adding frequency config variable to prevent github from blacklisting

* Added new argument for frequency of fetching new prs

* Updating documentation
2020-01-06 14:19:11 -08:00
Felix HandteandYann Collet 6f4341c432 Fix playTests.sh Under QEMU (#1923) 2019-12-26 11:16:23 -08:00
Yann Collet 63e435dda1 improve deprecation warning macro
fix #1488

although, curiously enough, I was never able to reproduce the issue
(according to the bug report, it should be present while using gcc 4.8).
2019-10-23 11:59:32 -07:00
Yann Collet 2ad75bb574 validated 'make test' for mingw environment 2019-10-18 15:37:07 -07:00
Yann Collet ad86a5d0bc rewrite FIO_createFilename_fromOutDir() 2019-10-18 11:15:10 -07:00
Nick Terrell 8e16b872e5 [tests] Add tests for big dictionaries 2019-06-21 17:58:24 -07:00
W. Felix Handte b878bd08dd Clean Up Temp Files Produced By playTests.sh 2019-06-07 15:32:28 -04:00
W. Felix Handte 6a0638048a Add Test 2019-05-24 17:21:44 -04:00
Yann Collet 8f56fa2f58 fixed poolTests
added poolTests to all
2019-04-11 09:50:39 -07:00
Yann Collet 59a7116cc2 benchfn dependencies reduced to only timefn
benchfn used to rely on mem.h, and util,
which in turn relied on platform.h.
Using benchfn outside of zstd required to bring all these dependencies.

Now, dependency is reduced to timefn only.
This required to create a separate timefn from util,
and rewrite benchfn and timefn to no longer need mem.h.

Separating timefn from util has a wide effect accross the code base,
as usage of time functions is widespread.
A lot of build scripts had to be updated to also include timefn.
2019-04-10 12:37:03 -07:00
Peter (Stig) EdwardsandGitHub 723418f8d8 -Wformat-security not needed with -Wformat=2 2019-02-01 09:29:40 +00:00
Lzu Tao 7b6a8840c5 Fix #1428 - zstdgrep returned 1 on match and unmatch
- Use ZCAT for testing zstdgrep in case of non-install yet
- tests: Add file test for zstdgrep
2019-01-06 23:22:55 +07:00
Lzu Tao d095adf9fb Add simple test for zstdgrep 2018-11-29 03:39:47 +07:00
Yann Collet d38063f8ae separated bench module into benchfn and benchzstd
it shall be possible to use benchfn
without any dependency on zstd.
2018-11-13 11:01:59 -08:00