Commit Graph
2475 Commits
Author SHA1 Message Date
Casey McGinty 2224ecd718 Remove dead code and method name typo 2018-09-11 11:56:50 -07:00
Casey McGinty d4337b6f1d Move ABRThandler func out of internal lib 2018-09-11 11:39:49 -07:00
Casey McGinty 2a3967b7c4 Refactor abort signal handling
- Print signal name to term
- Add -rdynamic option to generate Linux symbol names in backtrace
- Raise default signal after handler to ensure program termination
2018-09-11 10:40:45 -07:00
Casey McGinty a06574fc97 Print a stack trace on unexpected term signal (e.g. SIGABRT)
For OSX and Linux, add a signal handler to SIGABRT, SGIFPE, SIGILL,
SIGSEGV, and SIGBUS. When the program terminates unexpectedly the
handler will print the current stack to the terminal to help determine
the location of the failure.

On OSX the output will look like:

```
Stack trace:
4   zstd                                0x000000010927ed96 main + 16886
5   libdyld.dylib                       0x00007fff767d1015 start + 1
6   ???                                 0x0000000000000001 0x0 + 1
```

On Linux the output will look like:

```
Stack trace:
./zstd() [0x4b8e1b]
./zstd() [0x4b928a]
./zstd() [0x403dc2]
/lib64/libc.so.6(__libc_start_main+0xf5) [0x7f5e0fbb0445]
./zstd() [0x405754]
```

As is, the code does not function on WIN32.

See also: https://oroboro.com/stack-trace-on-crash/
2018-09-06 18:46:52 -07:00
Yann ColletandGitHub 50b216146f Merge pull request #1304 from facebook/largeNbDicts
contrib/largeNbDicts
2018-09-06 09:50:56 -07:00
Jennifer Liu 21721b75a3 Change default f to 20 2018-09-04 17:15:14 -07:00
Jennifer Liu 944c9986e0 Update comment on default steps of cover and fastcover 2018-08-30 15:37:29 -07:00
Jennifer Liu 16db0337b1 Always use splitPoint=1.0 for non-optimize cover and fastcover 2018-08-30 14:59:22 -07:00
Yann ColletandGitHub 31ebb26945 Merge pull request #1301 from terrelln/lit-size
[zstd] Fix seqStore growth
2018-08-28 17:10:25 -07:00
Nick Terrell 5e580de6da [zstd] Fix seqStore growth
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
  space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
2018-08-28 13:24:44 -07:00
Yann ColletandGitHub b37a0a6bde Merge pull request #1298 from facebook/bench
Refactored bench.c
2018-08-28 12:25:02 -07:00
modbwandGitHub d14edf259f Fixed memory leak detected by cppcheck
cppcheck (which is run regularly in our CI environment)  detected a possible memory leak.
2018-08-28 07:25:05 +02:00
Yann Collet 6782725155 first sketch for largeNbDicts test program 2018-08-26 19:29:12 -07:00
Yann ColletandGitHub af23d39eb8 Merge pull request #1297 from felixhandte/check-offset-table
Fix Missing Offset Table Check
2018-08-24 17:36:44 -07:00
W. Felix Handte 37f17ee237 Mark Repeated Offset Table as Needing Check 2018-08-24 14:33:34 -07:00
Nick TerrellandNick Terrell e34e917655 Fix compiler warning 2018-08-23 17:48:06 -07:00
Nick TerrellandNick Terrell 5ee5e71be3 [zstd] Add note about empty ZSTD_CDict 2018-08-23 17:48:06 -07:00
Nick TerrellandNick Terrell 924944e471 [zstd] Reuse the ZSTD_CCtx more often with small data. 2018-08-23 17:48:06 -07:00
Yann Collet 2e45badff4 refactored bench.c
for clarity and safety, especially at interface level
2018-08-23 14:21:18 -07:00
Jennifer LiuandNick Terrell 9d6ed9def3 Merge fastCover into DictBuilder (#1274)
* Minor fix

* Run non-optimize FASTCOVER 5 times in benchmark

* Merge fastCover into dictBuilder

* Fix mixed declaration issue

* Add fastcover to symbol.c

* Add fastCover.c and cover.h to build

* Change fastCover.c to fastcover.c

* Update benchmark to run FASTCOVER in dictBuilder

* Undo spliting fastcover_param into cover_param and f

* Remove convert param functions

* Assign f to parameter

* Add zdict.h to Makefile in lib

* Add cover.h to BUCK

* Cast 1 to U64 before shifting

* Remove trimming of zero freq head and tail in selectSegment and rebenchmark

* Remove f as a separate parameter of tryParam

* Read 8 bytes when d is 6

* Add trimming off zero frequency head and tail

* Use best functions from COVER and remove trimming part(which leads to worse compression ratio after previous bugs were fixed)

* Add finalize= argument to FASTCOVER to specify percentage of training samples passed to ZDICT_finalizeDictionary

* Change nbDmer to always read 8 bytes even when d=6

* Add skip=# argument to allow skipping dmers in computeFrequency in FASTCOVER

* Update comments and benchmarking result

* Change default method of ZDICT_trainFromBuffer to ZDICT_optimizeTrainFromBuffer_fastCover

* Add dictType enum and fix bug about passing zParam when converting to coverParam

* Combine finalize and skip into a single parameter

* Update acceleration parameters and benchmark on 3 sample sets

* Change default splitPoint of FASTCOVER to 0.75 and benchmark first 3 sample sets

* Initialize variables outside of for loop in benchmark.c

* Update benchmark result for hg-manifest

* Remove cover.h from install-includes

* Add explanation of f

* Set default compression level for trainFromBuffer to 3

* Add assertion of fastCoverParams in DiB_trainFromFiles

* Add checkTotalCompressedSize function + some minor fixes

* Add test for multithreading fastCovr

* Initialize segmentFreqs in every FASTCOVER_selectSegment and move mutex_unnlock to end of COVER_best_finish

* Free segmentFreqs

* Initialize segmentFreqs before calling FASTCOVER_buildDictionary instead of in FASTCOVER_selectSegment

* Add FASTCOVER_MEMMULT

* Minor fix

* Update benchmarking result
2018-08-23 12:06:20 -07:00
Yann Collet 36d6165a2d Makefile: added variable SCANBUILD
so that a different version of scan-build can be selected
2018-08-16 16:44:13 -07:00
Yann Collet 1515f0bb0d fixed more issues detected by recent version of scan-build
test run on Linux
2018-08-16 15:20:25 -07:00
Yann Collet 5291d9ac31 fix scope of scan-build tests
exclude zlib code
2018-08-15 17:41:44 -07:00
Yann Collet 42a02ab745 fixed minor warnings issued by scan-build 2018-08-15 14:36:02 -07:00
Yann Collet 3692c31598 Merge branch 'dev' into scanbuild 2018-08-15 13:50:49 -07:00
Yann Collet 6e66bbf5dd fixed several minor issues detected by scan-build
only notable one :
writeNCount() resists better vs invalid distributions
(though it should never happen within zstd anyway)
2018-08-14 16:55:35 -07:00
Yann Collet 79a35ac20d minor code comments improvements 2018-08-09 15:16:31 -07:00
W. Felix Handte 2ca7c69167 Fix CDict Attachment to Handle CDicts with Non-Zero Starts
CDicts were previously guaranteed to be generated with `lowLimit=dictLimit=0`.
This is no longer true, and so the old length and index calculations are no
longer valid. This diff fixes them to handle non-zero start indices in CDicts.
2018-08-07 18:14:14 -07:00
Yann Collet 5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Yann ColletandGitHub 5892dd5da4 Merge pull request #1255 from terrelln/norm-fix
[FSE] Fix division by zero
2018-08-02 11:48:56 -07:00
Nick Terrell dc5a67cb7b Disallow tableLog == srcLog 2018-08-02 11:12:17 -07:00
Jennifer Liu f5228f2c44 Refactoring 2018-07-31 13:58:54 -07:00
Jennifer Liu 4e29bc2469 Use CDict instead of CCtx in analyzeEntropy 2018-07-31 10:36:45 -07:00
cyan4973 3f535007e4 fix %zu support under minGW
and relevant test on Appveyor
2018-07-30 16:56:18 +02:00
cyan4973 aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
Nick Terrell 9889bca530 [FSE] Fix division by zero
When the primary normalization method fails, and
`(1 << tableLog) == (maxSymbolValue + 1)`, and every symbol gets assigned
normalized weight 1 or -1 in the first loop, then the next division can
raise `SIGFPE`.
2018-07-27 17:30:03 -07:00
Yann ColletandGitHub 6e490a2f09 Merge pull request #1237 from terrelln/init-cstream-adv
Set requestedParams in ZSTD_initCStream*()
2018-07-18 16:33:30 +02:00
cyan4973 9597b438e9 fix #1241
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
2018-07-17 18:52:57 +02:00
cyan4973 53e1f0504e zstdmt debug traces compatibles with mingw
since mingw does not have `sys/times.h`,
remove this path when detecting mingw compilation.
2018-07-17 14:39:44 +02:00
Nick TerrellandGitHub 45821fac0c Merge pull request #1225 from jennifermliu/dev
Split samples when building dictionary for COVER
2018-07-13 13:26:15 -07:00
Nick Terrell 6d222c437c Set requestedParams in ZSTD_initCStream*()
The correct parameters are used once, but once `ZSTD_resetCStream()` is
called the default parameters (level 3) are used. Fix this by setting
`requestedParams` in the `ZSTD_initCStream*()` functions.

The added tests both fail before this patch and pass after.
2018-07-12 18:35:55 -07:00
Jennifer Liu 612b346ed5 Add explanation for split=100 2018-07-11 15:50:28 -07:00
Jennifer Liu 5021441d86 Change default splitPoint to 100 2018-07-10 11:19:33 -07:00
Jennifer Liu 456f290e31 Change back to splitPoint<=0 2018-07-09 13:53:25 -07:00
Jennifer Liu 7efabb2cf6 Only make 0.0 default splitPoint 2018-07-09 12:26:53 -07:00
Yann Collet bbd78df59b add build macro NO_PREFETCH
prevent usage of prefetch intrinsic commands
which are not supported by c2rust
(see https://github.com/immunant/c2rust/issues/13)
2018-07-06 17:06:04 -07:00
Jennifer Liu 015a00af0f Change cover_sum back to 2 parameters and fix splitPoint issues 2018-07-06 14:24:18 -07:00
Jennifer Liu 0bbff01211 Fix testing parameter 2018-07-05 22:40:32 -07:00
Jennifer Liu a085d1aae1 Allow splitPoint==1.0 (using all samples for both training and testing) 2018-07-05 10:38:45 -07:00
Jennifer Liu 0881184c89 Some edits based on pull request comments 2018-07-03 17:53:27 -07:00