ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,506 Commits 17 Branches 77 Tags
fe7c645a7e5fd42e9da080efa59e75d63a44f374
Commit Graph
179 Commits
Author SHA1 Message Date
Christoph Grüninger b921f1aad6 Reduce scope of variables 
This improves readability, keeps variables local, and
prevents the unintended use (e.g. typo) later on.
Found by Cppcheck (variableScope)
 2024-02-11 22:00:03 +01:00
W. Felix Handte f746c37d00 Use File Descriptor in Setting Stat on Output File 
Note that the `fd` is only valid while the file is still open. So we need to
move the setting calls to before we close the file. However! We cannot do so
with the `utime()` call (even though `futimens()` exists) because the follow-
ing `close()` call to the `fd` will reset the atime of the file. So it seems
the `utime()` call has to happen after the file is closed.
 2023-02-06 14:02:31 -08:00
W. Felix Handte a5a2418df4 Introduce Variants of Some Functions that Take Optional File Descriptors 
Somewhat surprisingly, calling `fchmod()` is non-trivially faster than calling
`chmod()`, and so on.

This commit introduces alternate variants to some common file util functions
that take an optional fd. If present, they call the `f`-variant of the
underlying function. Otherwise, they fall back to the regular filename-taking
version of the function.
 2023-02-06 13:55:34 -08:00
Yann Collet 8c85b29e32 disable --rm on -o command 
make it more similar to -c (aka `stdout`) convention.
 2023-01-25 16:09:25 -08:00
W. Felix Handte 7a8c8f3fe7 Easy: Print Mode as Octal in chmod() Trace 2023-01-18 11:57:54 -08:00
W. Felix Handte 0d2d460223 Mimic gzip chown(gid), chmod(), chown(uid) Behavior 
Avoids a race condition in which we unintentionally open up permissions to
the wrong group.
 2023-01-18 11:57:54 -08:00
W. Felix Handte 0382076af7 Re-Use stat_t in FIO_compressFilename_srcFile() 2023-01-18 11:33:07 -08:00
Nick Terrell b6e8112261 [util] Add traces enabled by --trace-file-stat 
Print traces to stderr when --trace-file-stat is passed.
We trace all functions that read and write file metadata.
 2022-12-21 17:19:34 -08:00
W. Felix Handte 5d693cc38c Coalesce Almost All Copyright Notices to Standard Phrasing 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora -o -path ./tests/regression/data-cache -o -path ./tests/regression/cache \) -prune -o -type f); do sed -i '/Copyright .* \(Yann Collet\)\|\(Meta Platforms\)/ s/Copyright .*/Copyright (c) Meta Platforms, Inc. and affiliates./' $f; done

git checkout HEAD -- build/VS2010/libzstd-dll/libzstd-dll.rc build/VS2010/zstd/zstd.rc tests/test-license.py contrib/linux-kernel/test/include/linux/xxhash.h examples/streaming_compression_thread_pool.c lib/legacy/zstd_v0*.c lib/legacy/zstd_v0*.h
nano ./programs/windres/zstd.rc
nano ./build/VS2010/zstd/zstd.rc
nano ./build/VS2010/libzstd-dll/libzstd-dll.rc
```
 2022-12-20 12:52:34 -05:00
W. Felix Handte 8927f985ff Update Copyright Headers 'Facebook' -> 'Meta Platforms' 
```
for f in $(find . \( -path ./.git -o -path ./tests/fuzz/corpora \) -prune -o -type f);
do
  sed -i 's/Facebook, Inc\./Meta Platforms, Inc. and affiliates./' $f;
done
```
 2022-12-20 12:37:57 -05:00
Nick TerrellandNick Terrell e58a39f84e Allow tests to fake stdin/stdout/stderr is a console 
We've been unable to effectively test cases where stdin/stdout/stderr
are consoles, because in our test cases they generally aren't. Allow the
command line flags `--fake-std{in,out,err}-is-console` to tell the CLI
to pretend that std{in,out,err} is a console.
 2022-12-14 15:44:09 -08:00
Elliot GorokhovskyandGitHub b1182fe242 Merge pull request #3300 from yiyuaner/dev 
Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable
 2022-11-02 15:28:02 -04:00
appleman4321 5334339421 Rewrite checking condition bufStart pointer NULL 2022-11-01 02:31:09 +03:00
yiyuaner 361d86998a Fix the wrong check for buffer overrun in UTIL_mergeFileNamesTable 2022-10-24 20:37:15 +08:00
Elliot GorokhovskyandGitHub 28ceb63503 Merge pull request #3220 from embg/issue3200 
Disallow empty string as argument for --output-dir-flat and --output-dir-mirror
 2022-08-01 14:04:57 -04:00
Yonatan KomornikandGitHub ae4670466c stdin multiple file fixes (#3222) 
* Fixes for https://github.com/facebook/zstd/issues/3206 - bugs when handling stdin as part of multiple files.

* new line at end of multiple-files.sh
 2022-07-29 16:13:07 -07:00
Qiongsi WuandGitHub b1bbb0eb4c [AIX] Fix Compiler Flags and Bugs on AIX to Pass All Tests (#3219) 
* Fixing compiler warnings

* Replace the old -s flag with the -Wl,-s flag

* Fixing compiler warnings

* Fixing the linker strip flag and tests/code not working as expected on AIX
 2022-07-29 12:21:59 -07:00
Elliot Gorokhovsky e1873ad576 Fix buffer underflow for null dir1 2022-07-29 11:10:47 -07:00
Dominique Pelle b772f53952 Typo and grammar fixes 2022-03-12 08:58:04 +01:00
Xi RuoyaoandGitHub d109cef201 fix the assertion in readLinesFromFile (#3084) 
* fix the assertion in readLinesFromFile

When the file is not terminated by endline, readLineFromFile will append
a '\0' for the last line.  In this case pos + lineLength == dstCapacity.

* test: don't print very long text garbage
 2022-03-04 14:56:44 -05:00
binhdvoandGitHub d7e1736375 Fix build for cygwin/bsd (#2882) 2021-11-29 14:11:39 -05:00
Dimitris Apostolou ebbd675998 Fix typos 2021-11-13 10:04:04 +02:00
Yann Collet eab692211e removed pretty-print of sizes in benchmark 
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
 2021-09-03 12:51:02 -07:00
W. Felix Handte a719edbbc3 Pull utime() Call into Helper 2021-08-04 14:49:00 -04:00
Binh Vo 6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
W. Felix Handte 2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte 9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte 93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte 9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte 1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte eefdbcd93a Make the variable types match 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte b6b23dfe64 Convert names to CamelCase 2021-06-10 12:53:07 -04:00
Scott BakerandW. Felix Handte b70175e5ec Put the human_size() function in util.c 2021-06-10 12:53:07 -04:00
Yann Collet df05b2ba7c fix --filelist compatibility with Windows cr+lf line ending 2021-05-05 18:01:55 -07:00
W. Felix Handte b87f97b3ea Create Files with Desired Permissions; Avoid chmod(); Remove UTIL_chmod() 2021-05-05 13:10:34 -04:00
W. Felix Handte 33f3e293e8 Allow Reading from Block Devices with --force 2021-05-04 16:25:26 -04:00
Nick TerrellandNick Terrell a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files 
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
 2021-03-30 10:30:43 -07:00
Sen Huang f27e326456 Restrict dictmode regression tests only to advanced API, fix some compiler warnings 2021-03-25 10:39:08 -07:00
Paul BoneandPaul Bone 4d6c78fb89 Only set numPhysicalCores if ratio is valid 2021-03-03 10:59:00 +11:00
Paul BoneandPaul Bone eb1a09df61 If cpuinfo parsing fails fallback to sysconf 2021-03-03 10:58:51 +11:00
W. Felix Handte 61db590ad8 Detect .. in Paths Correctly 
This commit addresses #2509.
 2021-02-26 12:29:42 -05:00
W. Felix Handte a774c57973 Use umask() to Constrain Created File Permissions 
This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...
 2021-02-17 15:27:39 -05:00