ddidderr/zstd-rs
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #827 from facebook/dev

Browse Source 
Update branch
This commit is contained in:
Yann Collet
2017-08-31 18:10:31 -07:00
committed by GitHub
parent d01cd4ea47 d7ad99b2ab
commit 9ab7614d23
199 changed files with 6481 additions and 1730 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CONTRIBUTING.md
+1 -1
View File
@@ -39,4 +39,4 @@ outlined on that page and do not file a public issue.
## License
By contributing to Zstandard, you agree that your contributions will be licensed
under the [LICENSE](LICENSE) file in the root directory of this source tree.
under both the [LICENSE](LICENSE) file and the [COPYING](COPYING) file in the root directory of this source tree.
COPYING
+339
View File
@@ -0,0 +1,339 @@
GNU GENERAL PUBLIC LICENSE
Version 2, June 1991
Copyright (C) 1989, 1991 Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The licenses for most software are designed to take away your
freedom to share and change it. By contrast, the GNU General Public
License is intended to guarantee your freedom to share and change free
software--to make sure the software is free for all its users. This
General Public License applies to most of the Free Software
Foundation's software and to any other program whose authors commit to
using it. (Some other Free Software Foundation software is covered by
the GNU Lesser General Public License instead.) You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
this service if you wish), that you receive source code or can get it
if you want it, that you can change the software or use pieces of it
in new free programs; and that you know you can do these things.
To protect your rights, we need to make restrictions that forbid
anyone to deny you these rights or to ask you to surrender the rights.
These restrictions translate to certain responsibilities for you if you
distribute copies of the software, or if you modify it.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must give the recipients all the rights that
you have. You must make sure that they, too, receive or can get the
source code. And you must show them these terms so they know their
rights.
We protect your rights with two steps: (1) copyright the software, and
(2) offer you this license which gives you legal permission to copy,
distribute and/or modify the software.
Also, for each author's protection and ours, we want to make certain
that everyone understands that there is no warranty for this free
software. If the software is modified by someone else and passed on, we
want its recipients to know that what they have is not the original, so
that any problems introduced by others will not reflect on the original
authors' reputations.
Finally, any free program is threatened constantly by software
patents. We wish to avoid the danger that redistributors of a free
program will individually obtain patent licenses, in effect making the
program proprietary. To prevent this, we have made it clear that any
patent must be licensed for everyone's free use or not licensed at all.
The precise terms and conditions for copying, distribution and
modification follow.
GNU GENERAL PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. This License applies to any program or other work which contains
a notice placed by the copyright holder saying it may be distributed
under the terms of this General Public License. The "Program", below,
refers to any such program or work, and a "work based on the Program"
means either the Program or any derivative work under copyright law:
that is to say, a work containing the Program or a portion of it,
either verbatim or with modifications and/or translated into another
language. (Hereinafter, translation is included without limitation in
the term "modification".) Each licensee is addressed as "you".
Activities other than copying, distribution and modification are not
covered by this License; they are outside its scope. The act of
running the Program is not restricted, and the output from the Program
is covered only if its contents constitute a work based on the
Program (independent of having been made by running the Program).
Whether that is true depends on what the Program does.
1. You may copy and distribute verbatim copies of the Program's
source code as you receive it, in any medium, provided that you
conspicuously and appropriately publish on each copy an appropriate
copyright notice and disclaimer of warranty; keep intact all the
notices that refer to this License and to the absence of any warranty;
and give any other recipients of the Program a copy of this License
along with the Program.
You may charge a fee for the physical act of transferring a copy, and
you may at your option offer warranty protection in exchange for a fee.
2. You may modify your copy or copies of the Program or any portion
of it, thus forming a work based on the Program, and copy and
distribute such modifications or work under the terms of Section 1
above, provided that you also meet all of these conditions:
a) You must cause the modified files to carry prominent notices
stating that you changed the files and the date of any change.
b) You must cause any work that you distribute or publish, that in
whole or in part contains or is derived from the Program or any
part thereof, to be licensed as a whole at no charge to all third
parties under the terms of this License.
c) If the modified program normally reads commands interactively
when run, you must cause it, when started running for such
interactive use in the most ordinary way, to print or display an
announcement including an appropriate copyright notice and a
notice that there is no warranty (or else, saying that you provide
a warranty) and that users may redistribute the program under
these conditions, and telling the user how to view a copy of this
License. (Exception: if the Program itself is interactive but
does not normally print such an announcement, your work based on
the Program is not required to print an announcement.)
These requirements apply to the modified work as a whole. If
identifiable sections of that work are not derived from the Program,
and can be reasonably considered independent and separate works in
themselves, then this License, and its terms, do not apply to those
sections when you distribute them as separate works. But when you
distribute the same sections as part of a whole which is a work based
on the Program, the distribution of the whole must be on the terms of
this License, whose permissions for other licensees extend to the
entire whole, and thus to each and every part regardless of who wrote it.
Thus, it is not the intent of this section to claim rights or contest
your rights to work written entirely by you; rather, the intent is to
exercise the right to control the distribution of derivative or
collective works based on the Program.
In addition, mere aggregation of another work not based on the Program
with the Program (or with a work based on the Program) on a volume of
a storage or distribution medium does not bring the other work under
the scope of this License.
3. You may copy and distribute the Program (or a work based on it,
under Section 2) in object code or executable form under the terms of
Sections 1 and 2 above provided that you also do one of the following:
a) Accompany it with the complete corresponding machine-readable
source code, which must be distributed under the terms of Sections
1 and 2 above on a medium customarily used for software interchange; or,
b) Accompany it with a written offer, valid for at least three
years, to give any third party, for a charge no more than your
cost of physically performing source distribution, a complete
machine-readable copy of the corresponding source code, to be
distributed under the terms of Sections 1 and 2 above on a medium
customarily used for software interchange; or,
c) Accompany it with the information you received as to the offer
to distribute corresponding source code. (This alternative is
allowed only for noncommercial distribution and only if you
received the program in object code or executable form with such
an offer, in accord with Subsection b above.)
The source code for a work means the preferred form of the work for
making modifications to it. For an executable work, complete source
code means all the source code for all modules it contains, plus any
associated interface definition files, plus the scripts used to
control compilation and installation of the executable. However, as a
special exception, the source code distributed need not include
anything that is normally distributed (in either source or binary
form) with the major components (compiler, kernel, and so on) of the
operating system on which the executable runs, unless that component
itself accompanies the executable.
If distribution of executable or object code is made by offering
access to copy from a designated place, then offering equivalent
access to copy the source code from the same place counts as
distribution of the source code, even though third parties are not
compelled to copy the source along with the object code.
4. You may not copy, modify, sublicense, or distribute the Program
except as expressly provided under this License. Any attempt
otherwise to copy, modify, sublicense or distribute the Program is
void, and will automatically terminate your rights under this License.
However, parties who have received copies, or rights, from you under
this License will not have their licenses terminated so long as such
parties remain in full compliance.
5. You are not required to accept this License, since you have not
signed it. However, nothing else grants you permission to modify or
distribute the Program or its derivative works. These actions are
prohibited by law if you do not accept this License. Therefore, by
modifying or distributing the Program (or any work based on the
Program), you indicate your acceptance of this License to do so, and
all its terms and conditions for copying, distributing or modifying
the Program or works based on it.
6. Each time you redistribute the Program (or any work based on the
Program), the recipient automatically receives a license from the
original licensor to copy, distribute or modify the Program subject to
these terms and conditions. You may not impose any further
restrictions on the recipients' exercise of the rights granted herein.
You are not responsible for enforcing compliance by third parties to
this License.
7. If, as a consequence of a court judgment or allegation of patent
infringement or for any other reason (not limited to patent issues),
conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot
distribute so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you
may not distribute the Program at all. For example, if a patent
license would not permit royalty-free redistribution of the Program by
all those who receive copies directly or indirectly through you, then
the only way you could satisfy both it and this License would be to
refrain entirely from distribution of the Program.
If any portion of this section is held invalid or unenforceable under
any particular circumstance, the balance of the section is intended to
apply and the section as a whole is intended to apply in other
circumstances.
It is not the purpose of this section to induce you to infringe any
patents or other property right claims or to contest validity of any
such claims; this section has the sole purpose of protecting the
integrity of the free software distribution system, which is
implemented by public license practices. Many people have made
generous contributions to the wide range of software distributed
through that system in reliance on consistent application of that
system; it is up to the author/donor to decide if he or she is willing
to distribute software through any other system and a licensee cannot
impose that choice.
This section is intended to make thoroughly clear what is believed to
be a consequence of the rest of this License.
8. If the distribution and/or use of the Program is restricted in
certain countries either by patents or by copyrighted interfaces, the
original copyright holder who places the Program under this License
may add an explicit geographical distribution limitation excluding
those countries, so that distribution is permitted only in or among
countries not thus excluded. In such case, this License incorporates
the limitation as if written in the body of this License.
9. The Free Software Foundation may publish revised and/or new versions
of the General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the Program
specifies a version number of this License which applies to it and "any
later version", you have the option of following the terms and conditions
either of that version or of any later version published by the Free
Software Foundation. If the Program does not specify a version number of
this License, you may choose any version ever published by the Free Software
Foundation.
10. If you wish to incorporate parts of the Program into other free
programs whose distribution conditions are different, write to the author
to ask for permission. For software which is copyrighted by the Free
Software Foundation, write to the Free Software Foundation; we sometimes
make exceptions for this. Our decision will be guided by the two goals
of preserving the free status of all derivatives of our free software and
of promoting the sharing and reuse of software generally.
NO WARRANTY
11. BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY
FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN
OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES
PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED
OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS
TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU. SHOULD THE
PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING,
REPAIR OR CORRECTION.
12. IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR
REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES,
INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING
OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED
TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY
YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER
PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE
POSSIBILITY OF SUCH DAMAGES.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
convey the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Also add information on how to contact you by electronic and paper mail.
If the program is interactive, make it output a short notice like this
when it starts in an interactive mode:
Gnomovision version 69, Copyright (C) year name of author
Gnomovision comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, the commands you use may
be called something other than `show w' and `show c'; they could even be
mouse-clicks or menu items--whatever suits your program.
You should also get your employer (if you work as a programmer) or your
school, if any, to sign a "copyright disclaimer" for the program, if
necessary. Here is a sample; alter the names:
Yoyodyne, Inc., hereby disclaims all copyright interest in the program
`Gnomovision' (which makes passes at compilers) written by James Hacker.
<signature of Ty Coon>, 1 April 1989
Ty Coon, President of Vice
This General Public License does not permit incorporating your program into
proprietary programs. If your program is a subroutine library, you may
consider it more useful to permit linking proprietary applications with the
library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License.
LICENSE-examples
-11
View File
@@ -1,11 +0,0 @@
Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
The examples provided by Facebook are for non-commercial testing and evaluation
purposes only. Facebook reserves all rights not expressly granted.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
FACEBOOK BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Makefile
+19 -20
View File
@@ -1,10 +1,10 @@
# ################################################################
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PRGDIR = programs
@@ -29,15 +29,12 @@ default: lib-release zstd-release
all: | allmost examples manual
.PHONY: allmost
allmost:
$(MAKE) -C $(ZSTDDIR) all
$(MAKE) -C $(PRGDIR) all
$(MAKE) -C $(TESTDIR) all
allmost: allzstd
$(MAKE) -C $(ZWRAPDIR) all
#skip zwrapper, can't build that on alternate architectures without the proper zlib installed
.PHONY: allarch
allarch:
.PHONY: allzstd
allzstd:
$(MAKE) -C $(ZSTDDIR) all
$(MAKE) -C $(PRGDIR) all
$(MAKE) -C $(TESTDIR) all
@@ -74,12 +71,9 @@ zstdmt:
zlibwrapper:
$(MAKE) -C $(ZWRAPDIR) test
.PHONY: shortest
shortest:
$(MAKE) -C $(TESTDIR) $@
.PHONY: test
test:
.PHONY: test shortest
test shortest:
$(MAKE) -C $(PRGDIR) allVariants
$(MAKE) -C $(TESTDIR) $@
.PHONY: examples
@@ -146,6 +140,11 @@ gcc6build: clean
gcc-6 -v
CC=gcc-6 $(MAKE) all MOREFLAGS="-Werror"
.PHONY: gcc7build
gcc7build: clean
gcc-7 -v
CC=gcc-7 $(MAKE) all MOREFLAGS="-Werror"
.PHONY: clangbuild
clangbuild: clean
clang -v
@@ -156,16 +155,16 @@ m32build: clean
$(MAKE) all32
armbuild: clean
CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allarch
CC=arm-linux-gnueabi-gcc CFLAGS="-Werror" $(MAKE) allzstd
aarch64build: clean
CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allarch
CC=aarch64-linux-gnu-gcc CFLAGS="-Werror" $(MAKE) allzstd
ppcbuild: clean
CC=powerpc-linux-gnu-gcc CLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allarch
CC=powerpc-linux-gnu-gcc CLAGS="-m32 -Wno-attributes -Werror" $(MAKE) allzstd
ppc64build: clean
CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allarch
CC=powerpc-linux-gnu-gcc CFLAGS="-m64 -Werror" $(MAKE) allzstd
armfuzz: clean
CC=arm-linux-gnueabi-gcc QEMU_SYS=qemu-arm-static MOREFLAGS="-static" FUZZER_FLAGS=--no-big-tests $(MAKE) -C $(TESTDIR) fuzztest
NEWS
+16 -2
View File
@@ -1,9 +1,23 @@
v1.3.2
license : changed /examples license to BSD + GPLv2
license : fix a few header files to reflect new license (#825)
fix : a rare compression bug when compression generates very large distances (only possible at --ultra -22)
build: better compatibility with reproducible builds, by Bernhard M. Wiedemann (@bmwiedemann) (#818)
v1.3.1
perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt
New license : BSD + GPLv2
perf: substantially decreased memory usage in Multi-threading mode, thanks to reports by Tino Reichardt (@mcmilk)
perf: Multi-threading supports up to 256 threads. Cap at 256 when more are requested (#760)
build: fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (#718)
cli : improved and fixed --list command, by @ib (#772)
cli : command -vV to list supported formats, by @ib (#771)
build : fixed binary variants, reported by @svenha (#788)
build : fix Visual compilation for non x86/x64 targets, reported by Greg Slazinski (@GregSlazinski) (#718)
API exp : breaking change : ZSTD_getframeHeader() provides more information
API exp : breaking change : pinned down values of error codes
doc : fixed huffman example, by Ulrich Kunitz (@ulikunitz)
new : contrib/adaptive-compression, I/O driven compression strength, by Paul Cruz (@paulcruz74)
new : contrib/long_distance_matching, statistics by Stella Lau (@stellamplau)
updated : contrib/linux-kernel, by Nick Terrell (@terrelln)
v1.3.0
cli : new : `--list` command, by Paul Cruz
PATENTS
-33
View File
@@ -1,33 +0,0 @@
Additional Grant of Patent Rights Version 2
"Software" means the Zstandard software distributed by Facebook, Inc.
Facebook, Inc. ("Facebook") hereby grants to each recipient of the Software
("you") a perpetual, worldwide, royalty-free, non-exclusive, irrevocable
(subject to the termination provision below) license under any Necessary
Claims, to make, have made, use, sell, offer to sell, import, and otherwise
transfer the Software. For avoidance of doubt, no license is granted under
Facebook’s rights in any patent claims that are infringed by (i) modifications
to the Software made by you or any third party or (ii) the Software in
combination with any software or other technology.
The license granted hereunder will terminate, automatically and without notice,
if you (or any of your subsidiaries, corporate affiliates or agents) initiate
directly or indirectly, or take a direct financial interest in, any Patent
Assertion: (i) against Facebook or any of its subsidiaries or corporate
affiliates, (ii) against any party if such Patent Assertion arises in whole or
in part from any software, technology, product or service of Facebook or any of
its subsidiaries or corporate affiliates, or (iii) against any party relating
to the Software. Notwithstanding the foregoing, if Facebook or any of its
subsidiaries or corporate affiliates files a lawsuit alleging patent
infringement against you in the first instance, and you respond by filing a
patent infringement counterclaim in that lawsuit against that party that is
unrelated to the Software, the license granted hereunder will not terminate
under section (i) of this paragraph due to such counterclaim.
A "Necessary Claim" is a claim of a patent owned by Facebook that is
necessarily infringed by the Software standing alone.
A "Patent Assertion" is any lawsuit or other action alleging direct, indirect,
or contributory infringement or inducement to infringe any patent, including a
cross-claim or counterclaim.
README.md
+2 -2
View File
@@ -134,12 +134,12 @@ Going into `build` directory, you will find additional possibilities :
### Status
Zstandard is currently deployed within Facebook. It is used daily to compress and decompress very large amounts of data in multiple formats and use cases.
Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases.
Zstandard is considered safe for production environments.
### License
Zstandard is [BSD-licensed](LICENSE). We also provide an [additional patent grant](PATENTS).
Zstandard is dual-licensed under [BSD](LICENSE) and [GPLv2](COPYING).
### Contributing
appveyor.yml
+6 -6
View File
@@ -9,19 +9,19 @@
- COMPILER: "gcc"
HOST: "mingw"
PLATFORM: "x64"
SCRIPT: "make allarch && make -C tests test-symbols fullbench-dll fullbench-lib"
SCRIPT: "make allzstd MOREFLAGS=-static && make -C tests test-symbols fullbench-dll fullbench-lib"
ARTIFACT: "true"
BUILD: "true"
- COMPILER: "gcc"
HOST: "mingw"
PLATFORM: "x86"
SCRIPT: "make allarch"
SCRIPT: "make allzstd MOREFLAGS=-static"
ARTIFACT: "true"
BUILD: "true"
- COMPILER: "clang"
HOST: "mingw"
PLATFORM: "x64"
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allarch"
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allzstd"
BUILD: "true"
- COMPILER: "gcc"
@@ -172,15 +172,15 @@
- COMPILER: "gcc"
HOST: "mingw"
PLATFORM: "x64"
SCRIPT: "make allarch"
SCRIPT: "make allzstd"
- COMPILER: "gcc"
HOST: "mingw"
PLATFORM: "x86"
SCRIPT: "make allarch"
SCRIPT: "make allzstd"
- COMPILER: "clang"
HOST: "mingw"
PLATFORM: "x64"
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allarch"
SCRIPT: "MOREFLAGS='--target=x86_64-w64-mingw32 -Werror -Wconversion -Wno-sign-conversion' make allzstd"
- COMPILER: "visual"
HOST: "visual"
build/cmake/CMakeLists.txt
+3 -3
View File
@@ -2,9 +2,9 @@
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PROJECT(zstd)
build/cmake/contrib/CMakeLists.txt
+5 -9
View File
@@ -1,17 +1,13 @@
# ################################################################
# * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# * All rights reserved.
# *
# * This source code is licensed under the BSD-style license found in the
# * LICENSE file in the root directory of this source tree. An additional grant
# * of patent rights can be found in the PATENTS file in the same directory.
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# You can contact the author at :
# - zstd homepage : http://www.zstd.net/
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PROJECT(contrib)
ADD_SUBDIRECTORY(pzstd)
ADD_SUBDIRECTORY(gen_html)
build/cmake/contrib/gen_html/CMakeLists.txt
+5 -8
View File
@@ -1,13 +1,10 @@
# ################################################################
# * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# * All rights reserved.
# *
# * This source code is licensed under the BSD-style license found in the
# * LICENSE file in the root directory of this source tree. An additional grant
# * of patent rights can be found in the PATENTS file in the same directory.
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# You can contact the author at :
# - zstd homepage : http://www.zstd.net/
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PROJECT(gen_html)
build/cmake/contrib/pzstd/CMakeLists.txt
+5 -8
View File
@@ -1,13 +1,10 @@
# ################################################################
# * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# * All rights reserved.
# *
# * This source code is licensed under the BSD-style license found in the
# * LICENSE file in the root directory of this source tree. An additional grant
# * of patent rights can be found in the PATENTS file in the same directory.
# Copyright (c) 2016-present, Facebook, Inc.
# All rights reserved.
#
# You can contact the author at :
# - zstd homepage : http://www.zstd.net/
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PROJECT(pzstd)
build/cmake/lib/CMakeLists.txt
+9 -12
View File
@@ -1,13 +1,10 @@
# ################################################################
# * Copyright (c) 2014-present, Yann Collet, Facebook, Inc.
# * All rights reserved.
# *
# * This source code is licensed under the BSD-style license found in the
# * LICENSE file in the root directory of this source tree. An additional grant
# * of patent rights can be found in the PATENTS file in the same directory.
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# You can contact the author at :
# - zstd homepage : http://www.zstd.net/
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PROJECT(libzstd)
@@ -151,10 +148,10 @@ IF (UNIX)
ENDIF (UNIX)
# install target
INSTALL(FILES
${LIBRARY_DIR}/zstd.h
${LIBRARY_DIR}/deprecated/zbuff.h
${LIBRARY_DIR}/dictBuilder/zdict.h
INSTALL(FILES
${LIBRARY_DIR}/zstd.h
${LIBRARY_DIR}/deprecated/zbuff.h
${LIBRARY_DIR}/dictBuilder/zdict.h
${LIBRARY_DIR}/common/zstd_errors.h
DESTINATION "include")
build/cmake/programs/CMakeLists.txt
+5 -8
View File
@@ -1,13 +1,10 @@
# ################################################################
# * Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# * All rights reserved.
# *
# * This source code is licensed under the BSD-style license found in the
# * LICENSE file in the root directory of this source tree. An additional grant
# * of patent rights can be found in the PATENTS file in the same directory.
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# You can contact the author at :
# - zstd homepage : http://www.zstd.net/
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PROJECT(programs)
circle.yml
+3 -3
View File
@@ -3,7 +3,7 @@ dependencies:
- sudo dpkg --add-architecture i386
- sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test; sudo apt-get -y -qq update
- sudo apt-get -y install gcc-powerpc-linux-gnu gcc-arm-linux-gnueabi libc6-dev-armel-cross gcc-aarch64-linux-gnu libc6-dev-arm64-cross
- sudo apt-get -y install libstdc++-6-dev clang gcc g++ gcc-5 gcc-6 zlib1g-dev liblzma-dev
- sudo apt-get -y install libstdc++-7-dev clang gcc g++ gcc-5 gcc-6 gcc-7 zlib1g-dev liblzma-dev
- sudo apt-get -y install linux-libc-dev:i386 libc6-dev-i386
test:
@@ -45,7 +45,7 @@ test:
parallel: true
- ? |
if [[ "$CIRCLE_NODE_INDEX" == "0" ]] ; then make ppc64build && make clean; fi &&
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then true && make clean; fi #could add another test here
if [[ "$CIRCLE_NODE_TOTAL" < "2" ]] || [[ "$CIRCLE_NODE_INDEX" == "1" ]]; then make gcc7build && make clean; fi #could add another test here
:
parallel: true
- ? |
@@ -64,7 +64,7 @@ test:
#- gcc -v; make -C tests test32 MOREFLAGS="-I/usr/include/x86_64-linux-gnu" && make clean
#- make uasan && make clean
#- make asan32 && make clean
#- make -C tests test32 CC=clang MOREFLAGS="-g -fsanitize=address -I/usr/include/x86_64-linux-gnu"
#- make -C tests test32 CC=clang MOREFLAGS="-g -fsanitize=address -I/usr/include/x86_64-linux-gnu"
# Valgrind tests
#- CFLAGS="-O1 -g" make -C zlibWrapper valgrindTest && make clean
#- make -C tests valgrindTest && make clean
contrib/adaptive-compression/Makefile
+76
View File
@@ -0,0 +1,76 @@
ZSTDDIR = ../../lib
PRGDIR = ../../programs
ZSTDCOMMON_FILES := $(ZSTDDIR)/common/*.c
ZSTDCOMP_FILES := $(ZSTDDIR)/compress/*.c
ZSTDDECOMP_FILES := $(ZSTDDIR)/decompress/*.c
ZSTD_FILES := $(ZSTDDECOMP_FILES) $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES)
MULTITHREAD_LDFLAGS = -pthread
DEBUGFLAGS= -g -DZSTD_DEBUG=1
CPPFLAGS += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(PRGDIR)
CFLAGS ?= -O3
CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-Wstrict-prototypes -Wundef -Wformat-security \
-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
-Wredundant-decls
CFLAGS += $(DEBUGFLAGS)
CFLAGS += $(MOREFLAGS)
FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITHREAD_LDFLAGS)
all: adapt datagen
adapt: $(ZSTD_FILES) adapt.c
$(CC) $(FLAGS) $^ -o $@
adapt-debug: $(ZSTD_FILES) adapt.c
$(CC) $(FLAGS) -DDEBUG_MODE=2 $^ -o adapt
datagen : $(PRGDIR)/datagen.c datagencli.c
$(CC) $(FLAGS) $^ -o $@
test-adapt-correctness: datagen adapt
@./test-correctness.sh
@echo "test correctness complete"
test-adapt-performance: datagen adapt
@./test-performance.sh
@echo "test performance complete"
clean:
@$(RM) -f adapt datagen
@$(RM) -rf *.dSYM
@$(RM) -f tmp*
@$(RM) -f tests/*.zst
@$(RM) -f tests/tmp*
@echo "finished cleaning"
#-----------------------------------------------------------------------------
# make install is validated only for Linux, OSX, BSD, Hurd and Solaris targets
#-----------------------------------------------------------------------------
ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS))
ifneq (,$(filter $(shell uname),SunOS))
INSTALL ?= ginstall
else
INSTALL ?= install
endif
PREFIX ?= /usr/local
DESTDIR ?=
BINDIR ?= $(PREFIX)/bin
INSTALL_PROGRAM ?= $(INSTALL) -m 755
install: adapt
@echo Installing binaries
@$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)/
@$(INSTALL_PROGRAM) adapt $(DESTDIR)$(BINDIR)/zstd-adaptive
@echo zstd-adaptive installation completed
uninstall:
@$(RM) $(DESTDIR)$(BINDIR)/zstd-adaptive
@echo zstd-adaptive programs successfully uninstalled
endif
contrib/adaptive-compression/README.md
+91
View File
@@ -0,0 +1,91 @@
### Summary
`adapt` is a new compression tool targeted at optimizing performance across network connections and pipelines. The tool is aimed at sensing network speeds and adapting compression level based on network or pipe speeds.
In situations where the compression level does not appropriately match the network/pipe speed, compression may be bottlenecking the entire pipeline or the files may not be compressed as much as they potentially could be, therefore losing efficiency. It also becomes quite impractical to manually measure and set an optimalcompression level (which could potentially change over time).
### Using `adapt`
In order to build and use the tool, you can simply run `make adapt` in the `adaptive-compression` directory under `contrib`. This will generate an executable available for use. Another possible method of installation is running `make install`, which will create and install the binary as the command `zstd-adaptive`.
Similar to many other compression utilities, `zstd-adaptive` can be invoked by using the following format:
`zstd-adaptive [options] [file(s)]`
Supported options for the above format are described below.
`zstd-adaptive` also supports reading from `stdin` and writing to `stdout`, which is potentially more useful. By default, if no files are given, `zstd-adaptive` reads from and writes to standard I/O. Therefore, you can simply insert it within a pipeline like so:
`cat FILE | zstd-adaptive | ssh "cat - > tmp.zst"`
If a file is provided, it is also possible to force writing to stdout using the `-c` flag like so:
`zstd-adaptive -c FILE | ssh "cat - > tmp.zst"`
Several options described below can be used to control the behavior of `zstd-adaptive`. More specifically, using the `-l#` and `-u#` flags will will set upper and lower bounds so that the compression level will always be within that range. The `-i#` flag can also be used to change the initial compression level. If an initial compression level is not provided, the initial compression level will be chosen such that it is within the appropriate range (it becomes equal to the lower bound).
### Options
`-oFILE` : write output to `FILE`
`-i#` : provide initial compression level (must within the appropriate bounds)
`-h` : display help/information
`-f` : force the compression level to stay constant
`-c` : force write to `stdout`
`-p` : hide progress bar
`-q` : quiet mode -- do not show progress bar or other information
`-l#` : set a lower bound on the compression level (default is 1)
`-u#` : set an upper bound on the compression level (default is 22)
### Benchmarking / Test results
#### Artificial Tests
These artificial tests were run by using the `pv` command line utility in order to limit pipe speeds (25 MB/s read and 5 MB/s write limits were chosen to mimic severe throughput constraints). A 40 GB backup file was sent through a pipeline, compressed, and written out to a file. Compression time, size, and ratio were computed. Data for `zstd -15` was excluded from these tests because the test runs quite long.
<table>
<tr><th> 25 MB/s read limit </th></tr>
<tr><td>
| Compressor Name | Ratio | Compressed Size | Compression Time |
|:----------------|------:|----------------:|-----------------:|
| zstd -3 | 2.108 | 20.718 GB | 29m 48.530s |
| zstd-adaptive | 2.230 | 19.581 GB | 29m 48.798s |
</td><tr>
</table>
<table>
<tr><th> 5 MB/s write limit </th></tr>
<tr><td>
| Compressor Name | Ratio | Compressed Size | Compression Time |
|:----------------|------:|----------------:|-----------------:|
| zstd -3 | 2.108 | 20.718 GB | 1h 10m 43.076s |
| zstd-adaptive | 2.249 | 19.412 GB | 1h 06m 15.577s |
</td></tr>
</table>
The commands used for this test generally followed the form:
`cat FILE | pv -L 25m -q | COMPRESSION | pv -q > tmp.zst # impose 25 MB/s read limit`
`cat FILE | pv -q | COMPRESSION | pv -L 5m -q > tmp.zst # impose 5 MB/s write limit`
#### SSH Tests
The following tests were performed by piping a relatively large backup file (approximately 80 GB) through compression and over SSH to be stored on a server. The test data includes statistics for time and compressed size on `zstd` at several compression levels, as well as `zstd-adaptive`. The data highlights the potential advantages that `zstd-adaptive` has over using a low static compression level and the negative imapcts that using an excessively high static compression level can have on
pipe throughput.
| Compressor Name | Ratio | Compressed Size | Compression Time |
|:----------------|------:|----------------:|-----------------:|
| zstd -3 | 2.212 | 32.426 GB | 1h 17m 59.756s |
| zstd -15 | 2.374 | 30.213 GB | 2h 56m 59.441s |
| zstd-adaptive | 2.315 | 30.993 GB | 1h 18m 52.860s |
The commands used for this test generally followed the form:
`cat FILE | COMPRESSION | ssh dev "cat - > tmp.zst"`
contrib/adaptive-compression/adapt.c
+1137
View File
@@ -0,0 +1,1137 @@
/*
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdio.h> /* fprintf */
#include <stdlib.h> /* malloc, free */
#include <pthread.h> /* pthread functions */
#include <string.h> /* memset */
#include "zstd_internal.h"
#include "util.h"
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define PRINT(...) fprintf(stdout, __VA_ARGS__)
#define DEBUG(l, ...) { if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } }
#define FILE_CHUNK_SIZE 4 << 20
#define MAX_NUM_JOBS 2
#define stdinmark "/*stdin*\\"
#define stdoutmark "/*stdout*\\"
#define MAX_PATH 256
#define DEFAULT_DISPLAY_LEVEL 1
#define DEFAULT_COMPRESSION_LEVEL 6
#define MAX_COMPRESSION_LEVEL_CHANGE 2
#define CONVERGENCE_LOWER_BOUND 5
#define CLEVEL_DECREASE_COOLDOWN 5
#define CHANGE_BY_TWO_THRESHOLD 0.1
#define CHANGE_BY_ONE_THRESHOLD 0.65
#ifndef DEBUG_MODE
static int g_displayLevel = DEFAULT_DISPLAY_LEVEL;
#else
static int g_displayLevel = DEBUG_MODE;
#endif
static unsigned g_compressionLevel = DEFAULT_COMPRESSION_LEVEL;
static UTIL_time_t g_startTime;
static size_t g_streamedSize = 0;
static unsigned g_useProgressBar = 1;
static UTIL_freq_t g_ticksPerSecond;
static unsigned g_forceCompressionLevel = 0;
static unsigned g_minCLevel = 1;
static unsigned g_maxCLevel;
typedef struct {
void* start;
size_t size;
size_t capacity;
} buffer_t;
typedef struct {
size_t filled;
buffer_t buffer;
} inBuff_t;
typedef struct {
buffer_t src;
buffer_t dst;
unsigned jobID;
unsigned lastJobPlusOne;
size_t compressedSize;
size_t dictSize;
} jobDescription;
typedef struct {
pthread_mutex_t pMutex;
int noError;
} mutex_t;
typedef struct {
pthread_cond_t pCond;
int noError;
} cond_t;
typedef struct {
unsigned compressionLevel;
unsigned numJobs;
unsigned nextJobID;
unsigned threadError;
/*
* JobIDs for the next jobs to be created, compressed, and written
*/
unsigned jobReadyID;
unsigned jobCompressedID;
unsigned jobWriteID;
unsigned allJobsCompleted;
/*
* counter for how many jobs in a row the compression level has not changed
* if the counter becomes >= CONVERGENCE_LOWER_BOUND, the next time the
* compression level tries to change (by non-zero amount) resets the counter
* to 1 and does not apply the change
*/
unsigned convergenceCounter;
/*
* cooldown counter in order to prevent rapid successive decreases in compression level
* whenever compression level is decreased, cooldown is set to CLEVEL_DECREASE_COOLDOWN
* whenever adaptCompressionLevel() is called and cooldown != 0, it is decremented
* as long as cooldown != 0, the compression level cannot be decreased
*/
unsigned cooldown;
/*
* XWaitYCompletion
* Range from 0.0 to 1.0
* if the value is not 1.0, then this implies that thread X waited on thread Y to finish
* and thread Y was XWaitYCompletion finished at the time of the wait (i.e. compressWaitWriteCompletion=0.5
* implies that the compression thread waited on the write thread and it was only 50% finished writing a job)
*/
double createWaitCompressionCompletion;
double compressWaitCreateCompletion;
double compressWaitWriteCompletion;
double writeWaitCompressionCompletion;
/*
* Completion values
* Range from 0.0 to 1.0
* Jobs are divided into mini-chunks in order to measure completion
* these values are updated each time a thread finishes its operation on the
* mini-chunk (i.e. finishes writing out, compressing, etc. this mini-chunk).
*/
double compressionCompletion;
double writeCompletion;
double createCompletion;
mutex_t jobCompressed_mutex;
cond_t jobCompressed_cond;
mutex_t jobReady_mutex;
cond_t jobReady_cond;
mutex_t allJobsCompleted_mutex;
cond_t allJobsCompleted_cond;
mutex_t jobWrite_mutex;
cond_t jobWrite_cond;
mutex_t compressionCompletion_mutex;
mutex_t createCompletion_mutex;
mutex_t writeCompletion_mutex;
mutex_t compressionLevel_mutex;
size_t lastDictSize;
inBuff_t input;
jobDescription* jobs;
ZSTD_CCtx* cctx;
} adaptCCtx;
typedef struct {
adaptCCtx* ctx;
FILE* dstFile;
} outputThreadArg;
typedef struct {
FILE* srcFile;
adaptCCtx* ctx;
outputThreadArg* otArg;
} fcResources;
static void freeCompressionJobs(adaptCCtx* ctx)
{
unsigned u;
for (u=0; u<ctx->numJobs; u++) {
jobDescription job = ctx->jobs[u];
free(job.dst.start);
free(job.src.start);
}
}
static int destroyMutex(mutex_t* mutex)
{
if (mutex->noError) {
int const ret = pthread_mutex_destroy(&mutex->pMutex);
return ret;
}
return 0;
}
static int destroyCond(cond_t* cond)
{
if (cond->noError) {
int const ret = pthread_cond_destroy(&cond->pCond);
return ret;
}
return 0;
}
static int freeCCtx(adaptCCtx* ctx)
{
if (!ctx) return 0;
{
int error = 0;
error |= destroyMutex(&ctx->jobCompressed_mutex);
error |= destroyCond(&ctx->jobCompressed_cond);
error |= destroyMutex(&ctx->jobReady_mutex);
error |= destroyCond(&ctx->jobReady_cond);
error |= destroyMutex(&ctx->allJobsCompleted_mutex);
error |= destroyCond(&ctx->allJobsCompleted_cond);
error |= destroyMutex(&ctx->jobWrite_mutex);
error |= destroyCond(&ctx->jobWrite_cond);
error |= destroyMutex(&ctx->compressionCompletion_mutex);
error |= destroyMutex(&ctx->createCompletion_mutex);
error |= destroyMutex(&ctx->writeCompletion_mutex);
error |= destroyMutex(&ctx->compressionLevel_mutex);
error |= ZSTD_isError(ZSTD_freeCCtx(ctx->cctx));
free(ctx->input.buffer.start);
if (ctx->jobs){
freeCompressionJobs(ctx);
free(ctx->jobs);
}
free(ctx);
return error;
}
}
static int initMutex(mutex_t* mutex)
{
int const ret = pthread_mutex_init(&mutex->pMutex, NULL);
mutex->noError = !ret;
return ret;
}
static int initCond(cond_t* cond)
{
int const ret = pthread_cond_init(&cond->pCond, NULL);
cond->noError = !ret;
return ret;
}
static int initCCtx(adaptCCtx* ctx, unsigned numJobs)
{
ctx->compressionLevel = g_compressionLevel;
{
int pthreadError = 0;
pthreadError |= initMutex(&ctx->jobCompressed_mutex);
pthreadError |= initCond(&ctx->jobCompressed_cond);
pthreadError |= initMutex(&ctx->jobReady_mutex);
pthreadError |= initCond(&ctx->jobReady_cond);
pthreadError |= initMutex(&ctx->allJobsCompleted_mutex);
pthreadError |= initCond(&ctx->allJobsCompleted_cond);
pthreadError |= initMutex(&ctx->jobWrite_mutex);
pthreadError |= initCond(&ctx->jobWrite_cond);
pthreadError |= initMutex(&ctx->compressionCompletion_mutex);
pthreadError |= initMutex(&ctx->createCompletion_mutex);
pthreadError |= initMutex(&ctx->writeCompletion_mutex);
pthreadError |= initMutex(&ctx->compressionLevel_mutex);
if (pthreadError) return pthreadError;
}
ctx->numJobs = numJobs;
ctx->jobReadyID = 0;
ctx->jobCompressedID = 0;
ctx->jobWriteID = 0;
ctx->lastDictSize = 0;
ctx->createWaitCompressionCompletion = 1;
ctx->compressWaitCreateCompletion = 1;
ctx->compressWaitWriteCompletion = 1;
ctx->writeWaitCompressionCompletion = 1;
ctx->createCompletion = 1;
ctx->writeCompletion = 1;
ctx->compressionCompletion = 1;
ctx->convergenceCounter = 0;
ctx->cooldown = 0;
ctx->jobs = calloc(1, numJobs*sizeof(jobDescription));
if (!ctx->jobs) {
DISPLAY("Error: could not allocate space for jobs during context creation\n");
return 1;
}
/* initializing jobs */
{
unsigned jobNum;
for (jobNum=0; jobNum<numJobs; jobNum++) {
jobDescription* job = &ctx->jobs[jobNum];
job->src.start = malloc(2 * FILE_CHUNK_SIZE);
job->dst.start = malloc(ZSTD_compressBound(FILE_CHUNK_SIZE));
job->lastJobPlusOne = 0;
if (!job->src.start || !job->dst.start) {
DISPLAY("Could not allocate buffers for jobs\n");
return 1;
}
job->src.capacity = FILE_CHUNK_SIZE;
job->dst.capacity = ZSTD_compressBound(FILE_CHUNK_SIZE);
}
}
ctx->nextJobID = 0;
ctx->threadError = 0;
ctx->allJobsCompleted = 0;
ctx->cctx = ZSTD_createCCtx();
if (!ctx->cctx) {
DISPLAY("Error: could not allocate ZSTD_CCtx\n");
return 1;
}
ctx->input.filled = 0;
ctx->input.buffer.capacity = 2 * FILE_CHUNK_SIZE;
ctx->input.buffer.start = malloc(ctx->input.buffer.capacity);
if (!ctx->input.buffer.start) {
DISPLAY("Error: could not allocate input buffer\n");
return 1;
}
return 0;
}
static adaptCCtx* createCCtx(unsigned numJobs)
{
adaptCCtx* const ctx = calloc(1, sizeof(adaptCCtx));
if (ctx == NULL) {
DISPLAY("Error: could not allocate space for context\n");
return NULL;
}
{
int const error = initCCtx(ctx, numJobs);
if (error) {
freeCCtx(ctx);
return NULL;
}
return ctx;
}
}
static void signalErrorToThreads(adaptCCtx* ctx)
{
ctx->threadError = 1;
pthread_mutex_lock(&ctx->jobReady_mutex.pMutex);
pthread_cond_signal(&ctx->jobReady_cond.pCond);
pthread_mutex_unlock(&ctx->jobReady_mutex.pMutex);
pthread_mutex_lock(&ctx->jobCompressed_mutex.pMutex);
pthread_cond_broadcast(&ctx->jobCompressed_cond.pCond);
pthread_mutex_unlock(&ctx->jobReady_mutex.pMutex);
pthread_mutex_lock(&ctx->jobWrite_mutex.pMutex);
pthread_cond_signal(&ctx->jobWrite_cond.pCond);
pthread_mutex_unlock(&ctx->jobWrite_mutex.pMutex);
pthread_mutex_lock(&ctx->allJobsCompleted_mutex.pMutex);
pthread_cond_signal(&ctx->allJobsCompleted_cond.pCond);
pthread_mutex_unlock(&ctx->allJobsCompleted_mutex.pMutex);
}
static void waitUntilAllJobsCompleted(adaptCCtx* ctx)
{
if (!ctx) return;
pthread_mutex_lock(&ctx->allJobsCompleted_mutex.pMutex);
while (ctx->allJobsCompleted == 0 && !ctx->threadError) {
pthread_cond_wait(&ctx->allJobsCompleted_cond.pCond, &ctx->allJobsCompleted_mutex.pMutex);
}
pthread_mutex_unlock(&ctx->allJobsCompleted_mutex.pMutex);
}
/* map completion percentages to values for changing compression level */
static unsigned convertCompletionToChange(double completion)
{
if (completion < CHANGE_BY_TWO_THRESHOLD) {
return 2;
}
else if (completion < CHANGE_BY_ONE_THRESHOLD) {
return 1;
}
else {
return 0;
}
}
/*
* Compression level is changed depending on which part of the compression process is lagging
* Currently, three theads exist for job creation, compression, and file writing respectively.
* adaptCompressionLevel() increments or decrements compression level based on which of the threads is lagging
* job creation or file writing lag => increased compression level
* compression thread lag => decreased compression level
* detecting which thread is lagging is done by keeping track of how many calls each thread makes to pthread_cond_wait
*/
static void adaptCompressionLevel(adaptCCtx* ctx)
{
double createWaitCompressionCompletion;
double compressWaitCreateCompletion;
double compressWaitWriteCompletion;
double writeWaitCompressionCompletion;
double const threshold = 0.00001;
unsigned prevCompressionLevel;
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
prevCompressionLevel = ctx->compressionLevel;
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
if (g_forceCompressionLevel) {
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
ctx->compressionLevel = g_compressionLevel;
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
return;
}
DEBUG(2, "adapting compression level %u\n", prevCompressionLevel);
/* read and reset completion measurements */
pthread_mutex_lock(&ctx->compressionCompletion_mutex.pMutex);
DEBUG(2, "createWaitCompressionCompletion %f\n", ctx->createWaitCompressionCompletion);
DEBUG(2, "writeWaitCompressionCompletion %f\n", ctx->writeWaitCompressionCompletion);
createWaitCompressionCompletion = ctx->createWaitCompressionCompletion;
writeWaitCompressionCompletion = ctx->writeWaitCompressionCompletion;
pthread_mutex_unlock(&ctx->compressionCompletion_mutex.pMutex);
pthread_mutex_lock(&ctx->writeCompletion_mutex.pMutex);
DEBUG(2, "compressWaitWriteCompletion %f\n", ctx->compressWaitWriteCompletion);
compressWaitWriteCompletion = ctx->compressWaitWriteCompletion;
pthread_mutex_unlock(&ctx->writeCompletion_mutex.pMutex);
pthread_mutex_lock(&ctx->createCompletion_mutex.pMutex);
DEBUG(2, "compressWaitCreateCompletion %f\n", ctx->compressWaitCreateCompletion);
compressWaitCreateCompletion = ctx->compressWaitCreateCompletion;
pthread_mutex_unlock(&ctx->createCompletion_mutex.pMutex);
DEBUG(2, "convergence counter: %u\n", ctx->convergenceCounter);
assert(g_minCLevel <= prevCompressionLevel && g_maxCLevel >= prevCompressionLevel);
/* adaptation logic */
if (ctx->cooldown) ctx->cooldown--;
if ((1-createWaitCompressionCompletion > threshold || 1-writeWaitCompressionCompletion > threshold) && ctx->cooldown == 0) {
/* create or write waiting on compression */
/* use whichever one waited less because it was slower */
double const completion = MAX(createWaitCompressionCompletion, writeWaitCompressionCompletion);
unsigned const change = convertCompletionToChange(completion);
unsigned const boundChange = MIN(change, prevCompressionLevel - g_minCLevel);
if (ctx->convergenceCounter >= CONVERGENCE_LOWER_BOUND && boundChange != 0) {
/* reset convergence counter, might have been a spike */
ctx->convergenceCounter = 0;
DEBUG(2, "convergence counter reset, no change applied\n");
}
else if (boundChange != 0) {
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
ctx->compressionLevel -= boundChange;
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
ctx->cooldown = CLEVEL_DECREASE_COOLDOWN;
ctx->convergenceCounter = 1;
DEBUG(2, "create or write threads waiting on compression, tried to decrease compression level by %u\n\n", boundChange);
}
}
else if (1-compressWaitWriteCompletion > threshold || 1-compressWaitCreateCompletion > threshold) {
/* compress waiting on write */
double const completion = MIN(compressWaitWriteCompletion, compressWaitCreateCompletion);
unsigned const change = convertCompletionToChange(completion);
unsigned const boundChange = MIN(change, g_maxCLevel - prevCompressionLevel);
if (ctx->convergenceCounter >= CONVERGENCE_LOWER_BOUND && boundChange != 0) {
/* reset convergence counter, might have been a spike */
ctx->convergenceCounter = 0;
DEBUG(2, "convergence counter reset, no change applied\n");
}
else if (boundChange != 0) {
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
ctx->compressionLevel += boundChange;
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
ctx->cooldown = 0;
ctx->convergenceCounter = 1;
DEBUG(2, "compress waiting on write or create, tried to increase compression level by %u\n\n", boundChange);
}
}
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
if (ctx->compressionLevel == prevCompressionLevel) {
ctx->convergenceCounter++;
}
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
}
static size_t getUseableDictSize(unsigned compressionLevel)
{
ZSTD_parameters const params = ZSTD_getParams(compressionLevel, 0, 0);
unsigned const overlapLog = compressionLevel >= (unsigned)ZSTD_maxCLevel() ? 0 : 3;
size_t const overlapSize = 1 << (params.cParams.windowLog - overlapLog);
return overlapSize;
}
static void* compressionThread(void* arg)
{
adaptCCtx* const ctx = (adaptCCtx*)arg;
unsigned currJob = 0;
for ( ; ; ) {
unsigned const currJobIndex = currJob % ctx->numJobs;
jobDescription* const job = &ctx->jobs[currJobIndex];
DEBUG(2, "starting compression for job %u\n", currJob);
{
/* check if compression thread will have to wait */
unsigned willWaitForCreate = 0;
unsigned willWaitForWrite = 0;
pthread_mutex_lock(&ctx->jobReady_mutex.pMutex);
if (currJob + 1 > ctx->jobReadyID) willWaitForCreate = 1;
pthread_mutex_unlock(&ctx->jobReady_mutex.pMutex);
pthread_mutex_lock(&ctx->jobWrite_mutex.pMutex);
if (currJob - ctx->jobWriteID >= ctx->numJobs) willWaitForWrite = 1;
pthread_mutex_unlock(&ctx->jobWrite_mutex.pMutex);
pthread_mutex_lock(&ctx->createCompletion_mutex.pMutex);
if (willWaitForCreate) {
DEBUG(2, "compression will wait for create on job %u\n", currJob);
ctx->compressWaitCreateCompletion = ctx->createCompletion;
DEBUG(2, "create completion %f\n", ctx->compressWaitCreateCompletion);
}
else {
ctx->compressWaitCreateCompletion = 1;
}
pthread_mutex_unlock(&ctx->createCompletion_mutex.pMutex);
pthread_mutex_lock(&ctx->writeCompletion_mutex.pMutex);
if (willWaitForWrite) {
DEBUG(2, "compression will wait for write on job %u\n", currJob);
ctx->compressWaitWriteCompletion = ctx->writeCompletion;
DEBUG(2, "write completion %f\n", ctx->compressWaitWriteCompletion);
}
else {
ctx->compressWaitWriteCompletion = 1;
}
pthread_mutex_unlock(&ctx->writeCompletion_mutex.pMutex);
}
/* wait until job is ready */
pthread_mutex_lock(&ctx->jobReady_mutex.pMutex);
while (currJob + 1 > ctx->jobReadyID && !ctx->threadError) {
pthread_cond_wait(&ctx->jobReady_cond.pCond, &ctx->jobReady_mutex.pMutex);
}
pthread_mutex_unlock(&ctx->jobReady_mutex.pMutex);
/* wait until job previously in this space is written */
pthread_mutex_lock(&ctx->jobWrite_mutex.pMutex);
while (currJob - ctx->jobWriteID >= ctx->numJobs && !ctx->threadError) {
pthread_cond_wait(&ctx->jobWrite_cond.pCond, &ctx->jobWrite_mutex.pMutex);
}
pthread_mutex_unlock(&ctx->jobWrite_mutex.pMutex);
/* reset compression completion */
pthread_mutex_lock(&ctx->compressionCompletion_mutex.pMutex);
ctx->compressionCompletion = 0;
pthread_mutex_unlock(&ctx->compressionCompletion_mutex.pMutex);
/* adapt compression level */
if (currJob) adaptCompressionLevel(ctx);
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
DEBUG(2, "job %u compressed with level %u\n", currJob, ctx->compressionLevel);
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
/* compress the data */
{
size_t const compressionBlockSize = ZSTD_BLOCKSIZE_MAX; /* 128 KB */
unsigned cLevel;
unsigned blockNum = 0;
size_t remaining = job->src.size;
size_t srcPos = 0;
size_t dstPos = 0;
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
cLevel = ctx->compressionLevel;
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
/* reset compressed size */
job->compressedSize = 0;
DEBUG(2, "calling ZSTD_compressBegin()\n");
/* begin compression */
{
size_t const useDictSize = MIN(getUseableDictSize(cLevel), job->dictSize);
size_t const dictModeError = ZSTD_setCCtxParameter(ctx->cctx, ZSTD_p_forceRawDict, 1);
ZSTD_parameters params = ZSTD_getParams(cLevel, 0, useDictSize);
params.cParams.windowLog = 23;
{
size_t const initError = ZSTD_compressBegin_advanced(ctx->cctx, job->src.start + job->dictSize - useDictSize, useDictSize, params, 0);
size_t const windowSizeError = ZSTD_setCCtxParameter(ctx->cctx, ZSTD_p_forceWindow, 1);
if (ZSTD_isError(dictModeError) || ZSTD_isError(initError) || ZSTD_isError(windowSizeError)) {
DISPLAY("Error: something went wrong while starting compression\n");
signalErrorToThreads(ctx);
return arg;
}
}
}
DEBUG(2, "finished with ZSTD_compressBegin()\n");
do {
size_t const actualBlockSize = MIN(remaining, compressionBlockSize);
/* continue compression */
if (currJob != 0 || blockNum != 0) { /* not first block of first job flush/overwrite the frame header */
size_t const hSize = ZSTD_compressContinue(ctx->cctx, job->dst.start + dstPos, job->dst.capacity - dstPos, job->src.start + job->dictSize + srcPos, 0);
if (ZSTD_isError(hSize)) {
DISPLAY("Error: something went wrong while continuing compression\n");
job->compressedSize = hSize;
signalErrorToThreads(ctx);
return arg;
}
ZSTD_invalidateRepCodes(ctx->cctx);
}
{
size_t const ret = (job->lastJobPlusOne == currJob + 1 && remaining == actualBlockSize) ?
ZSTD_compressEnd (ctx->cctx, job->dst.start + dstPos, job->dst.capacity - dstPos, job->src.start + job->dictSize + srcPos, actualBlockSize) :
ZSTD_compressContinue(ctx->cctx, job->dst.start + dstPos, job->dst.capacity - dstPos, job->src.start + job->dictSize + srcPos, actualBlockSize);
if (ZSTD_isError(ret)) {
DISPLAY("Error: something went wrong during compression: %s\n", ZSTD_getErrorName(ret));
signalErrorToThreads(ctx);
return arg;
}
job->compressedSize += ret;
remaining -= actualBlockSize;
srcPos += actualBlockSize;
dstPos += ret;
blockNum++;
/* update completion */
pthread_mutex_lock(&ctx->compressionCompletion_mutex.pMutex);
ctx->compressionCompletion = 1 - (double)remaining/job->src.size;
pthread_mutex_unlock(&ctx->compressionCompletion_mutex.pMutex);
}
} while (remaining != 0);
job->dst.size = job->compressedSize;
}
pthread_mutex_lock(&ctx->jobCompressed_mutex.pMutex);
ctx->jobCompressedID++;
pthread_cond_broadcast(&ctx->jobCompressed_cond.pCond);
pthread_mutex_unlock(&ctx->jobCompressed_mutex.pMutex);
if (job->lastJobPlusOne == currJob + 1 || ctx->threadError) {
/* finished compressing all jobs */
break;
}
DEBUG(2, "finished compressing job %u\n", currJob);
currJob++;
}
return arg;
}
static void displayProgress(unsigned cLevel, unsigned last)
{
UTIL_time_t currTime;
UTIL_getTime(&currTime);
if (!g_useProgressBar) return;
{
double const timeElapsed = (double)(UTIL_getSpanTimeMicro(g_ticksPerSecond, g_startTime, currTime) / 1000.0);
double const sizeMB = (double)g_streamedSize / (1 << 20);
double const avgCompRate = sizeMB * 1000 / timeElapsed;
fprintf(stderr, "\r| Comp. Level: %2u | Time Elapsed: %7.2f s | Data Size: %7.1f MB | Avg Comp. Rate: %6.2f MB/s |", cLevel, timeElapsed/1000.0, sizeMB, avgCompRate);
if (last) {
fprintf(stderr, "\n");
}
else {
fflush(stderr);
}
}
}
static void* outputThread(void* arg)
{
outputThreadArg* const otArg = (outputThreadArg*)arg;
adaptCCtx* const ctx = otArg->ctx;
FILE* const dstFile = otArg->dstFile;
unsigned currJob = 0;
for ( ; ; ) {
unsigned const currJobIndex = currJob % ctx->numJobs;
jobDescription* const job = &ctx->jobs[currJobIndex];
unsigned willWaitForCompress = 0;
DEBUG(2, "starting write for job %u\n", currJob);
pthread_mutex_lock(&ctx->jobCompressed_mutex.pMutex);
if (currJob + 1 > ctx->jobCompressedID) willWaitForCompress = 1;
pthread_mutex_unlock(&ctx->jobCompressed_mutex.pMutex);
pthread_mutex_lock(&ctx->compressionCompletion_mutex.pMutex);
if (willWaitForCompress) {
/* write thread is waiting on compression thread */
ctx->writeWaitCompressionCompletion = ctx->compressionCompletion;
DEBUG(2, "writer thread waiting for nextJob: %u, writeWaitCompressionCompletion %f\n", currJob, ctx->writeWaitCompressionCompletion);
}
else {
ctx->writeWaitCompressionCompletion = 1;
}
pthread_mutex_unlock(&ctx->compressionCompletion_mutex.pMutex);
pthread_mutex_lock(&ctx->jobCompressed_mutex.pMutex);
while (currJob + 1 > ctx->jobCompressedID && !ctx->threadError) {
pthread_cond_wait(&ctx->jobCompressed_cond.pCond, &ctx->jobCompressed_mutex.pMutex);
}
pthread_mutex_unlock(&ctx->jobCompressed_mutex.pMutex);
/* reset write completion */
pthread_mutex_lock(&ctx->writeCompletion_mutex.pMutex);
ctx->writeCompletion = 0;
pthread_mutex_unlock(&ctx->writeCompletion_mutex.pMutex);
{
size_t const compressedSize = job->compressedSize;
size_t remaining = compressedSize;
if (ZSTD_isError(compressedSize)) {
DISPLAY("Error: an error occurred during compression\n");
signalErrorToThreads(ctx);
return arg;
}
{
size_t const blockSize = MAX(compressedSize >> 7, 1 << 10);
size_t pos = 0;
for ( ; ; ) {
size_t const writeSize = MIN(remaining, blockSize);
size_t const ret = fwrite(job->dst.start + pos, 1, writeSize, dstFile);
if (ret != writeSize) break;
pos += ret;
remaining -= ret;
/* update completion variable for writing */
pthread_mutex_lock(&ctx->writeCompletion_mutex.pMutex);
ctx->writeCompletion = 1 - (double)remaining/compressedSize;
pthread_mutex_unlock(&ctx->writeCompletion_mutex.pMutex);
if (remaining == 0) break;
}
if (pos != compressedSize) {
DISPLAY("Error: an error occurred during file write operation\n");
signalErrorToThreads(ctx);
return arg;
}
}
}
{
unsigned cLevel;
pthread_mutex_lock(&ctx->compressionLevel_mutex.pMutex);
cLevel = ctx->compressionLevel;
pthread_mutex_unlock(&ctx->compressionLevel_mutex.pMutex);
displayProgress(cLevel, job->lastJobPlusOne == currJob + 1);
}
pthread_mutex_lock(&ctx->jobWrite_mutex.pMutex);
ctx->jobWriteID++;
pthread_cond_signal(&ctx->jobWrite_cond.pCond);
pthread_mutex_unlock(&ctx->jobWrite_mutex.pMutex);
if (job->lastJobPlusOne == currJob + 1 || ctx->threadError) {
/* finished with all jobs */
pthread_mutex_lock(&ctx->allJobsCompleted_mutex.pMutex);
ctx->allJobsCompleted = 1;
pthread_cond_signal(&ctx->allJobsCompleted_cond.pCond);
pthread_mutex_unlock(&ctx->allJobsCompleted_mutex.pMutex);
break;
}
DEBUG(2, "finished writing job %u\n", currJob);
currJob++;
}
return arg;
}
static int createCompressionJob(adaptCCtx* ctx, size_t srcSize, int last)
{
unsigned const nextJob = ctx->nextJobID;
unsigned const nextJobIndex = nextJob % ctx->numJobs;
jobDescription* const job = &ctx->jobs[nextJobIndex];
job->src.size = srcSize;
job->jobID = nextJob;
if (last) job->lastJobPlusOne = nextJob + 1;
{
/* swap buffer */
void* const copy = job->src.start;
job->src.start = ctx->input.buffer.start;
ctx->input.buffer.start = copy;
}
job->dictSize = ctx->lastDictSize;
ctx->nextJobID++;
/* if not on the last job, reuse data as dictionary in next job */
if (!last) {
size_t const oldDictSize = ctx->lastDictSize;
memcpy(ctx->input.buffer.start, job->src.start + oldDictSize, srcSize);
ctx->lastDictSize = srcSize;
ctx->input.filled = srcSize;
}
/* signal job ready */
pthread_mutex_lock(&ctx->jobReady_mutex.pMutex);
ctx->jobReadyID++;
pthread_cond_signal(&ctx->jobReady_cond.pCond);
pthread_mutex_unlock(&ctx->jobReady_mutex.pMutex);
return 0;
}
static int performCompression(adaptCCtx* ctx, FILE* const srcFile, outputThreadArg* otArg)
{
/* early error check to exit */
if (!ctx || !srcFile || !otArg) {
return 1;
}
/* create output thread */
{
pthread_t out;
if (pthread_create(&out, NULL, &outputThread, otArg)) {
DISPLAY("Error: could not create output thread\n");
signalErrorToThreads(ctx);
return 1;
}
else if (pthread_detach(out)) {
DISPLAY("Error: could not detach output thread\n");
signalErrorToThreads(ctx);
return 1;
}
}
/* create compression thread */
{
pthread_t compression;
if (pthread_create(&compression, NULL, &compressionThread, ctx)) {
DISPLAY("Error: could not create compression thread\n");
signalErrorToThreads(ctx);
return 1;
}
else if (pthread_detach(compression)) {
DISPLAY("Error: could not detach compression thread\n");
signalErrorToThreads(ctx);
return 1;
}
}
{
unsigned currJob = 0;
/* creating jobs */
for ( ; ; ) {
size_t pos = 0;
size_t const readBlockSize = 1 << 15;
size_t remaining = FILE_CHUNK_SIZE;
unsigned const nextJob = ctx->nextJobID;
unsigned willWaitForCompress = 0;
DEBUG(2, "starting creation of job %u\n", currJob);
pthread_mutex_lock(&ctx->jobCompressed_mutex.pMutex);
if (nextJob - ctx->jobCompressedID >= ctx->numJobs) willWaitForCompress = 1;
pthread_mutex_unlock(&ctx->jobCompressed_mutex.pMutex);
pthread_mutex_lock(&ctx->compressionCompletion_mutex.pMutex);
if (willWaitForCompress) {
/* creation thread is waiting, take measurement of completion */
ctx->createWaitCompressionCompletion = ctx->compressionCompletion;
DEBUG(2, "create thread waiting for nextJob: %u, createWaitCompressionCompletion %f\n", nextJob, ctx->createWaitCompressionCompletion);
}
else {
ctx->createWaitCompressionCompletion = 1;
}
pthread_mutex_unlock(&ctx->compressionCompletion_mutex.pMutex);
/* wait until the job has been compressed */
pthread_mutex_lock(&ctx->jobCompressed_mutex.pMutex);
while (nextJob - ctx->jobCompressedID >= ctx->numJobs && !ctx->threadError) {
pthread_cond_wait(&ctx->jobCompressed_cond.pCond, &ctx->jobCompressed_mutex.pMutex);
}
pthread_mutex_unlock(&ctx->jobCompressed_mutex.pMutex);
/* reset create completion */
pthread_mutex_lock(&ctx->createCompletion_mutex.pMutex);
ctx->createCompletion = 0;
pthread_mutex_unlock(&ctx->createCompletion_mutex.pMutex);
while (remaining != 0 && !feof(srcFile)) {
size_t const ret = fread(ctx->input.buffer.start + ctx->input.filled + pos, 1, readBlockSize, srcFile);
if (ret != readBlockSize && !feof(srcFile)) {
/* error could not read correct number of bytes */
DISPLAY("Error: problem occurred during read from src file\n");
signalErrorToThreads(ctx);
return 1;
}
pos += ret;
remaining -= ret;
pthread_mutex_lock(&ctx->createCompletion_mutex.pMutex);
ctx->createCompletion = 1 - (double)remaining/((size_t)FILE_CHUNK_SIZE);
pthread_mutex_unlock(&ctx->createCompletion_mutex.pMutex);
}
if (remaining != 0 && !feof(srcFile)) {
DISPLAY("Error: problem occurred during read from src file\n");
signalErrorToThreads(ctx);
return 1;
}
g_streamedSize += pos;
/* reading was fine, now create the compression job */
{
int const last = feof(srcFile);
int const error = createCompressionJob(ctx, pos, last);
if (error != 0) {
signalErrorToThreads(ctx);
return error;
}
}
DEBUG(2, "finished creating job %u\n", currJob);
currJob++;
if (feof(srcFile)) {
break;
}
}
}
/* success -- created all jobs */
return 0;
}
static fcResources createFileCompressionResources(const char* const srcFilename, const char* const dstFilenameOrNull)
{
fcResources fcr;
unsigned const stdinUsed = !strcmp(srcFilename, stdinmark);
FILE* const srcFile = stdinUsed ? stdin : fopen(srcFilename, "rb");
const char* const outFilenameIntermediate = (stdinUsed && !dstFilenameOrNull) ? stdoutmark : dstFilenameOrNull;
const char* outFilename = outFilenameIntermediate;
char fileAndSuffix[MAX_PATH];
size_t const numJobs = MAX_NUM_JOBS;
memset(&fcr, 0, sizeof(fcr));
if (!outFilenameIntermediate) {
if (snprintf(fileAndSuffix, MAX_PATH, "%s.zst", srcFilename) + 1 > MAX_PATH) {
DISPLAY("Error: output filename is too long\n");
return fcr;
}
outFilename = fileAndSuffix;
}
{
unsigned const stdoutUsed = !strcmp(outFilename, stdoutmark);
FILE* const dstFile = stdoutUsed ? stdout : fopen(outFilename, "wb");
fcr.otArg = malloc(sizeof(outputThreadArg));
if (!fcr.otArg) {
DISPLAY("Error: could not allocate space for output thread argument\n");
return fcr;
}
fcr.otArg->dstFile = dstFile;
}
/* checking for errors */
if (!fcr.otArg->dstFile || !srcFile) {
DISPLAY("Error: some file(s) could not be opened\n");
return fcr;
}
/* creating context */
fcr.ctx = createCCtx(numJobs);
fcr.otArg->ctx = fcr.ctx;
fcr.srcFile = srcFile;
return fcr;
}
static int freeFileCompressionResources(fcResources* fcr)
{
int ret = 0;
waitUntilAllJobsCompleted(fcr->ctx);
ret |= (fcr->srcFile != NULL) ? fclose(fcr->srcFile) : 0;
ret |= (fcr->ctx != NULL) ? freeCCtx(fcr->ctx) : 0;
if (fcr->otArg) {
ret |= (fcr->otArg->dstFile != stdout) ? fclose(fcr->otArg->dstFile) : 0;
free(fcr->otArg);
/* no need to freeCCtx() on otArg->ctx because it should be the same context */
}
return ret;
}
static int compressFilename(const char* const srcFilename, const char* const dstFilenameOrNull)
{
int ret = 0;
fcResources fcr = createFileCompressionResources(srcFilename, dstFilenameOrNull);
UTIL_getTime(&g_startTime);
g_streamedSize = 0;
ret |= performCompression(fcr.ctx, fcr.srcFile, fcr.otArg);
ret |= freeFileCompressionResources(&fcr);
return ret;
}
static int compressFilenames(const char** filenameTable, unsigned numFiles, unsigned forceStdout)
{
int ret = 0;
unsigned fileNum;
for (fileNum=0; fileNum<numFiles; fileNum++) {
const char* filename = filenameTable[fileNum];
if (!forceStdout) {
ret |= compressFilename(filename, NULL);
}
else {
ret |= compressFilename(filename, stdoutmark);
}
}
return ret;
}
/*! readU32FromChar() :
@return : unsigned integer value read from input in `char` format
allows and interprets K, KB, KiB, M, MB and MiB suffix.
Will also modify `*stringPtr`, advancing it to position where it stopped reading.
Note : function result can overflow if digit string > MAX_UINT */
static unsigned readU32FromChar(const char** stringPtr)
{
unsigned result = 0;
while ((**stringPtr >='0') && (**stringPtr <='9'))
result *= 10, result += **stringPtr - '0', (*stringPtr)++ ;
if ((**stringPtr=='K') || (**stringPtr=='M')) {
result <<= 10;
if (**stringPtr=='M') result <<= 10;
(*stringPtr)++ ;
if (**stringPtr=='i') (*stringPtr)++;
if (**stringPtr=='B') (*stringPtr)++;
}
return result;
}
static void help(const char* progPath)
{
PRINT("Usage:\n");
PRINT(" %s [options] [file(s)]\n", progPath);
PRINT("\n");
PRINT("Options:\n");
PRINT(" -oFILE : specify the output file name\n");
PRINT(" -i# : provide initial compression level -- default %d, must be in the range [L, U] where L and U are bound values (see below for defaults)\n", DEFAULT_COMPRESSION_LEVEL);
PRINT(" -h : display help/information\n");
PRINT(" -f : force the compression level to stay constant\n");
PRINT(" -c : force write to stdout\n");
PRINT(" -p : hide progress bar\n");
PRINT(" -q : quiet mode -- do not show progress bar or other information\n");
PRINT(" -l# : provide lower bound for compression level -- default 1\n");
PRINT(" -u# : provide upper bound for compression level -- default %u\n", ZSTD_maxCLevel());
}
/* return 0 if successful, else return error */
int main(int argCount, const char* argv[])
{
const char* outFilename = NULL;
const char** filenameTable = (const char**)malloc(argCount*sizeof(const char*));
unsigned filenameIdx = 0;
unsigned forceStdout = 0;
unsigned providedInitialCLevel = 0;
int ret = 0;
int argNum;
filenameTable[0] = stdinmark;
g_maxCLevel = ZSTD_maxCLevel();
UTIL_initTimer(&g_ticksPerSecond);
if (filenameTable == NULL) {
DISPLAY("Error: could not allocate sapce for filename table.\n");
return 1;
}
for (argNum=1; argNum<argCount; argNum++) {
const char* argument = argv[argNum];
/* output filename designated with "-o" */
if (argument[0]=='-' && strlen(argument) > 1) {
switch (argument[1]) {
case 'o':
argument += 2;
outFilename = argument;
break;
case 'i':
argument += 2;
g_compressionLevel = readU32FromChar(&argument);
providedInitialCLevel = 1;
break;
case 'h':
help(argv[0]);
goto _main_exit;
case 'p':
g_useProgressBar = 0;
break;
case 'c':
forceStdout = 1;
outFilename = stdoutmark;
break;
case 'f':
g_forceCompressionLevel = 1;
break;
case 'q':
g_useProgressBar = 0;
g_displayLevel = 0;
break;
case 'l':
argument += 2;
g_minCLevel = readU32FromChar(&argument);
break;
case 'u':
argument += 2;
g_maxCLevel = readU32FromChar(&argument);
break;
default:
DISPLAY("Error: invalid argument provided\n");
ret = 1;
goto _main_exit;
}
continue;
}
/* regular files to be compressed */
filenameTable[filenameIdx++] = argument;
}
/* check initial, max, and min compression levels */
{
unsigned const minMaxInconsistent = g_minCLevel > g_maxCLevel;
unsigned const initialNotInRange = g_minCLevel > g_compressionLevel || g_maxCLevel < g_compressionLevel;
if (minMaxInconsistent || (initialNotInRange && providedInitialCLevel)) {
DISPLAY("Error: provided compression level parameters are invalid\n");
ret = 1;
goto _main_exit;
}
else if (initialNotInRange) {
g_compressionLevel = g_minCLevel;
}
}
/* error checking with number of files */
if (filenameIdx > 1 && (outFilename != NULL && strcmp(outFilename, stdoutmark))) {
DISPLAY("Error: multiple input files provided, cannot use specified output file\n");
ret = 1;
goto _main_exit;
}
/* compress files */
if (filenameIdx <= 1) {
ret |= compressFilename(filenameTable[0], outFilename);
}
else {
ret |= compressFilenames(filenameTable, filenameIdx, forceStdout);
}
_main_exit:
free(filenameTable);
return ret;
}
contrib/adaptive-compression/datagencli.c
+129
View File
@@ -0,0 +1,129 @@
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*-************************************
* Dependencies
**************************************/
#include "util.h" /* Compiler options */
#include <stdio.h> /* fprintf, stderr */
#include "datagen.h" /* RDG_generate */
/*-************************************
* Constants
**************************************/
#define KB *(1 <<10)
#define MB *(1 <<20)
#define GB *(1U<<30)
#define SIZE_DEFAULT ((64 KB) + 1)
#define SEED_DEFAULT 0
#define COMPRESSIBILITY_DEFAULT 50
/*-************************************
* Macros
**************************************/
#define DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) if (displayLevel>=l) { DISPLAY(__VA_ARGS__); }
static unsigned displayLevel = 2;
/*-*******************************************************
* Command line
*********************************************************/
static int usage(const char* programName)
{
DISPLAY( "Compressible data generator\n");
DISPLAY( "Usage :\n");
DISPLAY( " %s [args]\n", programName);
DISPLAY( "\n");
DISPLAY( "Arguments :\n");
DISPLAY( " -g# : generate # data (default:%i)\n", SIZE_DEFAULT);
DISPLAY( " -s# : Select seed (default:%i)\n", SEED_DEFAULT);
DISPLAY( " -P# : Select compressibility in %% (default:%i%%)\n",
COMPRESSIBILITY_DEFAULT);
DISPLAY( " -h : display help and exit\n");
return 0;
}
int main(int argc, const char** argv)
{
unsigned probaU32 = COMPRESSIBILITY_DEFAULT;
double litProba = 0.0;
U64 size = SIZE_DEFAULT;
U32 seed = SEED_DEFAULT;
const char* const programName = argv[0];
int argNb;
for(argNb=1; argNb<argc; argNb++) {
const char* argument = argv[argNb];
if(!argument) continue; /* Protection if argument empty */
/* Handle commands. Aggregated commands are allowed */
if (*argument=='-') {
argument++;
while (*argument!=0) {
switch(*argument)
{
case 'h':
return usage(programName);
case 'g':
argument++;
size=0;
while ((*argument>='0') && (*argument<='9'))
size *= 10, size += *argument++ - '0';
if (*argument=='K') { size <<= 10; argument++; }
if (*argument=='M') { size <<= 20; argument++; }
if (*argument=='G') { size <<= 30; argument++; }
if (*argument=='B') { argument++; }
break;
case 's':
argument++;
seed=0;
while ((*argument>='0') && (*argument<='9'))
seed *= 10, seed += *argument++ - '0';
break;
case 'P':
argument++;
probaU32 = 0;
while ((*argument>='0') && (*argument<='9'))
probaU32 *= 10, probaU32 += *argument++ - '0';
if (probaU32>100) probaU32 = 100;
break;
case 'L': /* hidden argument : Literal distribution probability */
argument++;
litProba=0.;
while ((*argument>='0') && (*argument<='9'))
litProba *= 10, litProba += *argument++ - '0';
if (litProba>100.) litProba=100.;
litProba /= 100.;
break;
case 'v':
displayLevel = 4;
argument++;
break;
default:
return usage(programName);
}
} } } /* for(argNb=1; argNb<argc; argNb++) */
DISPLAYLEVEL(4, "Compressible data Generator \n");
if (probaU32!=COMPRESSIBILITY_DEFAULT)
DISPLAYLEVEL(3, "Compressibility : %i%%\n", probaU32);
DISPLAYLEVEL(3, "Seed = %u \n", seed);
RDG_genStdout(size, (double)probaU32/100, litProba, seed);
DISPLAYLEVEL(1, "\n");
return 0;
}
contrib/adaptive-compression/test-correctness.sh
Executable
+252
View File
@@ -0,0 +1,252 @@
echo "correctness tests -- general"
./datagen -s1 -g1GB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s2 -g500MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s3 -g250MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s4 -g125MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s5 -g50MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s6 -g25MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s7 -g10MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s8 -g5MB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s9 -g500KB > tmp
./adapt -otmp.zst tmp
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- streaming"
./datagen -s10 -g1GB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s11 -g100MB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s12 -g10MB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s13 -g1MB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s14 -g100KB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s15 -g10KB > tmp
cat tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- read limit"
./datagen -s16 -g1GB > tmp
pv -L 50m -q tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s17 -g100MB > tmp
pv -L 50m -q tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s18 -g10MB > tmp
pv -L 50m -q tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s19 -g1MB > tmp
pv -L 50m -q tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s20 -g100KB > tmp
pv -L 50m -q tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s21 -g10KB > tmp
pv -L 50m -q tmp | ./adapt > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- write limit"
./datagen -s22 -g1GB > tmp
pv -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s23 -g100MB > tmp
pv -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s24 -g10MB > tmp
pv -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s25 -g1MB > tmp
pv -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s26 -g100KB > tmp
pv -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s27 -g10KB > tmp
pv -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- read and write limits"
./datagen -s28 -g1GB > tmp
pv -L 50m -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s29 -g100MB > tmp
pv -L 50m -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s30 -g10MB > tmp
pv -L 50m -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s31 -g1MB > tmp
pv -L 50m -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s32 -g100KB > tmp
pv -L 50m -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s33 -g10KB > tmp
pv -L 50m -q tmp | ./adapt | pv -L 5m -q > tmp.zst
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- forced compression level"
./datagen -s34 -g1GB > tmp
./adapt tmp -otmp.zst -i11 -f
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s35 -g100MB > tmp
./adapt tmp -otmp.zst -i11 -f
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s36 -g10MB > tmp
./adapt tmp -otmp.zst -i11 -f
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s37 -g1MB > tmp
./adapt tmp -otmp.zst -i11 -f
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s38 -g100KB > tmp
./adapt tmp -otmp.zst -i11 -f
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
./datagen -s39 -g10KB > tmp
./adapt tmp -otmp.zst -i11 -f
zstd -d tmp.zst -o tmp2
diff -s -q tmp tmp2
rm tmp*
echo -e "\ncorrectness tests -- window size test"
./datagen -s39 -g1GB | pv -L 25m -q | ./adapt -i1 | pv -q > tmp.zst
zstd -d tmp.zst
rm tmp*
echo -e "\ncorrectness tests -- testing bounds"
./datagen -s40 -g1GB | pv -L 25m -q | ./adapt -i1 -u4 | pv -q > tmp.zst
rm tmp*
./datagen -s41 -g1GB | ./adapt -i14 -l4 > tmp.zst
rm tmp*
make clean
contrib/adaptive-compression/test-performance.sh
Executable
+59
View File
@@ -0,0 +1,59 @@
echo "testing time -- no limits set"
./datagen -s1 -g1GB > tmp
time ./adapt -otmp1.zst tmp
time zstd -1 -o tmp2.zst tmp
rm tmp*
./datagen -s2 -g2GB > tmp
time ./adapt -otmp1.zst tmp
time zstd -1 -o tmp2.zst tmp
rm tmp*
./datagen -s3 -g4GB > tmp
time ./adapt -otmp1.zst tmp
time zstd -1 -o tmp2.zst tmp
rm tmp*
echo -e "\ntesting compression ratio -- no limits set"
./datagen -s4 -g1GB > tmp
time ./adapt -otmp1.zst tmp
time zstd -1 -o tmp2.zst tmp
ls -l tmp1.zst tmp2.zst
rm tmp*
./datagen -s5 -g2GB > tmp
time ./adapt -otmp1.zst tmp
time zstd -1 -o tmp2.zst tmp
ls -l tmp1.zst tmp2.zst
rm tmp*
./datagen -s6 -g4GB > tmp
time ./adapt -otmp1.zst tmp
time zstd -1 -o tmp2.zst tmp
ls -l tmp1.zst tmp2.zst
rm tmp*
echo e "\ntesting performance at various compression levels -- no limits set"
./datagen -s7 -g1GB > tmp
echo "adapt"
time ./adapt -i5 -f tmp -otmp1.zst
echo "zstdcli"
time zstd -5 tmp -o tmp2.zst
ls -l tmp1.zst tmp2.zst
rm tmp*
./datagen -s8 -g1GB > tmp
echo "adapt"
time ./adapt -i10 -f tmp -otmp1.zst
echo "zstdcli"
time zstd -10 tmp -o tmp2.zst
ls -l tmp1.zst tmp2.zst
rm tmp*
./datagen -s9 -g1GB > tmp
echo "adapt"
time ./adapt -i15 -f tmp -otmp1.zst
echo "zstdcli"
time zstd -15 tmp -o tmp2.zst
ls -l tmp1.zst tmp2.zst
rm tmp*
contrib/gen_html/Makefile
+5 -5
View File
@@ -1,11 +1,11 @@
# ##########################################################################
# ################################################################
# Copyright (c) 2016-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# ##########################################################################
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
CFLAGS ?= -O3
CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wno-comment
contrib/gen_html/gen_html.cpp
+3 -3
View File
@@ -2,9 +2,9 @@
* Copyright (c) 2016-present, Przemyslaw Skibinski, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <iostream>
contrib/linux-kernel/0000-cover-letter.patch
+38 -19
View File
@@ -1,7 +1,7 @@
From 0cd63464d182bb9708f8b25f7da3dc8e5ec6b4fa Mon Sep 17 00:00:00 2001
From 308795a7713ca6fcd468b60fba9a2fca99cee6a0 Mon Sep 17 00:00:00 2001
From: Nick Terrell <terrelln@fb.com>
Date: Thu, 20 Jul 2017 13:18:30 -0700
Subject: [PATCH v3 0/4] Add xxhash and zstd modules
Date: Tue, 8 Aug 2017 19:20:25 -0700
Subject: [PATCH v5 0/5] Add xxhash and zstd modules
Hi all,
@@ -16,27 +16,45 @@ Nick Terrell
Changelog:
v1 -> v2:
- Make pointer in lib/xxhash.c:394 non-const (1/4)
- Use div_u64() for division of u64s (2/4)
- Make pointer in lib/xxhash.c:394 non-const (1/5)
- Use div_u64() for division of u64s (2/5)
- Reduce stack usage of ZSTD_compressSequences(), ZSTD_buildSeqTable(),
ZSTD_decompressSequencesLong(), FSE_buildDTable(), FSE_decompress_wksp(),
HUF_writeCTable(), HUF_readStats(), HUF_readCTable(),
HUF_compressWeights(), HUF_readDTableX2(), and HUF_readDTableX4() (2/4)
- No zstd function uses more than 400 B of stack space (2/4)
HUF_compressWeights(), HUF_readDTableX2(), and HUF_readDTableX4() (2/5)
- No zstd function uses more than 400 B of stack space (2/5)
v2 -> v3:
- Work around gcc-7 bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388
(2/4)
- Fix bug in dictionary compression from upstream commit cc1522351f (2/4)
- Port upstream BtrFS commits e1ddce71d6, 389a6cfc2a, and 6acafd1eff (3/4)
- Change default compression level for BtrFS to 3 (3/4)
(2/5)
- Fix bug in dictionary compression from upstream commit cc1522351f (2/5)
- Port upstream BtrFS commits e1ddce71d6, 389a6cfc2a, and 6acafd1eff (3/5)
- Change default compression level for BtrFS to 3 (3/5)
Nick Terrell (4):
v3 -> v4:
- Fix compiler warnings (2/5)
- Add missing includes (3/5)
- Fix minor linter warnings (3/5, 4/5)
- Add crypto patch (5/5)
v4 -> v5:
- Fix rare compression bug from upstream commit 308047eb5d (2/5)
- Fix bug introduced in v3 when working around the gcc-7 bug (2/5)
- Fix ZSTD_DStream initialization code in squashfs (4/5)
- Fix patch documentation for patches written by Sean Purcell (4/5)
Nick Terrell (5):
lib: Add xxhash module
lib: Add zstd modules
btrfs: Add zstd support
squashfs: Add zstd support
crypto: Add zstd support
crypto/Kconfig | 9 +
crypto/Makefile | 1 +
crypto/testmgr.c | 10 +
crypto/testmgr.h | 71 +
crypto/zstd.c | 265 ++++
fs/btrfs/Kconfig | 2 +
fs/btrfs/Makefile | 2 +-
fs/btrfs/compression.c | 1 +
@@ -47,13 +65,13 @@ Nick Terrell (4):
fs/btrfs/props.c | 6 +
fs/btrfs/super.c | 12 +-
fs/btrfs/sysfs.c | 2 +
fs/btrfs/zstd.c | 435 ++++++
fs/btrfs/zstd.c | 432 ++++++
fs/squashfs/Kconfig | 14 +
fs/squashfs/Makefile | 1 +
fs/squashfs/decompressor.c | 7 +
fs/squashfs/decompressor.h | 4 +
fs/squashfs/squashfs_fs.h | 1 +
fs/squashfs/zstd_wrapper.c | 150 ++
fs/squashfs/zstd_wrapper.c | 151 ++
include/linux/xxhash.h | 236 +++
include/linux/zstd.h | 1157 +++++++++++++++
include/uapi/linux/btrfs.h | 8 +-
@@ -62,9 +80,9 @@ Nick Terrell (4):
lib/xxhash.c | 500 +++++++
lib/zstd/Makefile | 18 +
lib/zstd/bitstream.h | 374 +++++
lib/zstd/compress.c | 3479 ++++++++++++++++++++++++++++++++++++++++++++
lib/zstd/decompress.c | 2526 ++++++++++++++++++++++++++++++++
lib/zstd/entropy_common.c | 243 ++++
lib/zstd/compress.c | 3484 ++++++++++++++++++++++++++++++++++++++++++++
lib/zstd/decompress.c | 2528 ++++++++++++++++++++++++++++++++
lib/zstd/entropy_common.c | 243 +++
lib/zstd/error_private.h | 53 +
lib/zstd/fse.h | 575 ++++++++
lib/zstd/fse_compress.c | 795 ++++++++++
@@ -74,9 +92,10 @@ Nick Terrell (4):
lib/zstd/huf_decompress.c | 960 ++++++++++++
lib/zstd/mem.h | 151 ++
lib/zstd/zstd_common.c | 75 +
lib/zstd/zstd_internal.h | 250 ++++
lib/zstd/zstd_internal.h | 263 ++++
lib/zstd/zstd_opt.h | 1014 +++++++++++++
39 files changed, 14382 insertions(+), 12 deletions(-)
44 files changed, 14756 insertions(+), 12 deletions(-)
create mode 100644 crypto/zstd.c
create mode 100644 fs/btrfs/zstd.c
create mode 100644 fs/squashfs/zstd_wrapper.c
create mode 100644 include/linux/xxhash.h
contrib/linux-kernel/0001-lib-Add-xxhash-module.patch
+2 -2
View File
@@ -1,7 +1,7 @@
From fc7f26acbabda35f1c61dfc357dbb207dc8ed23d Mon Sep 17 00:00:00 2001
From a4b1ffb6e89bbccd519f9afa0910635668436105 Mon Sep 17 00:00:00 2001
From: Nick Terrell <terrelln@fb.com>
Date: Mon, 17 Jul 2017 17:07:18 -0700
Subject: [PATCH v3 1/4] lib: Add xxhash module
Subject: [PATCH v5 1/5] lib: Add xxhash module
Adds xxhash kernel module with xxh32 and xxh64 hashes. xxhash is an
extremely fast non-cryptographic hash algorithm for checksumming.
contrib/linux-kernel/0002-lib-Add-zstd-modules.patch
+77 -66
View File
@@ -1,7 +1,7 @@
From 686a6149b98250d66b5951e3ae05e79063e9de98 Mon Sep 17 00:00:00 2001
From 2b29ec569f8438a0307debd29873859ca6d407fc Mon Sep 17 00:00:00 2001
From: Nick Terrell <terrelln@fb.com>
Date: Mon, 17 Jul 2017 17:08:19 -0700
Subject: [PATCH v3 2/4] lib: Add zstd modules
Subject: [PATCH v5 2/5] lib: Add zstd modules
Add zstd compression and decompression kernel modules.
zstd offers a wide varity of compression speed and quality trade-offs.
@@ -114,26 +114,33 @@ v2 -> v3:
- Work around gcc-7 bug https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388
- Fix bug in dictionary compression from upstream commit cc1522351f
include/linux/zstd.h | 1157 +++++++++++++++
v3 -> v4:
- Fix minor compiler warnings
v4 -> v5:
- Fix rare compression bug from upstream commit 308047eb5d
- Fix bug introduced in v3 when working around the gcc-7 bug
include/linux/zstd.h | 1155 +++++++++++++++
lib/Kconfig | 8 +
lib/Makefile | 2 +
lib/zstd/Makefile | 18 +
lib/zstd/bitstream.h | 374 +++++
lib/zstd/compress.c | 3479 +++++++++++++++++++++++++++++++++++++++++++++
lib/zstd/compress.c | 3482 +++++++++++++++++++++++++++++++++++++++++++++
lib/zstd/decompress.c | 2526 ++++++++++++++++++++++++++++++++
lib/zstd/entropy_common.c | 243 ++++
lib/zstd/error_private.h | 53 +
lib/zstd/error_private.h | 51 +
lib/zstd/fse.h | 575 ++++++++
lib/zstd/fse_compress.c | 795 +++++++++++
lib/zstd/fse_decompress.c | 332 +++++
lib/zstd/huf.h | 212 +++
lib/zstd/huf_compress.c | 770 ++++++++++
lib/zstd/huf_decompress.c | 960 +++++++++++++
lib/zstd/mem.h | 151 ++
lib/zstd/zstd_common.c | 75 +
lib/zstd/zstd_internal.h | 250 ++++
lib/zstd/zstd_opt.h | 1014 +++++++++++++
19 files changed, 12994 insertions(+)
lib/zstd/mem.h | 149 ++
lib/zstd/zstd_common.c | 73 +
lib/zstd/zstd_internal.h | 261 ++++
lib/zstd/zstd_opt.h | 1012 +++++++++++++
19 files changed, 12998 insertions(+)
create mode 100644 include/linux/zstd.h
create mode 100644 lib/zstd/Makefile
create mode 100644 lib/zstd/bitstream.h
@@ -154,18 +161,16 @@ v2 -> v3:
diff --git a/include/linux/zstd.h b/include/linux/zstd.h
new file mode 100644
index 0000000..249575e
index 0000000..305efd0
--- /dev/null
+++ b/include/linux/zstd.h
@@ -0,0 +1,1157 @@
@@ -0,0 +1,1155 @@
+/*
+ * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -1753,18 +1758,16 @@ index 0000000..a826b99
+#endif /* BITSTREAM_H_MODULE */
diff --git a/lib/zstd/compress.c b/lib/zstd/compress.c
new file mode 100644
index 0000000..d60ab7d
index 0000000..ff18ae6
--- /dev/null
+++ b/lib/zstd/compress.c
@@ -0,0 +1,3479 @@
@@ -0,0 +1,3482 @@
+/**
+ * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -2342,7 +2345,7 @@ index 0000000..d60ab7d
+ mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
+}
+
+ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, size_t srcSize)
+ZSTD_STATIC size_t ZSTD_compressSequences_internal(ZSTD_CCtx *zc, void *dst, size_t dstCapacity)
+{
+ const int longOffsets = zc->params.cParams.windowLog > STREAM_ACCUMULATOR_MIN;
+ const seqStore_t *seqStorePtr = &(zc->seqStore);
@@ -2395,7 +2398,7 @@ index 0000000..d60ab7d
+ else
+ op[0] = 0xFF, ZSTD_writeLE16(op + 1, (U16)(nbSeq - LONGNBSEQ)), op += 3;
+ if (nbSeq == 0)
+ goto _check_compressibility;
+ return op - ostart;
+
+ /* seqHead : flags for FSE encoding type */
+ seqHead = op++;
@@ -2585,28 +2588,33 @@ index 0000000..d60ab7d
+ op += streamSize;
+ }
+ }
+
+/* check compressibility */
+_check_compressibility:
+ {
+ size_t const minGain = ZSTD_minGain(srcSize);
+ size_t const maxCSize = srcSize - minGain;
+ if ((size_t)(op - ostart) >= maxCSize) {
+ zc->flagStaticHufTable = HUF_repeat_none;
+ return 0;
+ }
+ }
+
+ /* confirm repcodes */
+ {
+ int i;
+ for (i = 0; i < ZSTD_REP_NUM; i++)
+ zc->rep[i] = zc->repToConfirm[i];
+ }
+
+ return op - ostart;
+}
+
+ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, size_t srcSize)
+{
+ size_t const cSize = ZSTD_compressSequences_internal(zc, dst, dstCapacity);
+ size_t const minGain = ZSTD_minGain(srcSize);
+ size_t const maxCSize = srcSize - minGain;
+ /* If the srcSize <= dstCapacity, then there is enough space to write a
+ * raw uncompressed block. Since we ran out of space, the block must not
+ * be compressible, so fall back to a raw uncompressed block.
+ */
+ int const uncompressibleError = cSize == ERROR(dstSize_tooSmall) && srcSize <= dstCapacity;
+ int i;
+
+ if (ZSTD_isError(cSize) && !uncompressibleError)
+ return cSize;
+ if (cSize >= maxCSize || uncompressibleError) {
+ zc->flagStaticHufTable = HUF_repeat_none;
+ return 0;
+ }
+ /* confirm repcodes */
+ for (i = 0; i < ZSTD_REP_NUM; i++)
+ zc->rep[i] = zc->repToConfirm[i];
+ return cSize;
+}
+
+/*! ZSTD_storeSeq() :
+ Store a sequence (literal length, literals, offset code and match length code) into seqStore_t.
+ `offsetCode` : distance to match, or 0 == repCode.
@@ -5238,7 +5246,7 @@ index 0000000..d60ab7d
+MODULE_DESCRIPTION("Zstd Compressor");
diff --git a/lib/zstd/decompress.c b/lib/zstd/decompress.c
new file mode 100644
index 0000000..62449ae
index 0000000..72df4828
--- /dev/null
+++ b/lib/zstd/decompress.c
@@ -0,0 +1,2526 @@
@@ -5248,8 +5256,6 @@ index 0000000..62449ae
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -6242,6 +6248,8 @@ index 0000000..62449ae
+ BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
+ FSE_updateState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
+
+ seq.match = NULL;
+
+ return seq;
+}
+
@@ -8019,18 +8027,16 @@ index 0000000..2b0a643
+}
diff --git a/lib/zstd/error_private.h b/lib/zstd/error_private.h
new file mode 100644
index 0000000..1a60b31
index 0000000..2062ff0
--- /dev/null
+++ b/lib/zstd/error_private.h
@@ -0,0 +1,53 @@
@@ -0,0 +1,51 @@
+/**
+ * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -11758,18 +11764,16 @@ index 0000000..6526482
+}
diff --git a/lib/zstd/mem.h b/lib/zstd/mem.h
new file mode 100644
index 0000000..3a0f34c
index 0000000..42a697b
--- /dev/null
+++ b/lib/zstd/mem.h
@@ -0,0 +1,151 @@
@@ -0,0 +1,149 @@
+/**
+ * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -11915,18 +11919,16 @@ index 0000000..3a0f34c
+#endif /* MEM_H_MODULE */
diff --git a/lib/zstd/zstd_common.c b/lib/zstd/zstd_common.c
new file mode 100644
index 0000000..a282624
index 0000000..e5f06d7
--- /dev/null
+++ b/lib/zstd/zstd_common.c
@@ -0,0 +1,75 @@
@@ -0,0 +1,73 @@
+/**
+ * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -11996,18 +11998,16 @@ index 0000000..a282624
+}
diff --git a/lib/zstd/zstd_internal.h b/lib/zstd/zstd_internal.h
new file mode 100644
index 0000000..f0ba474
index 0000000..a0fb83e
--- /dev/null
+++ b/lib/zstd/zstd_internal.h
@@ -0,0 +1,250 @@
@@ -0,0 +1,261 @@
+/**
+ * Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -12128,7 +12128,7 @@ index 0000000..f0ba474
+/*-*******************************************
+* Shared functions to include for inlining
+*********************************************/
+static void ZSTD_copy8(void *dst, const void *src) {
+ZSTD_STATIC void ZSTD_copy8(void *dst, const void *src) {
+ memcpy(dst, src, 8);
+}
+/*! ZSTD_wildcopy() :
@@ -12136,8 +12136,21 @@ index 0000000..f0ba474
+#define WILDCOPY_OVERLENGTH 8
+ZSTD_STATIC void ZSTD_wildcopy(void *dst, const void *src, ptrdiff_t length)
+{
+ if (length > 0)
+ memcpy(dst, src, length);
+ const BYTE* ip = (const BYTE*)src;
+ BYTE* op = (BYTE*)dst;
+ BYTE* const oend = op + length;
+ /* Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388.
+ * Avoid the bad case where the loop only runs once by handling the
+ * special case separately. This doesn't trigger the bug because it
+ * doesn't involve pointer/integer overflow.
+ */
+ if (length <= 8)
+ return ZSTD_copy8(dst, src);
+ do {
+ ZSTD_copy8(op, ip);
+ op += 8;
+ ip += 8;
+ } while (op < oend);
+}
+
+/*-*******************************************
@@ -12252,18 +12265,16 @@ index 0000000..f0ba474
+#endif /* ZSTD_CCOMMON_H_MODULE */
diff --git a/lib/zstd/zstd_opt.h b/lib/zstd/zstd_opt.h
new file mode 100644
index 0000000..55e1b4c
index 0000000..ecdd725
--- /dev/null
+++ b/lib/zstd/zstd_opt.h
@@ -0,0 +1,1014 @@
@@ -0,0 +1,1012 @@
+/**
+ * Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
+ * All rights reserved.
+ *
+ * This source code is licensed under the BSD-style license found in the
+ * LICENSE file in the root directory of https://github.com/facebook/zstd.
+ * An additional grant of patent rights can be found in the PATENTS file in the
+ * same directory.
+ *
+ * This program is free software; you can redistribute it and/or modify it under
+ * the terms of the GNU General Public License version 2 as published by the
@@ -13271,4 +13282,4 @@ index 0000000..55e1b4c
+
+#endif /* ZSTD_OPT_H_91842398743 */
--
2.9.3
2.9.5
contrib/linux-kernel/0003-btrfs-Add-zstd-support.patch
+20 -19
View File
@@ -1,7 +1,7 @@
From b0ef8fc63c9ca251ceca632f53aa1de8f1f17772 Mon Sep 17 00:00:00 2001
From 8a9dddfbf6551afea73911e367dd4be64d62b9fd Mon Sep 17 00:00:00 2001
From: Nick Terrell <terrelln@fb.com>
Date: Mon, 17 Jul 2017 17:08:39 -0700
Subject: [PATCH v3 3/4] btrfs: Add zstd support
Subject: [PATCH v5 3/5] btrfs: Add zstd support
Add zstd compression and decompression support to BtrFS. zstd at its
fastest level compresses almost as well as zlib, while offering much
@@ -67,6 +67,10 @@ v2 -> v3:
- Port upstream BtrFS commits e1ddce71d6, 389a6cfc2a, and 6acafd1eff
- Change default compression level for BtrFS to 3
v3 -> v4:
- Add missing includes, which fixes the aarch64 build
- Fix minor linter warnings
fs/btrfs/Kconfig | 2 +
fs/btrfs/Makefile | 2 +-
fs/btrfs/compression.c | 1 +
@@ -77,9 +81,9 @@ v2 -> v3:
fs/btrfs/props.c | 6 +
fs/btrfs/super.c | 12 +-
fs/btrfs/sysfs.c | 2 +
fs/btrfs/zstd.c | 435 +++++++++++++++++++++++++++++++++++++++++++++
fs/btrfs/zstd.c | 432 +++++++++++++++++++++++++++++++++++++++++++++
include/uapi/linux/btrfs.h | 8 +-
12 files changed, 471 insertions(+), 12 deletions(-)
12 files changed, 468 insertions(+), 12 deletions(-)
create mode 100644 fs/btrfs/zstd.c
diff --git a/fs/btrfs/Kconfig b/fs/btrfs/Kconfig
@@ -277,10 +281,10 @@ index c2d5f35..2b6d37c 100644
BTRFS_FEAT_ATTR_PTR(raid56),
diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c
new file mode 100644
index 0000000..1822068
index 0000000..607ce47
--- /dev/null
+++ b/fs/btrfs/zstd.c
@@ -0,0 +1,435 @@
@@ -0,0 +1,432 @@
+/*
+ * Copyright (c) 2016-present, Facebook, Inc.
+ * All rights reserved.
@@ -293,20 +297,16 @@ index 0000000..1822068
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program; if not, write to the
+ * Free Software Foundation, Inc., 59 Temple Place - Suite 330,
+ * Boston, MA 021110-1307, USA.
+ */
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/init.h>
+#include <linux/err.h>
+#include <linux/sched.h>
+#include <linux/pagemap.h>
+#include <linux/bio.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/mm.h>
+#include <linux/pagemap.h>
+#include <linux/refcount.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/zstd.h>
+#include "compression.h"
+
@@ -316,7 +316,8 @@ index 0000000..1822068
+
+static ZSTD_parameters zstd_get_btrfs_parameters(size_t src_len)
+{
+ ZSTD_parameters params = ZSTD_getParams(ZSTD_BTRFS_DEFAULT_LEVEL, src_len, 0);
+ ZSTD_parameters params = ZSTD_getParams(ZSTD_BTRFS_DEFAULT_LEVEL,
+ src_len, 0);
+
+ if (params.cParams.windowLog > ZSTD_BTRFS_MAX_WINDOWLOG)
+ params.cParams.windowLog = ZSTD_BTRFS_MAX_WINDOWLOG;
contrib/linux-kernel/0004-squashfs-Add-zstd-support.patch
+25 -17
View File
@@ -1,7 +1,7 @@
From 0cd63464d182bb9708f8b25f7da3dc8e5ec6b4fa Mon Sep 17 00:00:00 2001
From: Nick Terrell <terrelln@fb.com>
From 46bf8f6d30d6ddf2446c110f122482b5e5e16933 Mon Sep 17 00:00:00 2001
From: Sean Purcell <me@seanp.xyz>
Date: Mon, 17 Jul 2017 17:08:59 -0700
Subject: [PATCH v3 4/4] squashfs: Add zstd support
Subject: [PATCH v5 4/5] squashfs: Add zstd support
Add zstd compression and decompression support to SquashFS. zstd is a
great fit for SquashFS because it can compress at ratios approaching xz,
@@ -42,16 +42,23 @@ taking over the submission process.
zstd source repository: https://github.com/facebook/zstd
Cc: Sean Purcell <me@seanp.xyz>
Signed-off-by: Sean Purcell <me@seanp.xyz>
Signed-off-by: Nick Terrell <terrelln@fb.com>
---
v3 -> v4:
- Fix minor linter warnings
v4 -> v5:
- Fix ZSTD_DStream initialization code in squashfs
- Fix patch documentation to reflect that Sean Purcell is the author
fs/squashfs/Kconfig | 14 +++++
fs/squashfs/Makefile | 1 +
fs/squashfs/decompressor.c | 7 +++
fs/squashfs/decompressor.h | 4 ++
fs/squashfs/squashfs_fs.h | 1 +
fs/squashfs/zstd_wrapper.c | 150 +++++++++++++++++++++++++++++++++++++++++++++
6 files changed, 177 insertions(+)
fs/squashfs/zstd_wrapper.c | 151 +++++++++++++++++++++++++++++++++++++++++++++
6 files changed, 178 insertions(+)
create mode 100644 fs/squashfs/zstd_wrapper.c
diff --git a/fs/squashfs/Kconfig b/fs/squashfs/Kconfig
@@ -140,10 +147,10 @@ index 506f4ba..24d12fd 100644
__le32 s_magic;
diff --git a/fs/squashfs/zstd_wrapper.c b/fs/squashfs/zstd_wrapper.c
new file mode 100644
index 0000000..8cb7c76
index 0000000..eeaabf8
--- /dev/null
+++ b/fs/squashfs/zstd_wrapper.c
@@ -0,0 +1,150 @@
@@ -0,0 +1,151 @@
+/*
+ * Squashfs - a compressed read only filesystem for Linux
+ *
@@ -160,10 +167,6 @@ index 0000000..8cb7c76
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
+ *
+ * zstd_wrapper.c
+ */
+
@@ -182,15 +185,18 @@ index 0000000..8cb7c76
+struct workspace {
+ void *mem;
+ size_t mem_size;
+ size_t window_size;
+};
+
+static void *zstd_init(struct squashfs_sb_info *msblk, void *buff)
+{
+ struct workspace *wksp = kmalloc(sizeof(*wksp), GFP_KERNEL);
+
+ if (wksp == NULL)
+ goto failed;
+ wksp->mem_size = ZSTD_DStreamWorkspaceBound(max_t(size_t,
+ msblk->block_size, SQUASHFS_METADATA_SIZE));
+ wksp->window_size = max_t(size_t,
+ msblk->block_size, SQUASHFS_METADATA_SIZE);
+ wksp->mem_size = ZSTD_DStreamWorkspaceBound(wksp->window_size);
+ wksp->mem = vmalloc(wksp->mem_size);
+ if (wksp->mem == NULL)
+ goto failed;
@@ -226,7 +232,7 @@ index 0000000..8cb7c76
+ ZSTD_inBuffer in_buf = { NULL, 0, 0 };
+ ZSTD_outBuffer out_buf = { NULL, 0, 0 };
+
+ stream = ZSTD_initDStream(wksp->mem_size, wksp->mem, wksp->mem_size);
+ stream = ZSTD_initDStream(wksp->window_size, wksp->mem, wksp->mem_size);
+
+ if (!stream) {
+ ERROR("Failed to initialize zstd decompressor\n");
@@ -239,6 +245,7 @@ index 0000000..8cb7c76
+ do {
+ if (in_buf.pos == in_buf.size && k < b) {
+ int avail = min(length, msblk->devblksize - offset);
+
+ length -= avail;
+ in_buf.src = bh[k]->b_data + offset;
+ in_buf.size = avail;
@@ -249,8 +256,9 @@ index 0000000..8cb7c76
+ if (out_buf.pos == out_buf.size) {
+ out_buf.dst = squashfs_next_page(output);
+ if (out_buf.dst == NULL) {
+ /* shouldn't run out of pages before stream is
+ * done */
+ /* Shouldn't run out of pages
+ * before stream is done.
+ */
+ squashfs_finish_page(output);
+ goto out;
+ }
contrib/linux-kernel/0005-crypto-Add-zstd-support.patch
+424
View File
@@ -0,0 +1,424 @@
From 308795a7713ca6fcd468b60fba9a2fca99cee6a0 Mon Sep 17 00:00:00 2001
From: Nick Terrell <terrelln@fb.com>
Date: Wed, 2 Aug 2017 18:02:13 -0700
Subject: [PATCH v5 5/5] crypto: Add zstd support
Adds zstd support to crypto and scompress. Only supports the default
level.
Signed-off-by: Nick Terrell <terrelln@fb.com>
---
crypto/Kconfig | 9 ++
crypto/Makefile | 1 +
crypto/testmgr.c | 10 +++
crypto/testmgr.h | 71 +++++++++++++++
crypto/zstd.c | 265 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
5 files changed, 356 insertions(+)
create mode 100644 crypto/zstd.c
diff --git a/crypto/Kconfig b/crypto/Kconfig
index caa770e..4fc3936 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1662,6 +1662,15 @@ config CRYPTO_LZ4HC
help
This is the LZ4 high compression mode algorithm.
+config CRYPTO_ZSTD
+ tristate "Zstd compression algorithm"
+ select CRYPTO_ALGAPI
+ select CRYPTO_ACOMP2
+ select ZSTD_COMPRESS
+ select ZSTD_DECOMPRESS
+ help
+ This is the zstd algorithm.
+
comment "Random Number Generation"
config CRYPTO_ANSI_CPRNG
diff --git a/crypto/Makefile b/crypto/Makefile
index d41f033..b22e1e8 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -133,6 +133,7 @@ obj-$(CONFIG_CRYPTO_USER_API_HASH) += algif_hash.o
obj-$(CONFIG_CRYPTO_USER_API_SKCIPHER) += algif_skcipher.o
obj-$(CONFIG_CRYPTO_USER_API_RNG) += algif_rng.o
obj-$(CONFIG_CRYPTO_USER_API_AEAD) += algif_aead.o
+obj-$(CONFIG_CRYPTO_ZSTD) += zstd.o
ecdh_generic-y := ecc.o
ecdh_generic-y += ecdh.o
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 7125ba3..8a124d3 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -3603,6 +3603,16 @@ static const struct alg_test_desc alg_test_descs[] = {
.decomp = __VECS(zlib_deflate_decomp_tv_template)
}
}
+ }, {
+ .alg = "zstd",
+ .test = alg_test_comp,
+ .fips_allowed = 1,
+ .suite = {
+ .comp = {
+ .comp = __VECS(zstd_comp_tv_template),
+ .decomp = __VECS(zstd_decomp_tv_template)
+ }
+ }
}
};
diff --git a/crypto/testmgr.h b/crypto/testmgr.h
index 6ceb0e2..e6b5920 100644
--- a/crypto/testmgr.h
+++ b/crypto/testmgr.h
@@ -34631,4 +34631,75 @@ static const struct comp_testvec lz4hc_decomp_tv_template[] = {
},
};
+static const struct comp_testvec zstd_comp_tv_template[] = {
+ {
+ .inlen = 68,
+ .outlen = 39,
+ .input = "The algorithm is zstd. "
+ "The algorithm is zstd. "
+ "The algorithm is zstd.",
+ .output = "\x28\xb5\x2f\xfd\x00\x50\xf5\x00\x00\xb8\x54\x68\x65"
+ "\x20\x61\x6c\x67\x6f\x72\x69\x74\x68\x6d\x20\x69\x73"
+ "\x20\x7a\x73\x74\x64\x2e\x20\x01\x00\x55\x73\x36\x01"
+ ,
+ },
+ {
+ .inlen = 244,
+ .outlen = 151,
+ .input = "zstd, short for Zstandard, is a fast lossless "
+ "compression algorithm, targeting real-time "
+ "compression scenarios at zlib-level and better "
+ "compression ratios. The zstd compression library "
+ "provides in-memory compression and decompression "
+ "functions.",
+ .output = "\x28\xb5\x2f\xfd\x00\x50\x75\x04\x00\x42\x4b\x1e\x17"
+ "\x90\x81\x31\x00\xf2\x2f\xe4\x36\xc9\xef\x92\x88\x32"
+ "\xc9\xf2\x24\x94\xd8\x68\x9a\x0f\x00\x0c\xc4\x31\x6f"
+ "\x0d\x0c\x38\xac\x5c\x48\x03\xcd\x63\x67\xc0\xf3\xad"
+ "\x4e\x90\xaa\x78\xa0\xa4\xc5\x99\xda\x2f\xb6\x24\x60"
+ "\xe2\x79\x4b\xaa\xb6\x6b\x85\x0b\xc9\xc6\x04\x66\x86"
+ "\xe2\xcc\xe2\x25\x3f\x4f\x09\xcd\xb8\x9d\xdb\xc1\x90"
+ "\xa9\x11\xbc\x35\x44\x69\x2d\x9c\x64\x4f\x13\x31\x64"
+ "\xcc\xfb\x4d\x95\x93\x86\x7f\x33\x7f\x1a\xef\xe9\x30"
+ "\xf9\x67\xa1\x94\x0a\x69\x0f\x60\xcd\xc3\xab\x99\xdc"
+ "\x42\xed\x97\x05\x00\x33\xc3\x15\x95\x3a\x06\xa0\x0e"
+ "\x20\xa9\x0e\x82\xb9\x43\x45\x01",
+ },
+};
+
+static const struct comp_testvec zstd_decomp_tv_template[] = {
+ {
+ .inlen = 43,
+ .outlen = 68,
+ .input = "\x28\xb5\x2f\xfd\x04\x50\xf5\x00\x00\xb8\x54\x68\x65"
+ "\x20\x61\x6c\x67\x6f\x72\x69\x74\x68\x6d\x20\x69\x73"
+ "\x20\x7a\x73\x74\x64\x2e\x20\x01\x00\x55\x73\x36\x01"
+ "\x6b\xf4\x13\x35",
+ .output = "The algorithm is zstd. "
+ "The algorithm is zstd. "
+ "The algorithm is zstd.",
+ },
+ {
+ .inlen = 155,
+ .outlen = 244,
+ .input = "\x28\xb5\x2f\xfd\x04\x50\x75\x04\x00\x42\x4b\x1e\x17"
+ "\x90\x81\x31\x00\xf2\x2f\xe4\x36\xc9\xef\x92\x88\x32"
+ "\xc9\xf2\x24\x94\xd8\x68\x9a\x0f\x00\x0c\xc4\x31\x6f"
+ "\x0d\x0c\x38\xac\x5c\x48\x03\xcd\x63\x67\xc0\xf3\xad"
+ "\x4e\x90\xaa\x78\xa0\xa4\xc5\x99\xda\x2f\xb6\x24\x60"
+ "\xe2\x79\x4b\xaa\xb6\x6b\x85\x0b\xc9\xc6\x04\x66\x86"
+ "\xe2\xcc\xe2\x25\x3f\x4f\x09\xcd\xb8\x9d\xdb\xc1\x90"
+ "\xa9\x11\xbc\x35\x44\x69\x2d\x9c\x64\x4f\x13\x31\x64"
+ "\xcc\xfb\x4d\x95\x93\x86\x7f\x33\x7f\x1a\xef\xe9\x30"
+ "\xf9\x67\xa1\x94\x0a\x69\x0f\x60\xcd\xc3\xab\x99\xdc"
+ "\x42\xed\x97\x05\x00\x33\xc3\x15\x95\x3a\x06\xa0\x0e"
+ "\x20\xa9\x0e\x82\xb9\x43\x45\x01\xaa\x6d\xda\x0d",
+ .output = "zstd, short for Zstandard, is a fast lossless "
+ "compression algorithm, targeting real-time "
+ "compression scenarios at zlib-level and better "
+ "compression ratios. The zstd compression library "
+ "provides in-memory compression and decompression "
+ "functions.",
+ },
+};
#endif /* _CRYPTO_TESTMGR_H */
diff --git a/crypto/zstd.c b/crypto/zstd.c
new file mode 100644
index 0000000..9a76b3e
--- /dev/null
+++ b/crypto/zstd.c
@@ -0,0 +1,265 @@
+/*
+ * Cryptographic API.
+ *
+ * Copyright (c) 2017-present, Facebook, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published by
+ * the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+#include <linux/crypto.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/mm.h>
+#include <linux/module.h>
+#include <linux/net.h>
+#include <linux/vmalloc.h>
+#include <linux/zstd.h>
+#include <crypto/internal/scompress.h>
+
+
+#define ZSTD_DEF_LEVEL 3
+
+struct zstd_ctx {
+ ZSTD_CCtx *cctx;
+ ZSTD_DCtx *dctx;
+ void *cwksp;
+ void *dwksp;
+};
+
+static ZSTD_parameters zstd_params(void)
+{
+ return ZSTD_getParams(ZSTD_DEF_LEVEL, 0, 0);
+}
+
+static int zstd_comp_init(struct zstd_ctx *ctx)
+{
+ int ret = 0;
+ const ZSTD_parameters params = zstd_params();
+ const size_t wksp_size = ZSTD_CCtxWorkspaceBound(params.cParams);
+
+ ctx->cwksp = vzalloc(wksp_size);
+ if (!ctx->cwksp) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ctx->cctx = ZSTD_initCCtx(ctx->cwksp, wksp_size);
+ if (!ctx->cctx) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+out:
+ return ret;
+out_free:
+ vfree(ctx->cwksp);
+ goto out;
+}
+
+static int zstd_decomp_init(struct zstd_ctx *ctx)
+{
+ int ret = 0;
+ const size_t wksp_size = ZSTD_DCtxWorkspaceBound();
+
+ ctx->dwksp = vzalloc(wksp_size);
+ if (!ctx->dwksp) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ ctx->dctx = ZSTD_initDCtx(ctx->dwksp, wksp_size);
+ if (!ctx->dctx) {
+ ret = -EINVAL;
+ goto out_free;
+ }
+out:
+ return ret;
+out_free:
+ vfree(ctx->dwksp);
+ goto out;
+}
+
+static void zstd_comp_exit(struct zstd_ctx *ctx)
+{
+ vfree(ctx->cwksp);
+ ctx->cwksp = NULL;
+ ctx->cctx = NULL;
+}
+
+static void zstd_decomp_exit(struct zstd_ctx *ctx)
+{
+ vfree(ctx->dwksp);
+ ctx->dwksp = NULL;
+ ctx->dctx = NULL;
+}
+
+static int __zstd_init(void *ctx)
+{
+ int ret;
+
+ ret = zstd_comp_init(ctx);
+ if (ret)
+ return ret;
+ ret = zstd_decomp_init(ctx);
+ if (ret)
+ zstd_comp_exit(ctx);
+ return ret;
+}
+
+static void *zstd_alloc_ctx(struct crypto_scomp *tfm)
+{
+ int ret;
+ struct zstd_ctx *ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return ERR_PTR(-ENOMEM);
+
+ ret = __zstd_init(ctx);
+ if (ret) {
+ kfree(ctx);
+ return ERR_PTR(ret);
+ }
+
+ return ctx;
+}
+
+static int zstd_init(struct crypto_tfm *tfm)
+{
+ struct zstd_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return __zstd_init(ctx);
+}
+
+static void __zstd_exit(void *ctx)
+{
+ zstd_comp_exit(ctx);
+ zstd_decomp_exit(ctx);
+}
+
+static void zstd_free_ctx(struct crypto_scomp *tfm, void *ctx)
+{
+ __zstd_exit(ctx);
+ kzfree(ctx);
+}
+
+static void zstd_exit(struct crypto_tfm *tfm)
+{
+ struct zstd_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ __zstd_exit(ctx);
+}
+
+static int __zstd_compress(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
+{
+ size_t out_len;
+ struct zstd_ctx *zctx = ctx;
+ const ZSTD_parameters params = zstd_params();
+
+ out_len = ZSTD_compressCCtx(zctx->cctx, dst, *dlen, src, slen, params);
+ if (ZSTD_isError(out_len))
+ return -EINVAL;
+ *dlen = out_len;
+ return 0;
+}
+
+static int zstd_compress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ struct zstd_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return __zstd_compress(src, slen, dst, dlen, ctx);
+}
+
+static int zstd_scompress(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return __zstd_compress(src, slen, dst, dlen, ctx);
+}
+
+static int __zstd_decompress(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
+{
+ size_t out_len;
+ struct zstd_ctx *zctx = ctx;
+
+ out_len = ZSTD_decompressDCtx(zctx->dctx, dst, *dlen, src, slen);
+ if (ZSTD_isError(out_len))
+ return -EINVAL;
+ *dlen = out_len;
+ return 0;
+}
+
+static int zstd_decompress(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ struct zstd_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return __zstd_decompress(src, slen, dst, dlen, ctx);
+}
+
+static int zstd_sdecompress(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return __zstd_decompress(src, slen, dst, dlen, ctx);
+}
+
+static struct crypto_alg alg = {
+ .cra_name = "zstd",
+ .cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
+ .cra_ctxsize = sizeof(struct zstd_ctx),
+ .cra_module = THIS_MODULE,
+ .cra_init = zstd_init,
+ .cra_exit = zstd_exit,
+ .cra_u = { .compress = {
+ .coa_compress = zstd_compress,
+ .coa_decompress = zstd_decompress } }
+};
+
+static struct scomp_alg scomp = {
+ .alloc_ctx = zstd_alloc_ctx,
+ .free_ctx = zstd_free_ctx,
+ .compress = zstd_scompress,
+ .decompress = zstd_sdecompress,
+ .base = {
+ .cra_name = "zstd",
+ .cra_driver_name = "zstd-scomp",
+ .cra_module = THIS_MODULE,
+ }
+};
+
+static int __init zstd_mod_init(void)
+{
+ int ret;
+
+ ret = crypto_register_alg(&alg);
+ if (ret)
+ return ret;
+
+ ret = crypto_register_scomp(&scomp);
+ if (ret)
+ crypto_unregister_alg(&alg);
+
+ return ret;
+}
+
+static void __exit zstd_mod_fini(void)
+{
+ crypto_unregister_alg(&alg);
+ crypto_unregister_scomp(&scomp);
+}
+
+module_init(zstd_mod_init);
+module_exit(zstd_mod_fini);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Zstd Compression Algorithm");
+MODULE_ALIAS_CRYPTO("zstd");
--
2.9.3
contrib/linux-kernel/0006-squashfs-tools-Add-zstd-support.patch
+420
View File
@@ -0,0 +1,420 @@
From 57a3cf95b276946559f9e044c7352c11303bb9c1 Mon Sep 17 00:00:00 2001
From: Sean Purcell <me@seanp.xyz>
Date: Thu, 3 Aug 2017 17:47:03 -0700
Subject: [PATCH v6] squashfs-tools: Add zstd support
This patch adds zstd support to squashfs-tools. It works with zstd
versions >= 1.0.0. It was originally written by Sean Purcell.
Signed-off-by: Sean Purcell <me@seanp.xyz>
Signed-off-by: Nick Terrell <terrelln@fb.com>
---
v4 -> v5:
- Fix patch documentation to reflect that Sean Purcell is the author
- Don't strip trailing whitespace of unreleated code
- Make zstd_display_options() static
v5 -> v6:
- Fix build instructions in Makefile
squashfs-tools/Makefile | 20 ++++
squashfs-tools/compressor.c | 8 ++
squashfs-tools/squashfs_fs.h | 1 +
squashfs-tools/zstd_wrapper.c | 254 ++++++++++++++++++++++++++++++++++++++++++
squashfs-tools/zstd_wrapper.h | 48 ++++++++
5 files changed, 331 insertions(+)
create mode 100644 squashfs-tools/zstd_wrapper.c
create mode 100644 squashfs-tools/zstd_wrapper.h
diff --git a/squashfs-tools/Makefile b/squashfs-tools/Makefile
index 52d2582..22fc559 100644
--- a/squashfs-tools/Makefile
+++ b/squashfs-tools/Makefile
@@ -75,6 +75,18 @@ GZIP_SUPPORT = 1
#LZMA_SUPPORT = 1
#LZMA_DIR = ../../../../LZMA/lzma465
+
+########### Building ZSTD support ############
+#
+# The ZSTD library is supported
+# ZSTD homepage: http://zstd.net
+# ZSTD source repository: https://github.com/facebook/zstd
+#
+# To build using the ZSTD library - install the library and uncomment the
+# ZSTD_SUPPORT line below.
+#
+#ZSTD_SUPPORT = 1
+
######## Specifying default compression ########
#
# The next line specifies which compression algorithm is used by default
@@ -177,6 +189,14 @@ LIBS += -llz4
COMPRESSORS += lz4
endif
+ifeq ($(ZSTD_SUPPORT),1)
+CFLAGS += -DZSTD_SUPPORT
+MKSQUASHFS_OBJS += zstd_wrapper.o
+UNSQUASHFS_OBJS += zstd_wrapper.o
+LIBS += -lzstd
+COMPRESSORS += zstd
+endif
+
ifeq ($(XATTR_SUPPORT),1)
ifeq ($(XATTR_DEFAULT),1)
CFLAGS += -DXATTR_SUPPORT -DXATTR_DEFAULT
diff --git a/squashfs-tools/compressor.c b/squashfs-tools/compressor.c
index 525e316..02b5e90 100644
--- a/squashfs-tools/compressor.c
+++ b/squashfs-tools/compressor.c
@@ -65,6 +65,13 @@ static struct compressor xz_comp_ops = {
extern struct compressor xz_comp_ops;
#endif
+#ifndef ZSTD_SUPPORT
+static struct compressor zstd_comp_ops = {
+ ZSTD_COMPRESSION, "zstd"
+};
+#else
+extern struct compressor zstd_comp_ops;
+#endif
static struct compressor unknown_comp_ops = {
0, "unknown"
@@ -77,6 +84,7 @@ struct compressor *compressor[] = {
&lzo_comp_ops,
&lz4_comp_ops,
&xz_comp_ops,
+ &zstd_comp_ops,
&unknown_comp_ops
};
diff --git a/squashfs-tools/squashfs_fs.h b/squashfs-tools/squashfs_fs.h
index 791fe12..afca918 100644
--- a/squashfs-tools/squashfs_fs.h
+++ b/squashfs-tools/squashfs_fs.h
@@ -277,6 +277,7 @@ typedef long long squashfs_inode;
#define LZO_COMPRESSION 3
#define XZ_COMPRESSION 4
#define LZ4_COMPRESSION 5
+#define ZSTD_COMPRESSION 6
struct squashfs_super_block {
unsigned int s_magic;
diff --git a/squashfs-tools/zstd_wrapper.c b/squashfs-tools/zstd_wrapper.c
new file mode 100644
index 0000000..dcab75a
--- /dev/null
+++ b/squashfs-tools/zstd_wrapper.c
@@ -0,0 +1,254 @@
+/*
+ * Copyright (c) 2017
+ * Phillip Lougher <phillip@squashfs.org.uk>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * zstd_wrapper.c
+ *
+ * Support for ZSTD compression http://zstd.net
+ */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <zstd.h>
+#include <zstd_errors.h>
+
+#include "squashfs_fs.h"
+#include "zstd_wrapper.h"
+#include "compressor.h"
+
+static int compression_level = ZSTD_DEFAULT_COMPRESSION_LEVEL;
+
+/*
+ * This function is called by the options parsing code in mksquashfs.c
+ * to parse any -X compressor option.
+ *
+ * This function returns:
+ * >=0 (number of additional args parsed) on success
+ * -1 if the option was unrecognised, or
+ * -2 if the option was recognised, but otherwise bad in
+ * some way (e.g. invalid parameter)
+ *
+ * Note: this function sets internal compressor state, but does not
+ * pass back the results of the parsing other than success/failure.
+ * The zstd_dump_options() function is called later to get the options in
+ * a format suitable for writing to the filesystem.
+ */
+static int zstd_options(char *argv[], int argc)
+{
+ if (strcmp(argv[0], "-Xcompression-level") == 0) {
+ if (argc < 2) {
+ fprintf(stderr, "zstd: -Xcompression-level missing "
+ "compression level\n");
+ fprintf(stderr, "zstd: -Xcompression-level it should "
+ "be 1 <= n <= %d\n", ZSTD_maxCLevel());
+ goto failed;
+ }
+
+ compression_level = atoi(argv[1]);
+ if (compression_level < 1 ||
+ compression_level > ZSTD_maxCLevel()) {
+ fprintf(stderr, "zstd: -Xcompression-level invalid, it "
+ "should be 1 <= n <= %d\n", ZSTD_maxCLevel());
+ goto failed;
+ }
+
+ return 1;
+ }
+
+ return -1;
+failed:
+ return -2;
+}
+
+/*
+ * This function is called by mksquashfs to dump the parsed
+ * compressor options in a format suitable for writing to the
+ * compressor options field in the filesystem (stored immediately
+ * after the superblock).
+ *
+ * This function returns a pointer to the compression options structure
+ * to be stored (and the size), or NULL if there are no compression
+ * options.
+ */
+static void *zstd_dump_options(int block_size, int *size)
+{
+ static struct zstd_comp_opts comp_opts;
+
+ /* don't return anything if the options are all default */
+ if (compression_level == ZSTD_DEFAULT_COMPRESSION_LEVEL)
+ return NULL;
+
+ comp_opts.compression_level = compression_level;
+
+ SQUASHFS_INSWAP_COMP_OPTS(&comp_opts);
+
+ *size = sizeof(comp_opts);
+ return &comp_opts;
+}
+
+/*
+ * This function is a helper specifically for the append mode of
+ * mksquashfs. Its purpose is to set the internal compressor state
+ * to the stored compressor options in the passed compressor options
+ * structure.
+ *
+ * In effect this function sets up the compressor options
+ * to the same state they were when the filesystem was originally
+ * generated, this is to ensure on appending, the compressor uses
+ * the same compression options that were used to generate the
+ * original filesystem.
+ *
+ * Note, even if there are no compressor options, this function is still
+ * called with an empty compressor structure (size == 0), to explicitly
+ * set the default options, this is to ensure any user supplied
+ * -X options on the appending mksquashfs command line are over-ridden.
+ *
+ * This function returns 0 on sucessful extraction of options, and -1 on error.
+ */
+static int zstd_extract_options(int block_size, void *buffer, int size)
+{
+ struct zstd_comp_opts *comp_opts = buffer;
+
+ if (size == 0) {
+ /* Set default values */
+ compression_level = ZSTD_DEFAULT_COMPRESSION_LEVEL;
+ return 0;
+ }
+
+ /* we expect a comp_opts structure of sufficient size to be present */
+ if (size < sizeof(*comp_opts))
+ goto failed;
+
+ SQUASHFS_INSWAP_COMP_OPTS(comp_opts);
+
+ if (comp_opts->compression_level < 1 ||
+ comp_opts->compression_level > ZSTD_maxCLevel()) {
+ fprintf(stderr, "zstd: bad compression level in compression "
+ "options structure\n");
+ goto failed;
+ }
+
+ compression_level = comp_opts->compression_level;
+
+ return 0;
+
+failed:
+ fprintf(stderr, "zstd: error reading stored compressor options from "
+ "filesystem!\n");
+
+ return -1;
+}
+
+static void zstd_display_options(void *buffer, int size)
+{
+ struct zstd_comp_opts *comp_opts = buffer;
+
+ /* we expect a comp_opts structure of sufficient size to be present */
+ if (size < sizeof(*comp_opts))
+ goto failed;
+
+ SQUASHFS_INSWAP_COMP_OPTS(comp_opts);
+
+ if (comp_opts->compression_level < 1 ||
+ comp_opts->compression_level > ZSTD_maxCLevel()) {
+ fprintf(stderr, "zstd: bad compression level in compression "
+ "options structure\n");
+ goto failed;
+ }
+
+ printf("\tcompression-level %d\n", comp_opts->compression_level);
+
+ return;
+
+failed:
+ fprintf(stderr, "zstd: error reading stored compressor options from "
+ "filesystem!\n");
+}
+
+/*
+ * This function is called by mksquashfs to initialise the
+ * compressor, before compress() is called.
+ *
+ * This function returns 0 on success, and -1 on error.
+ */
+static int zstd_init(void **strm, int block_size, int datablock)
+{
+ ZSTD_CCtx *cctx = ZSTD_createCCtx();
+
+ if (!cctx) {
+ fprintf(stderr, "zstd: failed to allocate compression "
+ "context!\n");
+ return -1;
+ }
+
+ *strm = cctx;
+ return 0;
+}
+
+static int zstd_compress(void *strm, void *dest, void *src, int size,
+ int block_size, int *error)
+{
+ const size_t res = ZSTD_compressCCtx((ZSTD_CCtx*)strm, dest, block_size,
+ src, size, compression_level);
+
+ if (ZSTD_isError(res)) {
+ /* FIXME:
+ * zstd does not expose stable error codes. The error enum may
+ * change between versions. Until upstream zstd stablizes the
+ * error codes, we have no way of knowing why the error occurs.
+ * zstd shouldn't fail to compress any input unless there isn't
+ * enough output space. We assume that is the cause and return
+ * the special error code for not enough output space.
+ */
+ return 0;
+ }
+
+ return (int)res;
+}
+
+static int zstd_uncompress(void *dest, void *src, int size, int outsize,
+ int *error)
+{
+ const size_t res = ZSTD_decompress(dest, outsize, src, size);
+
+ if (ZSTD_isError(res)) {
+ fprintf(stderr, "\t%d %d\n", outsize, size);
+
+ *error = (int)ZSTD_getErrorCode(res);
+ return -1;
+ }
+
+ return (int)res;
+}
+
+static void zstd_usage(void)
+{
+ fprintf(stderr, "\t -Xcompression-level <compression-level>\n");
+ fprintf(stderr, "\t\t<compression-level> should be 1 .. %d (default "
+ "%d)\n", ZSTD_maxCLevel(), ZSTD_DEFAULT_COMPRESSION_LEVEL);
+}
+
+struct compressor zstd_comp_ops = {
+ .init = zstd_init,
+ .compress = zstd_compress,
+ .uncompress = zstd_uncompress,
+ .options = zstd_options,
+ .dump_options = zstd_dump_options,
+ .extract_options = zstd_extract_options,
+ .display_options = zstd_display_options,
+ .usage = zstd_usage,
+ .id = ZSTD_COMPRESSION,
+ .name = "zstd",
+ .supported = 1
+};
diff --git a/squashfs-tools/zstd_wrapper.h b/squashfs-tools/zstd_wrapper.h
new file mode 100644
index 0000000..4fbef0a
--- /dev/null
+++ b/squashfs-tools/zstd_wrapper.h
@@ -0,0 +1,48 @@
+#ifndef ZSTD_WRAPPER_H
+#define ZSTD_WRAPPER_H
+/*
+ * Squashfs
+ *
+ * Copyright (c) 2017
+ * Phillip Lougher <phillip@squashfs.org.uk>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2,
+ * or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * zstd_wrapper.h
+ *
+ */
+
+#ifndef linux
+#define __BYTE_ORDER BYTE_ORDER
+#define __BIG_ENDIAN BIG_ENDIAN
+#define __LITTLE_ENDIAN LITTLE_ENDIAN
+#else
+#include <endian.h>
+#endif
+
+#if __BYTE_ORDER == __BIG_ENDIAN
+extern unsigned int inswap_le16(unsigned short);
+extern unsigned int inswap_le32(unsigned int);
+
+#define SQUASHFS_INSWAP_COMP_OPTS(s) { \
+ (s)->compression_level = inswap_le32((s)->compression_level); \
+}
+#else
+#define SQUASHFS_INSWAP_COMP_OPTS(s)
+#endif
+
+/* Default compression */
+#define ZSTD_DEFAULT_COMPRESSION_LEVEL 15
+
+struct zstd_comp_opts {
+ int compression_level;
+};
+#endif
--
2.9.5
contrib/linux-kernel/README.md
+4 -4
View File
@@ -1,7 +1,7 @@
# Linux Kernel Patch
There are four pieces, the `xxhash` kernel module, the `zstd_compress` and `zstd_decompress` kernel modules, the BtrFS patch, and the SquashFS patch.
The patches are based off of the linux kernel master branch (version 4.10).
The patches are based off of the linux kernel master branch.
## xxHash kernel module
@@ -42,7 +42,7 @@ The patches are based off of the linux kernel master branch (version 4.10).
Benchmarks run on a Ubuntu 14.04 with 2 cores and 4 GiB of RAM.
The VM is running on a Macbook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of ram, and a SSD.
The kernel running was built from the master branch with the patch (version 4.10).
The kernel running was built from the master branch with the patch.
The compression benchmark is copying 10 copies of the
unzipped [silesia corpus](http://mattmahoney.net/dc/silesia.html) into a BtrFS
@@ -69,14 +69,14 @@ See `btrfs-benchmark.sh` for details.
* The patch is located in `squashfs.diff`
* Additionally `fs/squashfs/zstd_wrapper.c` is provided as a source for convenience.
* The patch has been tested on a 4.10 kernel.
* The patch has been tested on the master branch of the kernel.
### Benchmarks
Benchmarks run on a Ubuntu 14.04 with 2 cores and 4 GiB of RAM.
The VM is running on a Macbook Pro with a 3.1 GHz Intel Core i7 processor,
16 GB of ram, and a SSD.
The kernel running was built from the master branch with the patch (version 4.10).
The kernel running was built from the master branch with the patch.
The compression benchmark is the file tree from the SquashFS archive found in the
Ubuntu 16.10 desktop image (ubuntu-16.10-desktop-amd64.iso).
contrib/linux-kernel/fs/btrfs/zstd.c
+10 -13
View File
@@ -10,20 +10,16 @@
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
* General Public License for more details.
*
* You should have received a copy of the GNU General Public
* License along with this program; if not, write to the
* Free Software Foundation, Inc., 59 Temple Place - Suite 330,
* Boston, MA 021110-1307, USA.
*/
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
#include <linux/init.h>
#include <linux/err.h>
#include <linux/sched.h>
#include <linux/pagemap.h>
#include <linux/bio.h>
#include <linux/err.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/pagemap.h>
#include <linux/refcount.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/zstd.h>
#include "compression.h"
@@ -33,7 +29,8 @@
static ZSTD_parameters zstd_get_btrfs_parameters(size_t src_len)
{
ZSTD_parameters params = ZSTD_getParams(ZSTD_BTRFS_DEFAULT_LEVEL, src_len, 0);
ZSTD_parameters params = ZSTD_getParams(ZSTD_BTRFS_DEFAULT_LEVEL,
src_len, 0);
if (params.cParams.windowLog > ZSTD_BTRFS_MAX_WINDOWLOG)
params.cParams.windowLog = ZSTD_BTRFS_MAX_WINDOWLOG;
contrib/linux-kernel/fs/squashfs/zstd_wrapper.c
+10 -9
View File
@@ -14,10 +14,6 @@
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*
* zstd_wrapper.c
*/
@@ -36,15 +32,18 @@
struct workspace {
void *mem;
size_t mem_size;
size_t window_size;
};
static void *zstd_init(struct squashfs_sb_info *msblk, void *buff)
{
struct workspace *wksp = kmalloc(sizeof(*wksp), GFP_KERNEL);
if (wksp == NULL)
goto failed;
wksp->mem_size = ZSTD_DStreamWorkspaceBound(max_t(size_t,
msblk->block_size, SQUASHFS_METADATA_SIZE));
wksp->window_size = max_t(size_t,
msblk->block_size, SQUASHFS_METADATA_SIZE);
wksp->mem_size = ZSTD_DStreamWorkspaceBound(wksp->window_size);
wksp->mem = vmalloc(wksp->mem_size);
if (wksp->mem == NULL)
goto failed;
@@ -80,7 +79,7 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm,
ZSTD_inBuffer in_buf = { NULL, 0, 0 };
ZSTD_outBuffer out_buf = { NULL, 0, 0 };
stream = ZSTD_initDStream(wksp->mem_size, wksp->mem, wksp->mem_size);
stream = ZSTD_initDStream(wksp->window_size, wksp->mem, wksp->mem_size);
if (!stream) {
ERROR("Failed to initialize zstd decompressor\n");
@@ -93,6 +92,7 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm,
do {
if (in_buf.pos == in_buf.size && k < b) {
int avail = min(length, msblk->devblksize - offset);
length -= avail;
in_buf.src = bh[k]->b_data + offset;
in_buf.size = avail;
@@ -103,8 +103,9 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm,
if (out_buf.pos == out_buf.size) {
out_buf.dst = squashfs_next_page(output);
if (out_buf.dst == NULL) {
/* shouldn't run out of pages before stream is
* done */
/* Shouldn't run out of pages
* before stream is done.
*/
squashfs_finish_page(output);
goto out;
}
contrib/linux-kernel/include/linux/zstd.h
-2
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
contrib/linux-kernel/lib/zstd/compress.c
+26 -23
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
@@ -583,7 +581,7 @@ void ZSTD_seqToCodes(const seqStore_t *seqStorePtr)
mlCodeTable[seqStorePtr->longLengthPos] = MaxML;
}
ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, size_t srcSize)
ZSTD_STATIC size_t ZSTD_compressSequences_internal(ZSTD_CCtx *zc, void *dst, size_t dstCapacity)
{
const int longOffsets = zc->params.cParams.windowLog > STREAM_ACCUMULATOR_MIN;
const seqStore_t *seqStorePtr = &(zc->seqStore);
@@ -636,7 +634,7 @@ ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCa
else
op[0] = 0xFF, ZSTD_writeLE16(op + 1, (U16)(nbSeq - LONGNBSEQ)), op += 3;
if (nbSeq == 0)
goto _check_compressibility;
return op - ostart;
/* seqHead : flags for FSE encoding type */
seqHead = op++;
@@ -826,28 +824,33 @@ ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCa
op += streamSize;
}
}
/* check compressibility */
_check_compressibility:
{
size_t const minGain = ZSTD_minGain(srcSize);
size_t const maxCSize = srcSize - minGain;
if ((size_t)(op - ostart) >= maxCSize) {
zc->flagStaticHufTable = HUF_repeat_none;
return 0;
}
}
/* confirm repcodes */
{
int i;
for (i = 0; i < ZSTD_REP_NUM; i++)
zc->rep[i] = zc->repToConfirm[i];
}
return op - ostart;
}
ZSTD_STATIC size_t ZSTD_compressSequences(ZSTD_CCtx *zc, void *dst, size_t dstCapacity, size_t srcSize)
{
size_t const cSize = ZSTD_compressSequences_internal(zc, dst, dstCapacity);
size_t const minGain = ZSTD_minGain(srcSize);
size_t const maxCSize = srcSize - minGain;
/* If the srcSize <= dstCapacity, then there is enough space to write a
* raw uncompressed block. Since we ran out of space, the block must not
* be compressible, so fall back to a raw uncompressed block.
*/
int const uncompressibleError = cSize == ERROR(dstSize_tooSmall) && srcSize <= dstCapacity;
int i;
if (ZSTD_isError(cSize) && !uncompressibleError)
return cSize;
if (cSize >= maxCSize || uncompressibleError) {
zc->flagStaticHufTable = HUF_repeat_none;
return 0;
}
/* confirm repcodes */
for (i = 0; i < ZSTD_REP_NUM; i++)
zc->rep[i] = zc->repToConfirm[i];
return cSize;
}
/*! ZSTD_storeSeq() :
Store a sequence (literal length, literals, offset code and match length code) into seqStore_t.
`offsetCode` : distance to match, or 0 == repCode.
contrib/linux-kernel/lib/zstd/decompress.c
+2 -2
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
@@ -998,6 +996,8 @@ static seq_t ZSTD_decodeSequence(seqState_t *seqState)
BIT_reloadDStream(&seqState->DStream); /* <= 18 bits */
FSE_updateState(&seqState->stateOffb, &seqState->DStream); /* <= 8 bits */
seq.match = NULL;
return seq;
}
contrib/linux-kernel/lib/zstd/error_private.h
-2
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
contrib/linux-kernel/lib/zstd/mem.h
-2
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
contrib/linux-kernel/lib/zstd/zstd_common.c
-2
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
contrib/linux-kernel/lib/zstd/zstd_internal.h
+16 -5
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
@@ -126,7 +124,7 @@ static const U32 OF_defaultNormLog = OF_DEFAULTNORMLOG;
/*-*******************************************
* Shared functions to include for inlining
*********************************************/
static void ZSTD_copy8(void *dst, const void *src) {
ZSTD_STATIC void ZSTD_copy8(void *dst, const void *src) {
memcpy(dst, src, 8);
}
/*! ZSTD_wildcopy() :
@@ -134,8 +132,21 @@ static void ZSTD_copy8(void *dst, const void *src) {
#define WILDCOPY_OVERLENGTH 8
ZSTD_STATIC void ZSTD_wildcopy(void *dst, const void *src, ptrdiff_t length)
{
if (length > 0)
memcpy(dst, src, length);
const BYTE* ip = (const BYTE*)src;
BYTE* op = (BYTE*)dst;
BYTE* const oend = op + length;
/* Work around https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81388.
* Avoid the bad case where the loop only runs once by handling the
* special case separately. This doesn't trigger the bug because it
* doesn't involve pointer/integer overflow.
*/
if (length <= 8)
return ZSTD_copy8(dst, src);
do {
ZSTD_copy8(op, ip);
op += 8;
ip += 8;
} while (op < oend);
}
/*-*******************************************
contrib/linux-kernel/lib/zstd/zstd_opt.h
-2
View File
@@ -4,8 +4,6 @@
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of https://github.com/facebook/zstd.
* An additional grant of patent rights can be found in the PATENTS file in the
* same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
contrib/linux-kernel/test/DecompressCrash.c
+3 -3
View File
@@ -2,9 +2,9 @@
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*
contrib/linux-kernel/test/RoundTripCrash.c
+3 -3
View File
@@ -2,9 +2,9 @@
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*
contrib/linux-kernel/xxhash_test.c
+4 -10
View File
@@ -2,19 +2,13 @@
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
* Free Software Foundation. This program is dual-licensed; you may select
* either version 2 of the GNU General Public License ("GPL") or BSD license
* ("BSD").
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* DO_XXH should be 32 or 64 for xxh32 and xxh64 respectively */
#define DO_XXH 0
#define DO_XXH 0
/* DO_CRC should be 0 or 1 */
#define DO_CRC 0
/* Buffer size */
contrib/linux-kernel/zstd_compress_test.c
+3 -9
View File
@@ -2,15 +2,9 @@
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
* Free Software Foundation. This program is dual-licensed; you may select
* either version 2 of the GNU General Public License ("GPL") or BSD license
* ("BSD").
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* Compression level or 0 to disable */
contrib/linux-kernel/zstd_decompress_test.c
+3 -9
View File
@@ -2,15 +2,9 @@
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
*
* This program is free software; you can redistribute it and/or modify it under
* the terms of the GNU General Public License version 2 as published by the
* Free Software Foundation. This program is dual-licensed; you may select
* either version 2 of the GNU General Public License ("GPL") or BSD license
* ("BSD").
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* Compression level or 0 to disable */
contrib/long_distance_matching/Makefile
+36
View File
@@ -0,0 +1,36 @@
# ################################################################
# Copyright (c) 2017-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
# This Makefile presumes libzstd is installed, using `sudo make install`
CPPFLAGS+= -I../../lib/common
CFLAGS ?= -O3
DEBUGFLAGS = -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-Wstrict-prototypes -Wundef -Wpointer-arith -Wformat-security \
-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
-Wredundant-decls
CFLAGS += $(DEBUGFLAGS) $(MOREFLAGS)
FLAGS = $(CPPFLAGS) $(CFLAGS)
LDFLAGS += -lzstd
.PHONY: default all clean
default: all
all: ldm
ldm: ldm_common.c ldm.c main.c
$(CC) $(CPPFLAGS) $(CFLAGS) $^ $(LDFLAGS) -o $@
clean:
@rm -f core *.o tmp* result* *.ldm *.ldm.dec \
ldm
@echo Cleaning completed
contrib/long_distance_matching/README.md
+102
View File
@@ -0,0 +1,102 @@
This is a compression algorithm focused on finding long distance matches.
It is based upon lz4 and uses nearly the same block format (github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md). The number of bytes to encode the offset is four instead of two in lz4 to reflect the longer distance matching. The block format is described in `ldm.h`.
### Build
Run `make`.
### Compressing a file
`ldm <filename>`
Decompression and verification can be enabled by defining `DECOMPRESS_AND_VERIFY` in `main.c`.
The output file names are as follows:
- `<filename>.ldm` : compressed file
- `<filename>.ldm.dec` : decompressed file
### Parameters
There are various parameters that can be tuned. These parameters can be tuned in `ldm.h` or, alternatively if `ldm_params.h` is included, in `ldm_params.h` (for easier configuration).
The parameters are as follows and must all be defined:
- `LDM_MEMORY_USAGE` : the memory usage of the underlying hash table in bytes.
- `HASH_BUCKET_SIZE_LOG` : the log size of each bucket in the hash table (used in collision resolution).
- `LDM_LAG` : the lag (in bytes) in inserting entries into the hash table.
- `LDM_WINDOW_SIZE_LOG` : the log maximum window size when searching for matches.
- `LDM_MIN_MATCH_LENGTH` : the minimum match length.
- `INSERT_BY_TAG` : insert entries into the hash table as a function of the hash. This increases speed by reducing the number of hash table lookups and match comparisons. Certain hashes will never be inserted.
- `USE_CHECKSUM` : store a checksum with the hash table entries for faster comparison. This halves the number of entries the hash table can contain.
The optional parameter `HASH_ONLY_EVERY_LOG` is the log inverse frequency of insertion into the hash table. That is, an entry is inserted approximately every `1 << HASH_ONLY_EVERY_LOG` times. If this parameter is not defined, the value is computed as a function of the window size and memory usage to approximate an even coverage of the window.
### Benchmark
Below is a comparison of various compression methods on a tar of four versions of llvm (versions `3.9.0`, `3.9.1`, `4.0.0`, `4.0.1`) with a total size of `727900160` B.
| Method | Size | Ratio |
|:---|---:|---:|
|lrzip -p 32 -n -w 1 | `369968714` | `1.97`|
|ldm | `209391361` | `3.48`|
|lz4 | `189954338` | `3.83`|
|lrzip -p 32 -l -w 1 | `163940343` | `4.44`|
|zstd -1 | `126080293` | `5.77`|
|lrzip -p 32 -n | `124821009` | `5.83`|
|lrzip -p 32 -n -w 1 & zstd -1 | `120317909` | `6.05`|
|zstd -3 -o | `115290952` | `6.31`|
|lrzip -p 32 -g -L 9 -w 1 | `107168979` | `6.79`|
|zstd -6 -o | `102772098` | `7.08`|
|zstd -T16 -9 | `98040470` | `7.42`|
|lrzip -p 32 -n -w 1 & zstd -T32 -19 | `88050289` | `8.27`|
|zstd -T32 -19 | `83626098` | `8.70`|
|lrzip -p 32 -n & zstd -1 | `36335117` | `20.03`|
|ldm & zstd -6 | `32856232` | `22.15`|
|lrzip -p 32 -g -L 9 | `32243594` | `22.58`|
|lrzip -p 32 -n & zstd -6 | `30954572` | `23.52`|
|lrzip -p 32 -n & zstd -T32 -19 | `26472064` | `27.50`|
The method marked `ldm` was run with the following parameters:
| Parameter | Value |
|:---|---:|
| `LDM_MEMORY_USAGE` | `23`|
|`HASH_BUCKET_SIZE_LOG` | `3`|
|`LDM_LAG` | `0`|
|`LDM_WINDOW_SIZE_LOG` | `28`|
|`LDM_MIN_MATCH_LENGTH`| `64`|
|`INSERT_BY_TAG` | `1`|
|`USE_CHECKSUM` | `1`|
The compression speed was `220.5 MB/s`.
### Parameter selection
Below is a brief discussion of the effects of the parameters on the speed and compression ratio.
#### Speed
A large bottleneck in terms of speed is finding the matches and comparing to see if they are greater than the minimum match length. Generally:
- The fewer matches found (or the lower the percentage of the literals matched), the slower the algorithm will behave.
- Increasing `HASH_ONLY_EVERY_LOG` results in fewer inserts and, if `INSERT_BY_TAG` is set, fewer lookups in the table. This has a large effect on speed, as well as compression ratio.
- If `HASH_ONLY_EVERY_LOG` is not set, its value is calculated based on `LDM_WINDOW_SIZE_LOG` and `LDM_MEMORY_USAGE`. Increasing `LDM_WINDOW_SIZE_LOG` has the effect of increasing `HASH_ONLY_EVERY_LOG` and increasing `LDM_MEMORY_USAGE` decreases `HASH_ONLY_EVERY_LOG`.
- `USE_CHECKSUM` generally improves speed with hash table lookups.
#### Compression ratio
The compression ratio is highly correlated with the coverage of matches. As a long distance matcher, the algorithm was designed to "optimize" for long distance matches outside the zstd compression window. The compression ratio after recompressing the output of the long-distance matcher with zstd was a more important signal in development than the raw compression ratio itself.
Generally, increasing `LDM_MEMORY_USAGE` will improve the compression ratio. However when using the default computed value of `HASH_ONLY_EVERY_LOG`, this increases the frequency of insertion and lookup in the table and thus may result in a decrease in speed.
Below is a table showing the speed and compression ratio when compressing the llvm tar (as described above) using different settings for `LDM_MEMORY_USAGE`. The other parameters were the same as used in the benchmark above.
| `LDM_MEMORY_USAGE` | Ratio | Speed (MB/s) | Ratio after zstd -6 |
|---:| ---: | ---: | ---: |
| `18` | `1.85` | `232.4` | `10.92` |
| `21` | `2.79` | `233.9` | `15.92` |
| `23` | `3.48` | `220.5` | `18.29` |
| `25` | `4.56` | `140.8` | `19.21` |
### Compression statistics
Compression statistics (and the configuration) can be enabled/disabled via `COMPUTE_STATS` and `OUTPUT_CONFIGURATION` in `ldm.h`.
contrib/long_distance_matching/ldm.c
+857
View File
@@ -0,0 +1,857 @@
#include <limits.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "ldm.h"
#define LDM_HASHTABLESIZE (1 << (LDM_MEMORY_USAGE))
#define LDM_HASHTABLESIZE_U32 ((LDM_HASHTABLESIZE) >> 2)
#define LDM_HASHTABLESIZE_U64 ((LDM_HASHTABLESIZE) >> 3)
#if USE_CHECKSUM
#define LDM_HASH_ENTRY_SIZE_LOG 3
#else
#define LDM_HASH_ENTRY_SIZE_LOG 2
#endif
// Entries are inserted into the table HASH_ONLY_EVERY + 1 times "on average".
#ifndef HASH_ONLY_EVERY_LOG
#define HASH_ONLY_EVERY_LOG (LDM_WINDOW_SIZE_LOG-((LDM_MEMORY_USAGE)-(LDM_HASH_ENTRY_SIZE_LOG)))
#endif
#define HASH_ONLY_EVERY ((1 << (HASH_ONLY_EVERY_LOG)) - 1)
#define HASH_BUCKET_SIZE (1 << (HASH_BUCKET_SIZE_LOG))
#define NUM_HASH_BUCKETS_LOG ((LDM_MEMORY_USAGE)-(LDM_HASH_ENTRY_SIZE_LOG)-(HASH_BUCKET_SIZE_LOG))
#define HASH_CHAR_OFFSET 10
// Take the first match in the hash bucket only.
//#define ZSTD_SKIP
static const U64 prime8bytes = 11400714785074694791ULL;
// Type of the small hash used to index into the hash table.
typedef U32 hash_t;
#if USE_CHECKSUM
typedef struct LDM_hashEntry {
U32 offset;
U32 checksum;
} LDM_hashEntry;
#else
typedef struct LDM_hashEntry {
U32 offset;
} LDM_hashEntry;
#endif
struct LDM_compressStats {
U32 windowSizeLog, hashTableSizeLog;
U32 numMatches;
U64 totalMatchLength;
U64 totalLiteralLength;
U64 totalOffset;
U32 matchLengthHistogram[32];
U32 minOffset, maxOffset;
U32 offsetHistogram[32];
};
typedef struct LDM_hashTable LDM_hashTable;
struct LDM_CCtx {
size_t isize; /* Input size */
size_t maxOSize; /* Maximum output size */
const BYTE *ibase; /* Base of input */
const BYTE *ip; /* Current input position */
const BYTE *iend; /* End of input */
// Maximum input position such that hashing at the position does not exceed
// end of input.
const BYTE *ihashLimit;
// Maximum input position such that finding a match of at least the minimum
// match length does not exceed end of input.
const BYTE *imatchLimit;
const BYTE *obase; /* Base of output */
BYTE *op; /* Output */
const BYTE *anchor; /* Anchor to start of current (match) block */
LDM_compressStats stats; /* Compression statistics */
LDM_hashTable *hashTable;
const BYTE *lastPosHashed; /* Last position hashed */
U64 lastHash;
const BYTE *nextIp; // TODO: this is redundant (ip + step)
const BYTE *nextPosHashed;
U64 nextHash;
unsigned step; // ip step, should be 1.
const BYTE *lagIp;
U64 lagHash;
};
struct LDM_hashTable {
U32 numBuckets; // The number of buckets.
U32 numEntries; // numBuckets * HASH_BUCKET_SIZE.
LDM_hashEntry *entries;
BYTE *bucketOffsets; // A pointer (per bucket) to the next insert position.
};
static void HASH_destroyTable(LDM_hashTable *table) {
free(table->entries);
free(table->bucketOffsets);
free(table);
}
/**
* Create a hash table that can contain size elements.
* The number of buckets is determined by size >> HASH_BUCKET_SIZE_LOG.
*
* Returns NULL if table creation failed.
*/
static LDM_hashTable *HASH_createTable(U32 size) {
LDM_hashTable *table = malloc(sizeof(LDM_hashTable));
if (!table) return NULL;
table->numBuckets = size >> HASH_BUCKET_SIZE_LOG;
table->numEntries = size;
table->entries = calloc(size, sizeof(LDM_hashEntry));
table->bucketOffsets = calloc(size >> HASH_BUCKET_SIZE_LOG, sizeof(BYTE));
if (!table->entries || !table->bucketOffsets) {
HASH_destroyTable(table);
return NULL;
}
return table;
}
static LDM_hashEntry *getBucket(const LDM_hashTable *table, const hash_t hash) {
return table->entries + (hash << HASH_BUCKET_SIZE_LOG);
}
static unsigned ZSTD_NbCommonBytes (register size_t val) {
if (MEM_isLittleEndian()) {
if (MEM_64bits()) {
# if defined(_MSC_VER) && defined(_WIN64)
unsigned long r = 0;
_BitScanForward64( &r, (U64)val );
return (unsigned)(r>>3);
# elif defined(__GNUC__) && (__GNUC__ >= 3)
return (__builtin_ctzll((U64)val) >> 3);
# else
static const int DeBruijnBytePos[64] = { 0, 0, 0, 0, 0, 1, 1, 2,
0, 3, 1, 3, 1, 4, 2, 7,
0, 2, 3, 6, 1, 5, 3, 5,
1, 3, 4, 4, 2, 5, 6, 7,
7, 0, 1, 2, 3, 3, 4, 6,
2, 6, 5, 5, 3, 4, 5, 6,
7, 1, 2, 4, 6, 4, 4, 5,
7, 2, 6, 5, 7, 6, 7, 7 };
return DeBruijnBytePos[
((U64)((val & -(long long)val) * 0x0218A392CDABBD3FULL)) >> 58];
# endif
} else { /* 32 bits */
# if defined(_MSC_VER)
unsigned long r=0;
_BitScanForward( &r, (U32)val );
return (unsigned)(r>>3);
# elif defined(__GNUC__) && (__GNUC__ >= 3)
return (__builtin_ctz((U32)val) >> 3);
# else
static const int DeBruijnBytePos[32] = { 0, 0, 3, 0, 3, 1, 3, 0,
3, 2, 2, 1, 3, 2, 0, 1,
3, 3, 1, 2, 2, 2, 2, 0,
3, 1, 2, 0, 1, 0, 1, 1 };
return DeBruijnBytePos[
((U32)((val & -(S32)val) * 0x077CB531U)) >> 27];
# endif
}
} else { /* Big Endian CPU */
if (MEM_64bits()) {
# if defined(_MSC_VER) && defined(_WIN64)
unsigned long r = 0;
_BitScanReverse64( &r, val );
return (unsigned)(r>>3);
# elif defined(__GNUC__) && (__GNUC__ >= 3)
return (__builtin_clzll(val) >> 3);
# else
unsigned r;
/* calculate this way due to compiler complaining in 32-bits mode */
const unsigned n32 = sizeof(size_t)*4;
if (!(val>>n32)) { r=4; } else { r=0; val>>=n32; }
if (!(val>>16)) { r+=2; val>>=8; } else { val>>=24; }
r += (!val);
return r;
# endif
} else { /* 32 bits */
# if defined(_MSC_VER)
unsigned long r = 0;
_BitScanReverse( &r, (unsigned long)val );
return (unsigned)(r>>3);
# elif defined(__GNUC__) && (__GNUC__ >= 3)
return (__builtin_clz((U32)val) >> 3);
# else
unsigned r;
if (!(val>>16)) { r=2; val>>=8; } else { r=0; val>>=24; }
r += (!val);
return r;
# endif
}
}
}
// From lib/compress/zstd_compress.c
static size_t ZSTD_count(const BYTE *pIn, const BYTE *pMatch,
const BYTE *const pInLimit) {
const BYTE * const pStart = pIn;
const BYTE * const pInLoopLimit = pInLimit - (sizeof(size_t)-1);
while (pIn < pInLoopLimit) {
size_t const diff = MEM_readST(pMatch) ^ MEM_readST(pIn);
if (!diff) {
pIn += sizeof(size_t);
pMatch += sizeof(size_t);
continue;
}
pIn += ZSTD_NbCommonBytes(diff);
return (size_t)(pIn - pStart);
}
if (MEM_64bits()) {
if ((pIn < (pInLimit - 3)) && (MEM_read32(pMatch) == MEM_read32(pIn))) {
pIn += 4;
pMatch += 4;
}
}
if ((pIn < (pInLimit - 1)) && (MEM_read16(pMatch) == MEM_read16(pIn))) {
pIn += 2;
pMatch += 2;
}
if ((pIn < pInLimit) && (*pMatch == *pIn)) {
pIn++;
}
return (size_t)(pIn - pStart);
}
/**
* Count number of bytes that match backwards before pIn and pMatch.
*
* We count only bytes where pMatch > pBase and pIn > pAnchor.
*/
static size_t countBackwardsMatch(const BYTE *pIn, const BYTE *pAnchor,
const BYTE *pMatch, const BYTE *pBase) {
size_t matchLength = 0;
while (pIn > pAnchor && pMatch > pBase && pIn[-1] == pMatch[-1]) {
pIn--;
pMatch--;
matchLength++;
}
return matchLength;
}
/**
* Returns a pointer to the entry in the hash table matching the hash and
* checksum with the "longest match length" as defined below. The forward and
* backward match lengths are written to *pForwardMatchLength and
* *pBackwardMatchLength.
*
* The match length is defined based on cctx->ip and the entry's offset.
* The forward match is computed from cctx->ip and entry->offset + cctx->ibase.
* The backward match is computed backwards from cctx->ip and
* cctx->ibase only if the forward match is longer than LDM_MIN_MATCH_LENGTH.
*/
static LDM_hashEntry *HASH_getBestEntry(const LDM_CCtx *cctx,
const hash_t hash,
const U32 checksum,
U64 *pForwardMatchLength,
U64 *pBackwardMatchLength) {
LDM_hashTable *table = cctx->hashTable;
LDM_hashEntry *bucket = getBucket(table, hash);
LDM_hashEntry *cur;
LDM_hashEntry *bestEntry = NULL;
U64 bestMatchLength = 0;
#if !(USE_CHECKSUM)
(void)checksum;
#endif
for (cur = bucket; cur < bucket + HASH_BUCKET_SIZE; ++cur) {
const BYTE *pMatch = cur->offset + cctx->ibase;
// Check checksum for faster check.
#if USE_CHECKSUM
if (cur->checksum == checksum &&
cctx->ip - pMatch <= LDM_WINDOW_SIZE) {
#else
if (cctx->ip - pMatch <= LDM_WINDOW_SIZE) {
#endif
U64 forwardMatchLength = ZSTD_count(cctx->ip, pMatch, cctx->iend);
U64 backwardMatchLength, totalMatchLength;
// Only take matches where the forward match length is large enough
// for speed.
if (forwardMatchLength < LDM_MIN_MATCH_LENGTH) {
continue;
}
backwardMatchLength =
countBackwardsMatch(cctx->ip, cctx->anchor,
cur->offset + cctx->ibase,
cctx->ibase);
totalMatchLength = forwardMatchLength + backwardMatchLength;
if (totalMatchLength >= bestMatchLength) {
bestMatchLength = totalMatchLength;
*pForwardMatchLength = forwardMatchLength;
*pBackwardMatchLength = backwardMatchLength;
bestEntry = cur;
#ifdef ZSTD_SKIP
return cur;
#endif
}
}
}
if (bestEntry != NULL) {
return bestEntry;
}
return NULL;
}
/**
* Insert an entry into the hash table. The table uses a "circular buffer",
* with the oldest entry overwritten.
*/
static void HASH_insert(LDM_hashTable *table,
const hash_t hash, const LDM_hashEntry entry) {
*(getBucket(table, hash) + table->bucketOffsets[hash]) = entry;
table->bucketOffsets[hash]++;
table->bucketOffsets[hash] &= HASH_BUCKET_SIZE - 1;
}
static void HASH_outputTableOccupancy(const LDM_hashTable *table) {
U32 ctr = 0;
LDM_hashEntry *cur = table->entries;
LDM_hashEntry *end = table->entries + (table->numBuckets * HASH_BUCKET_SIZE);
for (; cur < end; ++cur) {
if (cur->offset == 0) {
ctr++;
}
}
// The number of buckets is repeated as a check for now.
printf("Num buckets, bucket size: %d (2^%d), %d\n",
table->numBuckets, NUM_HASH_BUCKETS_LOG, HASH_BUCKET_SIZE);
printf("Hash table size, empty slots, %% empty: %u, %u, %.3f\n",
table->numEntries, ctr,
100.0 * (double)(ctr) / table->numEntries);
}
// TODO: This can be done more efficiently, for example by using builtin
// functions (but it is not that important as it is only used for computing
// stats).
static int intLog2(U64 x) {
int ret = 0;
while (x >>= 1) {
ret++;
}
return ret;
}
void LDM_printCompressStats(const LDM_compressStats *stats) {
printf("=====================\n");
printf("Compression statistics\n");
printf("Window size, hash table size (bytes): 2^%u, 2^%u\n",
stats->windowSizeLog, stats->hashTableSizeLog);
printf("num matches, total match length, %% matched: %u, %llu, %.3f\n",
stats->numMatches,
stats->totalMatchLength,
100.0 * (double)stats->totalMatchLength /
(double)(stats->totalMatchLength + stats->totalLiteralLength));
printf("avg match length: %.1f\n", ((double)stats->totalMatchLength) /
(double)stats->numMatches);
printf("avg literal length, total literalLength: %.1f, %llu\n",
((double)stats->totalLiteralLength) / (double)stats->numMatches,
stats->totalLiteralLength);
printf("avg offset length: %.1f\n",
((double)stats->totalOffset) / (double)stats->numMatches);
printf("min offset, max offset: %u, %u\n",
stats->minOffset, stats->maxOffset);
printf("\n");
printf("offset histogram | match length histogram\n");
printf("offset/ML, num matches, %% of matches | num matches, %% of matches\n");
{
int i;
int logMaxOffset = intLog2(stats->maxOffset);
for (i = 0; i <= logMaxOffset; i++) {
printf("2^%*d: %10u %6.3f%% |2^%*d: %10u %6.3f \n",
2, i,
stats->offsetHistogram[i],
100.0 * (double) stats->offsetHistogram[i] /
(double) stats->numMatches,
2, i,
stats->matchLengthHistogram[i],
100.0 * (double) stats->matchLengthHistogram[i] /
(double) stats->numMatches);
}
}
printf("\n");
printf("=====================\n");
}
/**
* Return the upper (most significant) NUM_HASH_BUCKETS_LOG bits.
*/
static hash_t getSmallHash(U64 hash) {
return hash >> (64 - NUM_HASH_BUCKETS_LOG);
}
/**
* Return the 32 bits after the upper NUM_HASH_BUCKETS_LOG bits.
*/
static U32 getChecksum(U64 hash) {
return (hash >> (64 - 32 - NUM_HASH_BUCKETS_LOG)) & 0xFFFFFFFF;
}
#if INSERT_BY_TAG
static U32 lowerBitsFromHfHash(U64 hash) {
// The number of bits used so far is NUM_HASH_BUCKETS_LOG + 32.
// So there are 32 - NUM_HASH_BUCKETS_LOG bits left.
// Occasional hashing requires HASH_ONLY_EVERY_LOG bits.
// So if 32 - LDMHASHLOG < HASH_ONLY_EVERY_LOG, just return lower bits
// allowing for reuse of bits.
if (32 - NUM_HASH_BUCKETS_LOG < HASH_ONLY_EVERY_LOG) {
return hash & HASH_ONLY_EVERY;
} else {
// Otherwise shift by
// (32 - NUM_HASH_BUCKETS_LOG - HASH_ONLY_EVERY_LOG) bits first.
return (hash >> (32 - NUM_HASH_BUCKETS_LOG - HASH_ONLY_EVERY_LOG)) &
HASH_ONLY_EVERY;
}
}
#endif
/**
* Get a 64-bit hash using the first len bytes from buf.
*
* Giving bytes s = s_1, s_2, ... s_k, the hash is defined to be
* H(s) = s_1*(a^(k-1)) + s_2*(a^(k-2)) + ... + s_k*(a^0)
*
* where the constant a is defined to be prime8bytes.
*
* The implementation adds an offset to each byte, so
* H(s) = (s_1 + HASH_CHAR_OFFSET)*(a^(k-1)) + ...
*/
static U64 getHash(const BYTE *buf, U32 len) {
U64 ret = 0;
U32 i;
for (i = 0; i < len; i++) {
ret *= prime8bytes;
ret += buf[i] + HASH_CHAR_OFFSET;
}
return ret;
}
static U64 ipow(U64 base, U64 exp) {
U64 ret = 1;
while (exp) {
if (exp & 1) {
ret *= base;
}
exp >>= 1;
base *= base;
}
return ret;
}
static U64 updateHash(U64 hash, U32 len,
BYTE toRemove, BYTE toAdd) {
// TODO: this relies on compiler optimization.
// The exponential can be calculated explicitly as len is constant.
hash -= ((toRemove + HASH_CHAR_OFFSET) *
ipow(prime8bytes, len - 1));
hash *= prime8bytes;
hash += toAdd + HASH_CHAR_OFFSET;
return hash;
}
/**
* Update cctx->nextHash and cctx->nextPosHashed
* based on cctx->lastHash and cctx->lastPosHashed.
*
* This uses a rolling hash and requires that the last position hashed
* corresponds to cctx->nextIp - step.
*/
static void setNextHash(LDM_CCtx *cctx) {
cctx->nextHash = updateHash(
cctx->lastHash, LDM_HASH_LENGTH,
cctx->lastPosHashed[0],
cctx->lastPosHashed[LDM_HASH_LENGTH]);
cctx->nextPosHashed = cctx->nextIp;
#if LDM_LAG
if (cctx->ip - cctx->ibase > LDM_LAG) {
cctx->lagHash = updateHash(
cctx->lagHash, LDM_HASH_LENGTH,
cctx->lagIp[0], cctx->lagIp[LDM_HASH_LENGTH]);
cctx->lagIp++;
}
#endif
}
static void putHashOfCurrentPositionFromHash(LDM_CCtx *cctx, U64 hash) {
// Hash only every HASH_ONLY_EVERY times, based on cctx->ip.
// Note: this works only when cctx->step is 1.
#if LDM_LAG
if (cctx -> lagIp - cctx->ibase > 0) {
#if INSERT_BY_TAG
U32 hashEveryMask = lowerBitsFromHfHash(cctx->lagHash);
if (hashEveryMask == HASH_ONLY_EVERY) {
#else
if (((cctx->ip - cctx->ibase) & HASH_ONLY_EVERY) == HASH_ONLY_EVERY) {
#endif
U32 smallHash = getSmallHash(cctx->lagHash);
# if USE_CHECKSUM
U32 checksum = getChecksum(cctx->lagHash);
const LDM_hashEntry entry = { cctx->lagIp - cctx->ibase, checksum };
# else
const LDM_hashEntry entry = { cctx->lagIp - cctx->ibase };
# endif
HASH_insert(cctx->hashTable, smallHash, entry);
}
} else {
#endif // LDM_LAG
#if INSERT_BY_TAG
U32 hashEveryMask = lowerBitsFromHfHash(hash);
if (hashEveryMask == HASH_ONLY_EVERY) {
#else
if (((cctx->ip - cctx->ibase) & HASH_ONLY_EVERY) == HASH_ONLY_EVERY) {
#endif
U32 smallHash = getSmallHash(hash);
#if USE_CHECKSUM
U32 checksum = getChecksum(hash);
const LDM_hashEntry entry = { cctx->ip - cctx->ibase, checksum };
#else
const LDM_hashEntry entry = { cctx->ip - cctx->ibase };
#endif
HASH_insert(cctx->hashTable, smallHash, entry);
}
#if LDM_LAG
}
#endif
cctx->lastPosHashed = cctx->ip;
cctx->lastHash = hash;
}
/**
* Copy over the cctx->lastHash, and cctx->lastPosHashed
* fields from the "next" fields.
*
* This requires that cctx->ip == cctx->nextPosHashed.
*/
static void LDM_updateLastHashFromNextHash(LDM_CCtx *cctx) {
putHashOfCurrentPositionFromHash(cctx, cctx->nextHash);
}
/**
* Insert hash of the current position into the hash table.
*/
static void LDM_putHashOfCurrentPosition(LDM_CCtx *cctx) {
U64 hash = getHash(cctx->ip, LDM_HASH_LENGTH);
putHashOfCurrentPositionFromHash(cctx, hash);
}
size_t LDM_initializeCCtx(LDM_CCtx *cctx,
const void *src, size_t srcSize,
void *dst, size_t maxDstSize) {
cctx->isize = srcSize;
cctx->maxOSize = maxDstSize;
cctx->ibase = (const BYTE *)src;
cctx->ip = cctx->ibase;
cctx->iend = cctx->ibase + srcSize;
cctx->ihashLimit = cctx->iend - LDM_HASH_LENGTH;
cctx->imatchLimit = cctx->iend - LDM_MIN_MATCH_LENGTH;
cctx->obase = (BYTE *)dst;
cctx->op = (BYTE *)dst;
cctx->anchor = cctx->ibase;
memset(&(cctx->stats), 0, sizeof(cctx->stats));
#if USE_CHECKSUM
cctx->hashTable = HASH_createTable(LDM_HASHTABLESIZE_U64);
#else
cctx->hashTable = HASH_createTable(LDM_HASHTABLESIZE_U32);
#endif
if (!cctx->hashTable) return 1;
cctx->stats.minOffset = UINT_MAX;
cctx->stats.windowSizeLog = LDM_WINDOW_SIZE_LOG;
cctx->stats.hashTableSizeLog = LDM_MEMORY_USAGE;
cctx->lastPosHashed = NULL;
cctx->step = 1; // Fixed to be 1 for now. Changing may break things.
cctx->nextIp = cctx->ip + cctx->step;
cctx->nextPosHashed = 0;
return 0;
}
void LDM_destroyCCtx(LDM_CCtx *cctx) {
HASH_destroyTable(cctx->hashTable);
}
/**
* Finds the "best" match.
*
* Returns 0 if successful and 1 otherwise (i.e. no match can be found
* in the remaining input that is long enough).
*
* forwardMatchLength contains the forward length of the match.
*/
static int LDM_findBestMatch(LDM_CCtx *cctx, const BYTE **match,
U64 *forwardMatchLength, U64 *backwardMatchLength) {
LDM_hashEntry *entry = NULL;
cctx->nextIp = cctx->ip + cctx->step;
while (entry == NULL) {
U64 hash;
hash_t smallHash;
U32 checksum;
#if INSERT_BY_TAG
U32 hashEveryMask;
#endif
setNextHash(cctx);
hash = cctx->nextHash;
smallHash = getSmallHash(hash);
checksum = getChecksum(hash);
#if INSERT_BY_TAG
hashEveryMask = lowerBitsFromHfHash(hash);
#endif
cctx->ip = cctx->nextIp;
cctx->nextIp += cctx->step;
if (cctx->ip > cctx->imatchLimit) {
return 1;
}
#if INSERT_BY_TAG
if (hashEveryMask == HASH_ONLY_EVERY) {
entry = HASH_getBestEntry(cctx, smallHash, checksum,
forwardMatchLength, backwardMatchLength);
}
#else
entry = HASH_getBestEntry(cctx, smallHash, checksum,
forwardMatchLength, backwardMatchLength);
#endif
if (entry != NULL) {
*match = entry->offset + cctx->ibase;
}
putHashOfCurrentPositionFromHash(cctx, hash);
}
setNextHash(cctx);
return 0;
}
void LDM_encodeLiteralLengthAndLiterals(
LDM_CCtx *cctx, BYTE *pToken, const U64 literalLength) {
/* Encode the literal length. */
if (literalLength >= RUN_MASK) {
U64 len = (U64)literalLength - RUN_MASK;
*pToken = (RUN_MASK << ML_BITS);
for (; len >= 255; len -= 255) {
*(cctx->op)++ = 255;
}
*(cctx->op)++ = (BYTE)len;
} else {
*pToken = (BYTE)(literalLength << ML_BITS);
}
/* Encode the literals. */
memcpy(cctx->op, cctx->anchor, literalLength);
cctx->op += literalLength;
}
void LDM_outputBlock(LDM_CCtx *cctx,
const U64 literalLength,
const U32 offset,
const U64 matchLength) {
BYTE *pToken = cctx->op++;
/* Encode the literal length and literals. */
LDM_encodeLiteralLengthAndLiterals(cctx, pToken, literalLength);
/* Encode the offset. */
MEM_write32(cctx->op, offset);
cctx->op += LDM_OFFSET_SIZE;
/* Encode the match length. */
if (matchLength >= ML_MASK) {
U64 matchLengthRemaining = matchLength;
*pToken += ML_MASK;
matchLengthRemaining -= ML_MASK;
MEM_write32(cctx->op, 0xFFFFFFFF);
while (matchLengthRemaining >= 4*0xFF) {
cctx->op += 4;
MEM_write32(cctx->op, 0xffffffff);
matchLengthRemaining -= 4*0xFF;
}
cctx->op += matchLengthRemaining / 255;
*(cctx->op)++ = (BYTE)(matchLengthRemaining % 255);
} else {
*pToken += (BYTE)(matchLength);
}
}
// TODO: maxDstSize is unused. This function may seg fault when writing
// beyond the size of dst, as it does not check maxDstSize. Writing to
// a buffer and performing checks is a possible solution.
//
// This is based upon lz4.
size_t LDM_compress(const void *src, size_t srcSize,
void *dst, size_t maxDstSize) {
LDM_CCtx cctx;
const BYTE *match = NULL;
U64 forwardMatchLength = 0;
U64 backwardsMatchLength = 0;
if (LDM_initializeCCtx(&cctx, src, srcSize, dst, maxDstSize)) {
// Initialization failed.
return 0;
}
#ifdef OUTPUT_CONFIGURATION
LDM_outputConfiguration();
#endif
/* Hash the first position and put it into the hash table. */
LDM_putHashOfCurrentPosition(&cctx);
cctx.lagIp = cctx.ip;
cctx.lagHash = cctx.lastHash;
/**
* Find a match.
* If no more matches can be found (i.e. the length of the remaining input
* is less than the minimum match length), then stop searching for matches
* and encode the final literals.
*/
while (!LDM_findBestMatch(&cctx, &match, &forwardMatchLength,
&backwardsMatchLength)) {
#ifdef COMPUTE_STATS
cctx.stats.numMatches++;
#endif
cctx.ip -= backwardsMatchLength;
match -= backwardsMatchLength;
/**
* Write current block (literals, literal length, match offset, match
* length) and update pointers and hashes.
*/
{
const U64 literalLength = cctx.ip - cctx.anchor;
const U32 offset = cctx.ip - match;
const U64 matchLength = forwardMatchLength +
backwardsMatchLength -
LDM_MIN_MATCH_LENGTH;
LDM_outputBlock(&cctx, literalLength, offset, matchLength);
#ifdef COMPUTE_STATS
cctx.stats.totalLiteralLength += literalLength;
cctx.stats.totalOffset += offset;
cctx.stats.totalMatchLength += matchLength + LDM_MIN_MATCH_LENGTH;
cctx.stats.minOffset =
offset < cctx.stats.minOffset ? offset : cctx.stats.minOffset;
cctx.stats.maxOffset =
offset > cctx.stats.maxOffset ? offset : cctx.stats.maxOffset;
cctx.stats.offsetHistogram[(U32)intLog2(offset)]++;
cctx.stats.matchLengthHistogram[
(U32)intLog2(matchLength + LDM_MIN_MATCH_LENGTH)]++;
#endif
// Move ip to end of block, inserting hashes at each position.
cctx.nextIp = cctx.ip + cctx.step;
while (cctx.ip < cctx.anchor + LDM_MIN_MATCH_LENGTH +
matchLength + literalLength) {
if (cctx.ip > cctx.lastPosHashed) {
// TODO: Simplify.
LDM_updateLastHashFromNextHash(&cctx);
setNextHash(&cctx);
}
cctx.ip++;
cctx.nextIp++;
}
}
// Set start of next block to current input pointer.
cctx.anchor = cctx.ip;
LDM_updateLastHashFromNextHash(&cctx);
}
/* Encode the last literals (no more matches). */
{
const U64 lastRun = cctx.iend - cctx.anchor;
BYTE *pToken = cctx.op++;
LDM_encodeLiteralLengthAndLiterals(&cctx, pToken, lastRun);
}
#ifdef COMPUTE_STATS
LDM_printCompressStats(&cctx.stats);
HASH_outputTableOccupancy(cctx.hashTable);
#endif
{
const size_t ret = cctx.op - cctx.obase;
LDM_destroyCCtx(&cctx);
return ret;
}
}
void LDM_outputConfiguration(void) {
printf("=====================\n");
printf("Configuration\n");
printf("LDM_WINDOW_SIZE_LOG: %d\n", LDM_WINDOW_SIZE_LOG);
printf("LDM_MIN_MATCH_LENGTH, LDM_HASH_LENGTH: %d, %d\n",
LDM_MIN_MATCH_LENGTH, LDM_HASH_LENGTH);
printf("LDM_MEMORY_USAGE: %d\n", LDM_MEMORY_USAGE);
printf("HASH_ONLY_EVERY_LOG: %d\n", HASH_ONLY_EVERY_LOG);
printf("HASH_BUCKET_SIZE_LOG: %d\n", HASH_BUCKET_SIZE_LOG);
printf("LDM_LAG: %d\n", LDM_LAG);
printf("USE_CHECKSUM: %d\n", USE_CHECKSUM);
printf("INSERT_BY_TAG: %d\n", INSERT_BY_TAG);
printf("HASH_CHAR_OFFSET: %d\n", HASH_CHAR_OFFSET);
printf("=====================\n");
}
contrib/long_distance_matching/ldm.h
+197
View File
@@ -0,0 +1,197 @@
#ifndef LDM_H
#define LDM_H
#include "mem.h" // from /lib/common/mem.h
//#include "ldm_params.h"
// =============================================================================
// Modify the parameters in ldm_params.h if "ldm_params.h" is included.
// Otherwise, modify the parameters here.
// =============================================================================
#ifndef LDM_PARAMS_H
// Defines the size of the hash table.
// Note that this is not the number of buckets.
// Currently this should be less than WINDOW_SIZE_LOG + 4.
#define LDM_MEMORY_USAGE 23
// The number of entries in a hash bucket.
#define HASH_BUCKET_SIZE_LOG 3 // The maximum is 4 for now.
// Defines the lag in inserting elements into the hash table.
#define LDM_LAG 0
// The maximum window size when searching for matches.
// The maximum value is 30
#define LDM_WINDOW_SIZE_LOG 28
// The minimum match length.
// This should be a multiple of four.
#define LDM_MIN_MATCH_LENGTH 64
// If INSERT_BY_TAG, insert entries into the hash table as a function of the
// hash. Certain hashes will not be inserted.
//
// Otherwise, insert as a function of the position.
#define INSERT_BY_TAG 1
// Store a checksum with the hash table entries for faster comparison.
// This halves the number of entries the hash table can contain.
#define USE_CHECKSUM 1
#endif
// Output compression statistics.
#define COMPUTE_STATS
// Output the configuration.
#define OUTPUT_CONFIGURATION
// If defined, forces the probability of insertion to be approximately
// one per (1 << HASH_ONLY_EVERY_LOG). If not defined, the probability will be
// calculated based on the memory usage and window size for "even" insertion
// throughout the window.
// #define HASH_ONLY_EVERY_LOG 8
// =============================================================================
// The number of bytes storing the compressed and decompressed size
// in the header.
#define LDM_COMPRESSED_SIZE 8
#define LDM_DECOMPRESSED_SIZE 8
#define LDM_HEADER_SIZE ((LDM_COMPRESSED_SIZE)+(LDM_DECOMPRESSED_SIZE))
#define ML_BITS 4
#define ML_MASK ((1U<<ML_BITS)-1)
#define RUN_BITS (8-ML_BITS)
#define RUN_MASK ((1U<<RUN_BITS)-1)
// The number of bytes storing the offset.
#define LDM_OFFSET_SIZE 4
#define LDM_WINDOW_SIZE (1 << (LDM_WINDOW_SIZE_LOG))
// TODO: Match lengths that are too small do not use the hash table efficiently.
// There should be a minimum hash length given the hash table size.
#define LDM_HASH_LENGTH LDM_MIN_MATCH_LENGTH
typedef struct LDM_compressStats LDM_compressStats;
typedef struct LDM_CCtx LDM_CCtx;
typedef struct LDM_DCtx LDM_DCtx;
/**
* Compresses src into dst.
* Returns the compressed size if successful, 0 otherwise.
*
* NB: This currently ignores maxDstSize and assumes enough space is available.
*
* Block format (see lz4 documentation for more information):
* github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md
*
* A block is composed of sequences. Each sequence begins with a token, which
* is a one-byte value separated into two 4-bit fields.
*
* The first field uses the four high bits of the token and encodes the literal
* length. If the field value is 0, there is no literal. If it is 15,
* additional bytes are added (each ranging from 0 to 255) to the previous
* value to produce a total length.
*
* Following the token and optional length bytes are the literals.
*
* Next are the 4 bytes representing the offset of the match (2 in lz4),
* representing the position to copy the literals.
*
* The lower four bits of the token encode the match length. With additional
* bytes added similarly to the additional literal length bytes after the offset.
*
* The last sequence is incomplete and stops right after the literals.
*/
size_t LDM_compress(const void *src, size_t srcSize,
void *dst, size_t maxDstSize);
/**
* Initialize the compression context.
*
* Allocates memory for the hash table.
*
* Returns 0 if successful, 1 otherwise.
*/
size_t LDM_initializeCCtx(LDM_CCtx *cctx,
const void *src, size_t srcSize,
void *dst, size_t maxDstSize);
/**
* Frees up memory allocated in LDM_initializeCCtx().
*/
void LDM_destroyCCtx(LDM_CCtx *cctx);
/**
* Prints the distribution of offsets in the hash table.
*
* The offsets are defined as the distance of the hash table entry from the
* current input position of the cctx.
*/
void LDM_outputHashTableOffsetHistogram(const LDM_CCtx *cctx);
/**
* Outputs compression statistics to stdout.
*/
void LDM_printCompressStats(const LDM_compressStats *stats);
/**
* Encode the literal length followed by the literals.
*
* The literal length is written to the upper four bits of pToken, with
* additional bytes written to the output as needed (see lz4).
*
* This is followed by literalLength bytes corresponding to the literals.
*/
void LDM_encodeLiteralLengthAndLiterals(LDM_CCtx *cctx, BYTE *pToken,
const U64 literalLength);
/**
* Write current block (literals, literal length, match offset,
* match length).
*/
void LDM_outputBlock(LDM_CCtx *cctx,
const U64 literalLength,
const U32 offset,
const U64 matchLength);
/**
* Decompresses src into dst.
*
* Note: assumes src does not have a header.
*/
size_t LDM_decompress(const void *src, size_t srcSize,
void *dst, size_t maxDstSize);
/**
* Initialize the decompression context.
*/
void LDM_initializeDCtx(LDM_DCtx *dctx,
const void *src, size_t compressedSize,
void *dst, size_t maxDecompressedSize);
/**
* Reads the header from src and writes the compressed size and
* decompressed size into compressedSize and decompressedSize respectively.
*
* NB: LDM_compress and LDM_decompress currently do not add/read headers.
*/
void LDM_readHeader(const void *src, U64 *compressedSize,
U64 *decompressedSize);
/**
* Write the compressed and decompressed size.
*/
void LDM_writeHeader(void *memPtr, U64 compressedSize,
U64 decompressedSize);
/**
* Output the configuration used.
*/
void LDM_outputConfiguration(void);
#endif /* LDM_H */
contrib/long_distance_matching/ldm_common.c
+109
View File
@@ -0,0 +1,109 @@
#include <stdio.h>
#include "ldm.h"
/**
* This function reads the header at the beginning of src and writes
* the compressed and decompressed size to compressedSize and
* decompressedSize.
*
* The header consists of 16 bytes: 8 bytes each in little-endian format
* of the compressed size and the decompressed size.
*/
void LDM_readHeader(const void *src, U64 *compressedSize,
U64 *decompressedSize) {
const BYTE *ip = (const BYTE *)src;
*compressedSize = MEM_readLE64(ip);
*decompressedSize = MEM_readLE64(ip + 8);
}
/**
* Writes the 16-byte header (8-bytes each of the compressedSize and
* decompressedSize in little-endian format) to memPtr.
*/
void LDM_writeHeader(void *memPtr, U64 compressedSize,
U64 decompressedSize) {
MEM_writeLE64(memPtr, compressedSize);
MEM_writeLE64((BYTE *)memPtr + 8, decompressedSize);
}
struct LDM_DCtx {
size_t compressedSize;
size_t maxDecompressedSize;
const BYTE *ibase; /* Base of input */
const BYTE *ip; /* Current input position */
const BYTE *iend; /* End of source */
const BYTE *obase; /* Base of output */
BYTE *op; /* Current output position */
const BYTE *oend; /* End of output */
};
void LDM_initializeDCtx(LDM_DCtx *dctx,
const void *src, size_t compressedSize,
void *dst, size_t maxDecompressedSize) {
dctx->compressedSize = compressedSize;
dctx->maxDecompressedSize = maxDecompressedSize;
dctx->ibase = src;
dctx->ip = (const BYTE *)src;
dctx->iend = dctx->ip + dctx->compressedSize;
dctx->op = dst;
dctx->oend = dctx->op + dctx->maxDecompressedSize;
}
size_t LDM_decompress(const void *src, size_t compressedSize,
void *dst, size_t maxDecompressedSize) {
LDM_DCtx dctx;
LDM_initializeDCtx(&dctx, src, compressedSize, dst, maxDecompressedSize);
while (dctx.ip < dctx.iend) {
BYTE *cpy;
const BYTE *match;
size_t length, offset;
/* Get the literal length. */
const unsigned token = *(dctx.ip)++;
if ((length = (token >> ML_BITS)) == RUN_MASK) {
unsigned s;
do {
s = *(dctx.ip)++;
length += s;
} while (s == 255);
}
/* Copy the literals. */
cpy = dctx.op + length;
memcpy(dctx.op, dctx.ip, length);
dctx.ip += length;
dctx.op = cpy;
//TODO: dynamic offset size?
/* Encode the offset. */
offset = MEM_read32(dctx.ip);
dctx.ip += LDM_OFFSET_SIZE;
match = dctx.op - offset;
/* Get the match length. */
length = token & ML_MASK;
if (length == ML_MASK) {
unsigned s;
do {
s = *(dctx.ip)++;
length += s;
} while (s == 255);
}
length += LDM_MIN_MATCH_LENGTH;
/* Copy match. */
cpy = dctx.op + length;
// TODO: this can be made more efficient.
while (match < cpy - offset && dctx.op < dctx.oend) {
*(dctx.op)++ = *match++;
}
}
return dctx.op - (BYTE *)dst;
}
contrib/long_distance_matching/ldm_params.h
+12
View File
@@ -0,0 +1,12 @@
#ifndef LDM_PARAMS_H
#define LDM_PARAMS_H
#define LDM_MEMORY_USAGE 23
#define HASH_BUCKET_SIZE_LOG 3
#define LDM_LAG 0
#define LDM_WINDOW_SIZE_LOG 28
#define LDM_MIN_MATCH_LENGTH 64
#define INSERT_BY_TAG 1
#define USE_CHECKSUM 1
#endif // LDM_PARAMS_H
contrib/long_distance_matching/main.c
+269
View File
@@ -0,0 +1,269 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/time.h>
#include <sys/types.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <unistd.h>
#include <zstd.h>
#include <fcntl.h>
#include "ldm.h"
#include "zstd.h"
// #define DECOMPRESS_AND_VERIFY
/* Compress file given by fname and output to oname.
* Returns 0 if successful, error code otherwise.
*
* This adds a header from LDM_writeHeader to the beginning of the output.
*
* This might seg fault if the compressed size is > the decompress
* size due to the mmapping and output file size allocated to be the input size
* The compress function should check before writing or buffer writes.
*/
static int compress(const char *fname, const char *oname) {
int fdin, fdout;
struct stat statbuf;
char *src, *dst;
size_t maxCompressedSize, compressedSize;
struct timeval tv1, tv2;
double timeTaken;
/* Open the input file. */
if ((fdin = open(fname, O_RDONLY)) < 0) {
perror("Error in file opening");
return 1;
}
/* Open the output file. */
if ((fdout = open(oname, O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600)) < 0) {
perror("Can't create output file");
return 1;
}
/* Find the size of the input file. */
if (fstat (fdin, &statbuf) < 0) {
perror("Fstat error");
return 1;
}
maxCompressedSize = (statbuf.st_size + LDM_HEADER_SIZE);
// Handle case where compressed size is > decompressed size.
// TODO: The compress function should check before writing or buffer writes.
maxCompressedSize += statbuf.st_size / 255;
ftruncate(fdout, maxCompressedSize);
/* mmap the input file. */
if ((src = mmap(0, statbuf.st_size, PROT_READ, MAP_SHARED, fdin, 0))
== (caddr_t) - 1) {
perror("mmap error for input");
return 1;
}
/* mmap the output file. */
if ((dst = mmap(0, maxCompressedSize, PROT_READ | PROT_WRITE,
MAP_SHARED, fdout, 0)) == (caddr_t) - 1) {
perror("mmap error for output");
return 1;
}
gettimeofday(&tv1, NULL);
compressedSize = LDM_HEADER_SIZE +
LDM_compress(src, statbuf.st_size,
dst + LDM_HEADER_SIZE, maxCompressedSize);
gettimeofday(&tv2, NULL);
// Write the header.
LDM_writeHeader(dst, compressedSize, statbuf.st_size);
// Truncate file to compressedSize.
ftruncate(fdout, compressedSize);
printf("%25s : %10lu -> %10lu - %s \n", fname,
(size_t)statbuf.st_size, (size_t)compressedSize, oname);
printf("Compression ratio: %.2fx --- %.1f%%\n",
(double)statbuf.st_size / (double)compressedSize,
(double)compressedSize / (double)(statbuf.st_size) * 100.0);
timeTaken = (double) (tv2.tv_usec - tv1.tv_usec) / 1000000 +
(double) (tv2.tv_sec - tv1.tv_sec),
printf("Total compress time = %.3f seconds, Average scanning speed: %.3f MB/s\n",
timeTaken,
((double)statbuf.st_size / (double) (1 << 20)) / timeTaken);
// Close files.
close(fdin);
close(fdout);
return 0;
}
#ifdef DECOMPRESS_AND_VERIFY
/* Decompress file compressed using LDM_compress.
* The input file should have the LDM_HEADER followed by payload.
* Returns 0 if succesful, and an error code otherwise.
*/
static int decompress(const char *fname, const char *oname) {
int fdin, fdout;
struct stat statbuf;
char *src, *dst;
U64 compressedSize, decompressedSize;
size_t outSize;
/* Open the input file. */
if ((fdin = open(fname, O_RDONLY)) < 0) {
perror("Error in file opening");
return 1;
}
/* Open the output file. */
if ((fdout = open(oname, O_RDWR | O_CREAT | O_TRUNC, (mode_t)0600)) < 0) {
perror("Can't create output file");
return 1;
}
/* Find the size of the input file. */
if (fstat (fdin, &statbuf) < 0) {
perror("Fstat error");
return 1;
}
/* mmap the input file. */
if ((src = mmap(0, statbuf.st_size, PROT_READ, MAP_SHARED, fdin, 0))
== (caddr_t) - 1) {
perror("mmap error for input");
return 1;
}
/* Read the header. */
LDM_readHeader(src, &compressedSize, &decompressedSize);
ftruncate(fdout, decompressedSize);
/* mmap the output file */
if ((dst = mmap(0, decompressedSize, PROT_READ | PROT_WRITE,
MAP_SHARED, fdout, 0)) == (caddr_t) - 1) {
perror("mmap error for output");
return 1;
}
outSize = LDM_decompress(
src + LDM_HEADER_SIZE, statbuf.st_size - LDM_HEADER_SIZE,
dst, decompressedSize);
printf("Ret size out: %zu\n", outSize);
close(fdin);
close(fdout);
return 0;
}
/* Compare two files.
* Returns 0 iff they are the same.
*/
static int compare(FILE *fp0, FILE *fp1) {
int result = 0;
while (result == 0) {
char b0[1024];
char b1[1024];
const size_t r0 = fread(b0, 1, sizeof(b0), fp0);
const size_t r1 = fread(b1, 1, sizeof(b1), fp1);
result = (int)r0 - (int)r1;
if (0 == r0 || 0 == r1) break;
if (0 == result) result = memcmp(b0, b1, r0);
}
return result;
}
/* Verify the input file is the same as the decompressed file. */
static int verify(const char *inpFilename, const char *decFilename) {
FILE *inpFp, *decFp;
if ((inpFp = fopen(inpFilename, "rb")) == NULL) {
perror("Could not open input file\n");
return 1;
}
if ((decFp = fopen(decFilename, "rb")) == NULL) {
perror("Could not open decompressed file\n");
return 1;
}
printf("verify : %s <-> %s\n", inpFilename, decFilename);
{
const int cmp = compare(inpFp, decFp);
if(0 == cmp) {
printf("verify : OK\n");
} else {
printf("verify : NG\n");
return 1;
}
}
fclose(decFp);
fclose(inpFp);
return 0;
}
#endif
int main(int argc, const char *argv[]) {
const char * const exeName = argv[0];
char inpFilename[256] = { 0 };
char ldmFilename[256] = { 0 };
char decFilename[256] = { 0 };
if (argc < 2) {
printf("Wrong arguments\n");
printf("Usage:\n");
printf("%s FILE\n", exeName);
return 1;
}
snprintf(inpFilename, 256, "%s", argv[1]);
snprintf(ldmFilename, 256, "%s.ldm", argv[1]);
snprintf(decFilename, 256, "%s.ldm.dec", argv[1]);
printf("inp = [%s]\n", inpFilename);
printf("ldm = [%s]\n", ldmFilename);
printf("dec = [%s]\n", decFilename);
/* Compress */
{
if (compress(inpFilename, ldmFilename)) {
printf("Compress error\n");
return 1;
}
}
#ifdef DECOMPRESS_AND_VERIFY
/* Decompress */
{
struct timeval tv1, tv2;
gettimeofday(&tv1, NULL);
if (decompress(ldmFilename, decFilename)) {
printf("Decompress error\n");
return 1;
}
gettimeofday(&tv2, NULL);
printf("Total decompress time = %f seconds\n",
(double) (tv2.tv_usec - tv1.tv_usec) / 1000000 +
(double) (tv2.tv_sec - tv1.tv_sec));
}
/* verify */
if (verify(inpFilename, decFilename)) {
printf("Verification error\n");
return 1;
}
#endif
return 0;
}
contrib/pzstd/ErrorHolder.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/Logging.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/Makefile
+5 -5
View File
@@ -1,11 +1,11 @@
# ##########################################################################
# ################################################################
# Copyright (c) 2016-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# ##########################################################################
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
# Standard variables for installation
DESTDIR ?=
contrib/pzstd/Options.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "Options.h"
#include "util.h"
contrib/pzstd/Options.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/Pzstd.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "Pzstd.h"
#include "SkippableFrame.h"
contrib/pzstd/Pzstd.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/SkippableFrame.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "SkippableFrame.h"
#include "mem.h"
contrib/pzstd/SkippableFrame.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/main.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "ErrorHolder.h"
#include "Options.h"
contrib/pzstd/test/OptionsTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "Options.h"
contrib/pzstd/test/PzstdTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "Pzstd.h"
extern "C" {
contrib/pzstd/test/RoundTrip.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/test/RoundTripTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
extern "C" {
#include "datagen.h"
contrib/pzstd/utils/Buffer.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/utils/FileSystem.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/utils/Likely.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
contrib/pzstd/utils/Range.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
contrib/pzstd/utils/ResourcePool.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/utils/ScopeGuard.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/utils/ThreadPool.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/utils/WorkQueue.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#pragma once
contrib/pzstd/utils/test/BufferTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "utils/Buffer.h"
#include "utils/Range.h"
contrib/pzstd/utils/test/RangeTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "utils/Range.h"
contrib/pzstd/utils/test/ResourcePoolTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "utils/ResourcePool.h"
contrib/pzstd/utils/test/ScopeGuardTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "utils/ScopeGuard.h"
contrib/pzstd/utils/test/ThreadPoolTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "utils/ThreadPool.h"
contrib/pzstd/utils/test/WorkQueueTest.cpp
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "utils/Buffer.h"
#include "utils/WorkQueue.h"
contrib/seekable_format/examples/Makefile
+3 -3
View File
@@ -2,9 +2,9 @@
# Copyright (c) 2017-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
# This Makefile presumes libzstd is built, using `make` in / or /lib/
contrib/seekable_format/examples/parallel_compression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2017-present, Facebook, Inc.
/*
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdlib.h> // malloc, free, exit, atoi
contrib/seekable_format/examples/parallel_processing.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2017-present, Facebook, Inc.
/*
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*
contrib/seekable_format/examples/seekable_compression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2017-present, Facebook, Inc.
/*
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdlib.h> // malloc, free, exit, atoi
contrib/seekable_format/examples/seekable_decompression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
contrib/seekable_format/zstdseek_compress.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdlib.h> /* malloc, free */
doc/educational_decoder/Makefile
+34
View File
@@ -0,0 +1,34 @@
HARNESS_FILES=*.c
MULTITHREAD_LDFLAGS = -pthread
DEBUGFLAGS= -g -DZSTD_DEBUG=1
CPPFLAGS += -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-I$(ZSTDDIR)/dictBuilder -I$(ZSTDDIR)/deprecated -I$(PRGDIR)
CFLAGS ?= -O3
CFLAGS += -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-Wstrict-prototypes -Wundef -Wformat-security \
-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
-Wredundant-decls
CFLAGS += $(DEBUGFLAGS)
CFLAGS += $(MOREFLAGS)
FLAGS = $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $(MULTITHREAD_LDFLAGS)
harness: $(HARNESS_FILES)
$(CC) $(FLAGS) $^ -o $@
clean:
@$(RM) -f harness
@$(RM) -rf harness.dSYM
test: harness
@zstd README.md -o tmp.zst
@./harness tmp.zst tmp
@diff -s tmp README.md
@$(RM) -f tmp*
@zstd --train harness.c zstd_decompress.c zstd_decompress.h README.md
@zstd -D dictionary README.md -o tmp.zst
@./harness tmp.zst tmp dictionary
@diff -s tmp README.md
@$(RM) -f tmp* dictionary
@make clean
doc/educational_decoder/harness.c
+11 -6
View File
@@ -2,9 +2,9 @@
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdio.h>
@@ -87,7 +87,7 @@ int main(int argc, char **argv) {
}
size_t decompressed_size = ZSTD_get_decompressed_size(input, input_size);
if (decompressed_size == -1) {
if (decompressed_size == (size_t)-1) {
decompressed_size = MAX_COMPRESSION_RATIO * input_size;
fprintf(stderr, "WARNING: Compressed data does not contain "
"decompressed size, going to assume the compression "
@@ -106,9 +106,15 @@ int main(int argc, char **argv) {
return 1;
}
dictionary_t* const parsed_dict = create_dictionary();
if (dict) {
parse_dictionary(parsed_dict, dict, dict_size);
}
size_t decompressed =
ZSTD_decompress_with_dict(output, decompressed_size,
input, input_size, dict, dict_size);
input, input_size, parsed_dict);
free_dictionary(parsed_dict);
write_file(argv[2], output, decompressed);
@@ -117,4 +123,3 @@ int main(int argc, char **argv) {
free(dict);
input = output = dict = NULL;
}
doc/educational_decoder/zstd_decompress.c
+188 -243
View File
@@ -2,9 +2,9 @@
* Copyright (c) 2017-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/// Zstandard educational decoder implementation
@@ -14,21 +14,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
/// Zstandard decompression functions.
/// `dst` must point to a space at least as large as the reconstructed output.
size_t ZSTD_decompress(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len);
/// If `dict != NULL` and `dict_len >= 8`, does the same thing as
/// `ZSTD_decompress` but uses the provided dict
size_t ZSTD_decompress_with_dict(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len,
const void *const dict, const size_t dict_len);
/// Get the decompressed size of an input stream so memory can be allocated in
/// advance
/// Returns -1 if the size can't be determined
size_t ZSTD_get_decompressed_size(const void *const src, const size_t src_len);
#include "zstd_decompress.h"
/******* UTILITY MACROS AND TYPES *********************************************/
// Max block size decompressed size is 128 KB and literal blocks can't be
@@ -108,10 +94,10 @@ static inline size_t IO_istream_len(const istream_t *const in);
/// Advances the stream by `len` bytes, and returns a pointer to the chunk that
/// was skipped. The stream must be byte aligned.
static inline const u8 *IO_read_bytes(istream_t *const in, size_t len);
static inline const u8 *IO_get_read_ptr(istream_t *const in, size_t len);
/// Advances the stream by `len` bytes, and returns a pointer to the chunk that
/// was skipped so it can be written to.
static inline u8 *IO_write_bytes(ostream_t *const out, size_t len);
static inline u8 *IO_get_write_ptr(ostream_t *const out, size_t len);
/// Advance the inner state by `len` bytes. The stream must be byte aligned.
static inline void IO_advance_input(istream_t *const in, size_t len);
@@ -307,7 +293,7 @@ typedef struct {
/// The decoded contents of a dictionary so that it doesn't have to be repeated
/// for each frame that uses it
typedef struct {
struct dictionary_s {
// Entropy tables
HUF_dtable literals_dtable;
FSE_dtable ll_dtable;
@@ -322,7 +308,7 @@ typedef struct {
u64 previous_offsets[3];
u32 dictionary_id;
} dictionary_t;
};
/// A tuple containing the parts necessary to decode and execute a ZSTD sequence
/// command
@@ -367,27 +353,36 @@ static void execute_sequences(frame_context_t *const ctx, ostream_t *const out,
const sequence_command_t *const sequences,
const size_t num_sequences);
// Parse a provided dictionary blob for use in decompression
static void parse_dictionary(dictionary_t *const dict, const u8 *src,
size_t src_len);
static void free_dictionary(dictionary_t *const dict);
// Copies literals and returns the total literal length that was copied
static u32 copy_literals(const size_t seq, istream_t *litstream,
ostream_t *const out);
// Given an offset code from a sequence command (either an actual offset value
// or an index for previous offset), computes the correct offset and udpates
// the offset history
static size_t compute_offset(sequence_command_t seq, u64 *const offset_hist);
// Given an offset, match length, and total output, as well as the frame
// context for the dictionary, determines if the dictionary is used and
// executes the copy operation
static void execute_match_copy(frame_context_t *const ctx, size_t offset,
size_t match_length, size_t total_output,
ostream_t *const out);
/******* END ZSTD HELPER STRUCTS AND PROTOTYPES *******************************/
size_t ZSTD_decompress(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len) {
return ZSTD_decompress_with_dict(dst, dst_len, src, src_len, NULL, 0);
dictionary_t* uninit_dict = create_dictionary();
size_t const decomp_size = ZSTD_decompress_with_dict(dst, dst_len, src,
src_len, uninit_dict);
free_dictionary(uninit_dict);
return decomp_size;
}
size_t ZSTD_decompress_with_dict(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len,
const void *const dict,
const size_t dict_len) {
dictionary_t parsed_dict;
memset(&parsed_dict, 0, sizeof(dictionary_t));
// dict_len < 8 is not a valid dictionary
if (dict && dict_len > 8) {
parse_dictionary(&parsed_dict, (const u8 *)dict, dict_len);
}
dictionary_t* parsed_dict) {
istream_t in = IO_make_istream(src, src_len);
ostream_t out = IO_make_ostream(dst, dst_len);
@@ -396,11 +391,9 @@ size_t ZSTD_decompress_with_dict(void *const dst, const size_t dst_len,
// Multiple frames can be appended into a single file or stream. A frame is
// totally independent, has a defined beginning and end, and a set of
// parameters which tells the decoder how to decompress it."
while (IO_istream_len(&in) > 0) {
decode_frame(&out, &in, &parsed_dict);
}
free_dictionary(&parsed_dict);
/* this decoder assumes decompression of a single frame */
decode_frame(&out, &in, parsed_dict);
return out.ptr - (u8 *)dst;
}
@@ -424,30 +417,6 @@ static void decompress_data(frame_context_t *const ctx, ostream_t *const out,
static void decode_frame(ostream_t *const out, istream_t *const in,
const dictionary_t *const dict) {
const u32 magic_number = IO_read_bits(in, 32);
// Skippable frame
//
// "Magic_Number
//
// 4 Bytes, little-endian format. Value : 0x184D2A5?, which means any value
// from 0x184D2A50 to 0x184D2A5F. All 16 values are valid to identify a
// skippable frame."
if ((magic_number & ~0xFU) == 0x184D2A50U) {
// "Skippable frames allow the insertion of user-defined data into a
// flow of concatenated frames. Its design is pretty straightforward,
// with the sole objective to allow the decoder to quickly skip over
// user-defined data and continue decoding.
//
// Skippable frames defined in this specification are compatible with
// LZ4 ones."
const size_t frame_size = IO_read_bits(in, 32);
// skip over frame
IO_advance_input(in, frame_size);
return;
}
// Zstandard frame
//
// "Magic_Number
@@ -460,8 +429,8 @@ static void decode_frame(ostream_t *const out, istream_t *const in,
return;
}
// not a real frame
ERROR("Invalid magic number");
// not a real frame or a skippable frame
ERROR("Tried to decode non-ZSTD frame");
}
/// Decode a frame that contains compressed data. Not all frames do as there
@@ -672,8 +641,8 @@ static void decompress_data(frame_context_t *const ctx, ostream_t *const out,
case 0: {
// "Raw_Block - this is an uncompressed block. Block_Size is the
// number of bytes to read and copy."
const u8 *const read_ptr = IO_read_bytes(in, block_len);
u8 *const write_ptr = IO_write_bytes(out, block_len);
const u8 *const read_ptr = IO_get_read_ptr(in, block_len);
u8 *const write_ptr = IO_get_write_ptr(out, block_len);
// Copy the raw data into the output
memcpy(write_ptr, read_ptr, block_len);
@@ -685,8 +654,8 @@ static void decompress_data(frame_context_t *const ctx, ostream_t *const out,
// "RLE_Block - this is a single byte, repeated N times. In which
// case, Block_Size is the size to regenerate, while the
// "compressed" block is just 1 byte (the byte to repeat)."
const u8 *const read_ptr = IO_read_bytes(in, 1);
u8 *const write_ptr = IO_write_bytes(out, block_len);
const u8 *const read_ptr = IO_get_read_ptr(in, 1);
u8 *const write_ptr = IO_get_write_ptr(out, block_len);
// Copy `block_len` copies of `read_ptr[0]` to the output
memset(write_ptr, read_ptr[0], block_len);
@@ -832,13 +801,13 @@ static size_t decode_literals_simple(istream_t *const in, u8 **const literals,
switch (block_type) {
case 0: {
// "Raw_Literals_Block - Literals are stored uncompressed."
const u8 *const read_ptr = IO_read_bytes(in, size);
const u8 *const read_ptr = IO_get_read_ptr(in, size);
memcpy(*literals, read_ptr, size);
break;
}
case 1: {
// "RLE_Literals_Block - Literals consist of a single byte value repeated N times."
const u8 *const read_ptr = IO_read_bytes(in, 1);
const u8 *const read_ptr = IO_get_read_ptr(in, 1);
memset(*literals, read_ptr[0], size);
break;
}
@@ -949,7 +918,7 @@ static void decode_huf_table(HUF_dtable *const dtable, istream_t *const in) {
num_symbs = header - 127;
const size_t bytes = (num_symbs + 1) / 2;
const u8 *const weight_src = IO_read_bytes(in, bytes);
const u8 *const weight_src = IO_get_read_ptr(in, bytes);
for (int i = 0; i < num_symbs; i++) {
// "They are encoded forward, 2
@@ -1157,7 +1126,7 @@ static void decompress_sequences(frame_context_t *const ctx, istream_t *in,
}
const size_t len = IO_istream_len(in);
const u8 *const src = IO_read_bytes(in, len);
const u8 *const src = IO_get_read_ptr(in, len);
// "After writing the last bit containing information, the compressor writes
// a single 1-bit and then fills the byte with 0-7 0 bits of padding."
@@ -1262,7 +1231,7 @@ static void decode_seq_table(FSE_dtable *const table, istream_t *const in,
}
case seq_rle: {
// "RLE_Mode : it's a single code, repeated Number_of_Sequences times."
const u8 symb = IO_read_bytes(in, 1)[0];
const u8 symb = IO_get_read_ptr(in, 1)[0];
FSE_init_dtable_rle(table, symb);
break;
}
@@ -1303,145 +1272,146 @@ static void execute_sequences(frame_context_t *const ctx, ostream_t *const out,
for (size_t i = 0; i < num_sequences; i++) {
const sequence_command_t seq = sequences[i];
{
// If the sequence asks for more literals than are left, the
// sequence must be corrupted
if (seq.literal_length > IO_istream_len(&litstream)) {
CORRUPTION();
}
u8 *const write_ptr = IO_write_bytes(out, seq.literal_length);
const u8 *const read_ptr =
IO_read_bytes(&litstream, seq.literal_length);
// Copy literals to output
memcpy(write_ptr, read_ptr, seq.literal_length);
total_output += seq.literal_length;
const u32 literals_size = copy_literals(seq.literal_length, &litstream, out);
total_output += literals_size;
}
size_t offset;
size_t const offset = compute_offset(seq, offset_hist);
// Offsets are special, we need to handle the repeat offsets
if (seq.offset <= 3) {
// "The first 3 values define a repeated offset and we will call
// them Repeated_Offset1, Repeated_Offset2, and Repeated_Offset3.
// They are sorted in recency order, with Repeated_Offset1 meaning
// 'most recent one'".
size_t const match_length = seq.match_length;
// Use 0 indexing for the array
u32 idx = seq.offset - 1;
if (seq.literal_length == 0) {
// "There is an exception though, when current sequence's
// literals length is 0. In this case, repeated offsets are
// shifted by one, so Repeated_Offset1 becomes Repeated_Offset2,
// Repeated_Offset2 becomes Repeated_Offset3, and
// Repeated_Offset3 becomes Repeated_Offset1 - 1_byte."
idx++;
}
execute_match_copy(ctx, offset, match_length, total_output, out);
if (idx == 0) {
offset = offset_hist[0];
} else {
// If idx == 3 then literal length was 0 and the offset was 3,
// as per the exception listed above
offset = idx < 3 ? offset_hist[idx] : offset_hist[0] - 1;
// If idx == 1 we don't need to modify offset_hist[2], since
// we're using the second-most recent code
if (idx > 1) {
offset_hist[2] = offset_hist[1];
}
offset_hist[1] = offset_hist[0];
offset_hist[0] = offset;
}
} else {
// When it's not a repeat offset:
// "if (Offset_Value > 3) offset = Offset_Value - 3;"
offset = seq.offset - 3;
// Shift back history
offset_hist[2] = offset_hist[1];
offset_hist[1] = offset_hist[0];
offset_hist[0] = offset;
}
size_t match_length = seq.match_length;
u8 *write_ptr = IO_write_bytes(out, match_length);
if (total_output <= ctx->header.window_size) {
// In this case offset might go back into the dictionary
if (offset > total_output + ctx->dict_content_len) {
// The offset goes beyond even the dictionary
CORRUPTION();
}
if (offset > total_output) {
// "The rest of the dictionary is its content. The content act
// as a "past" in front of data to compress or decompress, so it
// can be referenced in sequence commands."
const size_t dict_copy =
MIN(offset - total_output, match_length);
const size_t dict_offset =
ctx->dict_content_len - (offset - total_output);
memcpy(write_ptr, ctx->dict_content + dict_offset, dict_copy);
write_ptr += dict_copy;
match_length -= dict_copy;
}
} else if (offset > ctx->header.window_size) {
CORRUPTION();
}
// We must copy byte by byte because the match length might be larger
// than the offset
// ex: if the output so far was "abc", a command with offset=3 and
// match_length=6 would produce "abcabcabc" as the new output
for (size_t i = 0; i < match_length; i++) {
*write_ptr = *(write_ptr - offset);
write_ptr++;
}
total_output += seq.match_length;
total_output += match_length;
}
// Copy any leftover literals
{
size_t len = IO_istream_len(&litstream);
u8 *const write_ptr = IO_write_bytes(out, len);
const u8 *const read_ptr = IO_read_bytes(&litstream, len);
memcpy(write_ptr, read_ptr, len);
copy_literals(len, &litstream, out);
total_output += len;
}
ctx->current_total_output = total_output;
}
static u32 copy_literals(const size_t literal_length, istream_t *litstream,
ostream_t *const out) {
// If the sequence asks for more literals than are left, the
// sequence must be corrupted
if (literal_length > IO_istream_len(litstream)) {
CORRUPTION();
}
u8 *const write_ptr = IO_get_write_ptr(out, literal_length);
const u8 *const read_ptr =
IO_get_read_ptr(litstream, literal_length);
// Copy literals to output
memcpy(write_ptr, read_ptr, literal_length);
return literal_length;
}
static size_t compute_offset(sequence_command_t seq, u64 *const offset_hist) {
size_t offset;
// Offsets are special, we need to handle the repeat offsets
if (seq.offset <= 3) {
// "The first 3 values define a repeated offset and we will call
// them Repeated_Offset1, Repeated_Offset2, and Repeated_Offset3.
// They are sorted in recency order, with Repeated_Offset1 meaning
// 'most recent one'".
// Use 0 indexing for the array
u32 idx = seq.offset - 1;
if (seq.literal_length == 0) {
// "There is an exception though, when current sequence's
// literals length is 0. In this case, repeated offsets are
// shifted by one, so Repeated_Offset1 becomes Repeated_Offset2,
// Repeated_Offset2 becomes Repeated_Offset3, and
// Repeated_Offset3 becomes Repeated_Offset1 - 1_byte."
idx++;
}
if (idx == 0) {
offset = offset_hist[0];
} else {
// If idx == 3 then literal length was 0 and the offset was 3,
// as per the exception listed above
offset = idx < 3 ? offset_hist[idx] : offset_hist[0] - 1;
// If idx == 1 we don't need to modify offset_hist[2], since
// we're using the second-most recent code
if (idx > 1) {
offset_hist[2] = offset_hist[1];
}
offset_hist[1] = offset_hist[0];
offset_hist[0] = offset;
}
} else {
// When it's not a repeat offset:
// "if (Offset_Value > 3) offset = Offset_Value - 3;"
offset = seq.offset - 3;
// Shift back history
offset_hist[2] = offset_hist[1];
offset_hist[1] = offset_hist[0];
offset_hist[0] = offset;
}
return offset;
}
static void execute_match_copy(frame_context_t *const ctx, size_t offset,
size_t match_length, size_t total_output,
ostream_t *const out) {
u8 *write_ptr = IO_get_write_ptr(out, match_length);
if (total_output <= ctx->header.window_size) {
// In this case offset might go back into the dictionary
if (offset > total_output + ctx->dict_content_len) {
// The offset goes beyond even the dictionary
CORRUPTION();
}
if (offset > total_output) {
// "The rest of the dictionary is its content. The content act
// as a "past" in front of data to compress or decompress, so it
// can be referenced in sequence commands."
const size_t dict_copy =
MIN(offset - total_output, match_length);
const size_t dict_offset =
ctx->dict_content_len - (offset - total_output);
memcpy(write_ptr, ctx->dict_content + dict_offset, dict_copy);
write_ptr += dict_copy;
match_length -= dict_copy;
}
} else if (offset > ctx->header.window_size) {
CORRUPTION();
}
// We must copy byte by byte because the match length might be larger
// than the offset
// ex: if the output so far was "abc", a command with offset=3 and
// match_length=6 would produce "abcabcabc" as the new output
for (size_t j = 0; j < match_length; j++) {
*write_ptr = *(write_ptr - offset);
write_ptr++;
}
}
/******* END SEQUENCE EXECUTION ***********************************************/
/******* OUTPUT SIZE COUNTING *************************************************/
static void traverse_frame(const frame_header_t *const header, istream_t *const in);
/// Get the decompressed size of an input stream so memory can be allocated in
/// advance.
/// This is more complex than the implementation in the reference
/// implementation, as this API allows for the decompression of multiple
/// concatenated frames.
/// This implementation assumes `src` points to a single ZSTD-compressed frame
size_t ZSTD_get_decompressed_size(const void *src, const size_t src_len) {
istream_t in = IO_make_istream(src, src_len);
size_t dst_size = 0;
// Each frame header only gives us the size of its frame, so iterate over
// all
// frames
while (IO_istream_len(&in) > 0) {
// get decompressed size from ZSTD frame header
{
const u32 magic_number = IO_read_bits(&in, 32);
if ((magic_number & ~0xFU) == 0x184D2A50U) {
// skippable frame, this has no impact on output size
const size_t frame_size = IO_read_bits(&in, 32);
IO_advance_input(&in, frame_size);
} else if (magic_number == 0xFD2FB528U) {
if (magic_number == 0xFD2FB528U) {
// ZSTD frame
frame_header_t header;
parse_frame_header(&header, &in);
@@ -1451,68 +1421,42 @@ size_t ZSTD_get_decompressed_size(const void *src, const size_t src_len) {
return -1;
}
dst_size += header.frame_content_size;
// Consume the input from the frame to reach the start of the next
traverse_frame(&header, &in);
return header.frame_content_size;
} else {
// not a real frame
ERROR("Invalid magic number");
// not a real frame or skippable frame
ERROR("ZSTD frame magic number did not match");
}
}
return dst_size;
}
/// Iterate over each block in a frame to find the end of it, to get to the
/// start of the next frame
static void traverse_frame(const frame_header_t *const header, istream_t *const in) {
int last_block = 0;
do {
// Parse the block header
last_block = IO_read_bits(in, 1);
const int block_type = IO_read_bits(in, 2);
const size_t block_len = IO_read_bits(in, 21);
switch (block_type) {
case 0: // Raw block, block_len bytes
IO_advance_input(in, block_len);
break;
case 1: // RLE block, 1 byte
IO_advance_input(in, 1);
break;
case 2: // Compressed block, compressed size is block_len
IO_advance_input(in, block_len);
break;
case 3:
// Reserved block type
CORRUPTION();
break;
default:
IMPOSSIBLE();
}
} while (!last_block);
if (header->content_checksum_flag) {
IO_advance_input(in, 4);
}
}
/******* END OUTPUT SIZE COUNTING *********************************************/
/******* DICTIONARY PARSING ***************************************************/
#define DICT_SIZE_ERROR() ERROR("Dictionary size cannot be less than 8 bytes")
#define NULL_SRC() ERROR("Tried to create dictionary with pointer to null src");
dictionary_t* create_dictionary() {
dictionary_t* dict = calloc(1, sizeof(dictionary_t));
if (!dict) {
BAD_ALLOC();
}
return dict;
}
static void init_dictionary_content(dictionary_t *const dict,
istream_t *const in);
static void parse_dictionary(dictionary_t *const dict, const u8 *src,
void parse_dictionary(dictionary_t *const dict, const void *src,
size_t src_len) {
const u8 *byte_src = (const u8 *)src;
memset(dict, 0, sizeof(dictionary_t));
if (src == NULL) { /* cannot initialize dictionary with null src */
NULL_SRC();
}
if (src_len < 8) {
INP_SIZE();
DICT_SIZE_ERROR();
}
istream_t in = IO_make_istream(src, src_len);
istream_t in = IO_make_istream(byte_src, src_len);
const u32 magic_number = IO_read_bits(&in, 32);
if (magic_number != 0xEC30A437) {
@@ -1564,13 +1508,13 @@ static void init_dictionary_content(dictionary_t *const dict,
BAD_ALLOC();
}
const u8 *const content = IO_read_bytes(in, dict->content_size);
const u8 *const content = IO_get_read_ptr(in, dict->content_size);
memcpy(dict->content, content, dict->content_size);
}
/// Free an allocated dictionary
static void free_dictionary(dictionary_t *const dict) {
void free_dictionary(dictionary_t *const dict) {
HUF_free_dtable(&dict->literals_dtable);
FSE_free_dtable(&dict->ll_dtable);
FSE_free_dtable(&dict->of_dtable);
@@ -1579,6 +1523,8 @@ static void free_dictionary(dictionary_t *const dict) {
free(dict->content);
memset(dict, 0, sizeof(dictionary_t));
free(dict);
}
/******* END DICTIONARY PARSING ***********************************************/
@@ -1657,7 +1603,7 @@ static inline size_t IO_istream_len(const istream_t *const in) {
/// Returns a pointer where `len` bytes can be read, and advances the internal
/// state. The stream must be byte aligned.
static inline const u8 *IO_read_bytes(istream_t *const in, size_t len) {
static inline const u8 *IO_get_read_ptr(istream_t *const in, size_t len) {
if (len > in->len) {
INP_SIZE();
}
@@ -1671,7 +1617,7 @@ static inline const u8 *IO_read_bytes(istream_t *const in, size_t len) {
return ptr;
}
/// Returns a pointer to write `len` bytes to, and advances the internal state
static inline u8 *IO_write_bytes(ostream_t *const out, size_t len) {
static inline u8 *IO_get_write_ptr(ostream_t *const out, size_t len) {
if (len > out->len) {
OUT_SIZE();
}
@@ -1710,7 +1656,7 @@ static inline istream_t IO_make_istream(const u8 *in, size_t len) {
/// `in` must be byte aligned
static inline istream_t IO_make_sub_istream(istream_t *const in, size_t len) {
// Consume `len` bytes of the parent stream
const u8 *const ptr = IO_read_bytes(in, len);
const u8 *const ptr = IO_get_read_ptr(in, len);
// Make a substream using the pointer to those `len` bytes
return IO_make_istream(ptr, len);
@@ -1814,7 +1760,7 @@ static size_t HUF_decompress_1stream(const HUF_dtable *const dtable,
if (len == 0) {
INP_SIZE();
}
const u8 *const src = IO_read_bytes(in, len);
const u8 *const src = IO_get_read_ptr(in, len);
// "Each bitstream must be read backward, that is starting from the end down
// to the beginning. Therefore it's necessary to know the size of each
@@ -2065,7 +2011,7 @@ static size_t FSE_decompress_interleaved2(const FSE_dtable *const dtable,
if (len == 0) {
INP_SIZE();
}
const u8 *const src = IO_read_bytes(in, len);
const u8 *const src = IO_get_read_ptr(in, len);
// "Each bitstream must be read backward, that is starting from the end down
// to the beginning. Therefore it's necessary to know the size of each
@@ -2192,7 +2138,7 @@ static void FSE_init_dtable(FSE_dtable *const dtable,
}
// Now we can fill baseline and num bits
for (int i = 0; i < size; i++) {
for (size_t i = 0; i < size; i++) {
u8 symbol = dtable->symbols[i];
u16 next_state_desc = state_desc[symbol]++;
// Fills in the table appropriately, next_state_desc increases by symbol
@@ -2355,4 +2301,3 @@ static void FSE_copy_dtable(FSE_dtable *const dst, const FSE_dtable *const src)
memcpy(dst->new_state_base, src->new_state_base, size * sizeof(u16));
}
/******* END FSE PRIMITIVES ***************************************************/
doc/educational_decoder/zstd_decompress.h
+52 -10
View File
@@ -1,16 +1,58 @@
/*
* Copyright (c) 2017-present, Facebook, Inc.
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
size_t ZSTD_decompress(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len);
size_t ZSTD_decompress_with_dict(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len,
const void *const dict, const size_t dict_len);
size_t ZSTD_get_decompressed_size(const void *const src, const size_t src_len);
/******* EXPOSED TYPES ********************************************************/
/*
* Contains the parsed contents of a dictionary
* This includes Huffman and FSE tables used for decoding and data on offsets
*/
typedef struct dictionary_s dictionary_t;
/******* END EXPOSED TYPES ****************************************************/
/******* DECOMPRESSION FUNCTIONS **********************************************/
/// Zstandard decompression functions.
/// `dst` must point to a space at least as large as the reconstructed output.
size_t ZSTD_decompress(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len);
/// If `dict != NULL` and `dict_len >= 8`, does the same thing as
/// `ZSTD_decompress` but uses the provided dict
size_t ZSTD_decompress_with_dict(void *const dst, const size_t dst_len,
const void *const src, const size_t src_len,
dictionary_t* parsed_dict);
/// Get the decompressed size of an input stream so memory can be allocated in
/// advance
/// Returns -1 if the size can't be determined
/// Assumes decompression of a single frame
size_t ZSTD_get_decompressed_size(const void *const src, const size_t src_len);
/******* END DECOMPRESSION FUNCTIONS ******************************************/
/******* DICTIONARY MANAGEMENT ***********************************************/
/*
* Return a valid dictionary_t pointer for use with dictionary initialization
* or decompression
*/
dictionary_t* create_dictionary();
/*
* Parse a provided dictionary blob for use in decompression
* `src` -- must point to memory space representing the dictionary
* `src_len` -- must provide the dictionary size
* `dict` -- will contain the parsed contents of the dictionary and
* can be used for decompression
*/
void parse_dictionary(dictionary_t *const dict, const void *src,
size_t src_len);
/*
* Free internal Huffman tables, FSE tables, and dictionary content
*/
void free_dictionary(dictionary_t *const dict);
/******* END DICTIONARY MANAGEMENT *******************************************/
doc/zstd_compression_format.md
+23 -15
View File
@@ -16,7 +16,7 @@ Distribution of this document is unlimited.
### Version
0.2.5 (31/03/17)
0.2.6 (19/08/17)
Introduction
@@ -106,7 +106,7 @@ The structure of a single Zstandard frame is following:
| `Magic_Number` | `Frame_Header` |`Data_Block`| [More data blocks] | [`Content_Checksum`] |
|:--------------:|:--------------:|:----------:| ------------------ |:--------------------:|
| 4 bytes | 2-14 bytes | n bytes | | 0-4 bytes |
| 4 bytes | 2-14 bytes | n bytes | | 0-4 bytes |
__`Magic_Number`__
@@ -1249,23 +1249,30 @@ Consequently, a last byte of `0` is not possible.
And the final-bit-flag itself is not part of the useful bitstream.
Hence, the last byte contains between 0 and 7 useful bits.
For example, if the literal sequence "0145" was encoded using the prefix codes above,
it would be encoded as:
```
00000001 01110000
```
Starting from the end,
it's possible to read the bitstream in a __little-endian__ fashion,
keeping track of already used bits. Since the bitstream is encoded in reverse
order, starting from the end read symbols in forward order.
For example, if the literal sequence "0145" was encoded using above prefix code,
it would be encoded (in reverse order) as:
|Symbol | 5 | 4 | 1 | 0 | Padding |
|--------|------|------|----|---|---------|
|Encoding|`0000`|`0001`|`01`|`1`| `10000` |
|Encoding|`0000`|`0001`|`01`|`1`| `00001` |
Starting from the end,
it's possible to read the bitstream in a __little-endian__ fashion,
keeping track of already used bits. Since the bitstream is encoded in reverse
order, by starting at the end the symbols can be read in forward order.
Resulting in following 2-bytes bitstream :
```
00010000 00001101
```
Reading the last `Max_Number_of_Bits` bits,
it's then possible to compare extracted value to decoding table,
Here is an alternative representation with the symbol codes separated by underscore:
```
0001_0000 00001_1_01
```
Reading highest `Max_Number_of_Bits` bits,
it's possible to compare extracted value to decoding table,
determining the symbol to decode and number of bits to discard.
The process continues up to reading the required number of symbols per stream.
@@ -1516,12 +1523,13 @@ to crosscheck that an implementation build its decoding tables correctly.
Version changes
---------------
- 0.2.6 : fixed an error in huffman example, by Ulrich Kunitz
- 0.2.5 : minor typos and clarifications
- 0.2.4 : section restructuring, by Sean Purcell
- 0.2.3 : clarified several details, by Sean Purcell
- 0.2.2 : added predefined codes, by Johannes Rudolph
- 0.2.1 : clarify field names, by Przemyslaw Skibinski
- 0.2.0 : numerous format adjustments for zstd v0.8
- 0.2.0 : numerous format adjustments for zstd v0.8+
- 0.1.2 : limit Huffman tree depth to 11 bits
- 0.1.1 : reserved dictID ranges
- 0.1.0 : initial release
examples/Makefile
+3 -3
View File
@@ -2,9 +2,9 @@
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
# This Makefile presumes libzstd is installed, using `sudo make install`
examples/dictionary_compression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
examples/dictionary_decompression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
examples/multiple_streaming_compression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
examples/simple_compression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
examples/simple_decompression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdlib.h> // malloc, exit
examples/streaming_compression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
examples/streaming_decompression.c
+5 -4
View File
@@ -1,9 +1,10 @@
/**
* Copyright 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the license found in the
* LICENSE-examples file in the root directory of this source tree.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/Makefile
+7 -9
View File
@@ -1,13 +1,11 @@
# ##########################################################################
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# ################################################################
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This Makefile is validated for Linux, macOS, *BSD, Hurd, Solaris, MSYS2 targets
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# ##########################################################################
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
# Version numbers
LIBVER_MAJOR_SCRIPT:=`sed -n '/define ZSTD_VERSION_MAJOR/s/.*[[:blank:]]\([0-9][0-9]*\).*/\1/p' < ./zstd.h`
@@ -31,7 +29,7 @@ CFLAGS += $(DEBUGFLAGS) $(MOREFLAGS)
FLAGS = $(CPPFLAGS) $(CFLAGS)
ZSTD_FILES := $(wildcard common/*.c compress/*.c decompress/*.c dictBuilder/*.c deprecated/*.c)
ZSTD_FILES := $(sort $(wildcard common/*.c compress/*.c decompress/*.c dictBuilder/*.c deprecated/*.c))
ZSTD_LEGACY_SUPPORT ?= 4
lib/common/bitstream.h
+18 -15
View File
@@ -169,25 +169,28 @@ MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits);
****************************************************************/
MEM_STATIC unsigned BIT_highbit32 (register U32 val)
{
assert(val != 0);
{
# if defined(_MSC_VER) /* Visual */
unsigned long r=0;
_BitScanReverse ( &r, val );
return (unsigned) r;
unsigned long r=0;
_BitScanReverse ( &r, val );
return (unsigned) r;
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* Use GCC Intrinsic */
return 31 - __builtin_clz (val);
return 31 - __builtin_clz (val);
# else /* Software version */
static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
11, 14, 16, 18, 22, 25, 3, 30,
8, 12, 20, 28, 15, 17, 24, 7,
19, 27, 23, 6, 26, 5, 4, 31 };
U32 v = val;
v |= v >> 1;
v |= v >> 2;
v |= v >> 4;
v |= v >> 8;
v |= v >> 16;
return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
static const unsigned DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29,
11, 14, 16, 18, 22, 25, 3, 30,
8, 12, 20, 28, 15, 17, 24, 7,
19, 27, 23, 6, 26, 5, 4, 31 };
U32 v = val;
v |= v >> 1;
v |= v >> 2;
v |= v >> 4;
v |= v >> 8;
v |= v >> 16;
return DeBruijnClz[ (U32) (v * 0x07C4ACDDU) >> 27];
# endif
}
}
/*===== Local Constants =====*/
lib/common/compiler.h
+85
View File
@@ -0,0 +1,85 @@
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_COMPILER_H
#define ZSTD_COMPILER_H
/*-*******************************************************
* Compiler specifics
*********************************************************/
/* force inlining */
#if defined (__GNUC__) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# define INLINE_KEYWORD inline
#else
# define INLINE_KEYWORD
#endif
#if defined(__GNUC__)
# define FORCE_INLINE_ATTR __attribute__((always_inline))
#elif defined(_MSC_VER)
# define FORCE_INLINE_ATTR __forceinline
#else
# define FORCE_INLINE_ATTR
#endif
/**
* FORCE_INLINE_TEMPLATE is used to define C "templates", which take constant
* parameters. They must be inlined for the compiler to elimininate the constant
* branches.
*/
#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR
/**
* HINT_INLINE is used to help the compiler generate better code. It is *not*
* used for "templates", so it can be tweaked based on the compilers
* performance.
*
* gcc-4.8 and gcc-4.9 have been shown to benefit from leaving off the
* always_inline attribute.
*
* clang up to 5.0.0 (trunk) benefit tremendously from the always_inline
* attribute.
*/
#if !defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 4 && __GNUC_MINOR__ >= 8 && __GNUC__ < 5
# define HINT_INLINE static INLINE_KEYWORD
#else
# define HINT_INLINE static INLINE_KEYWORD FORCE_INLINE_ATTR
#endif
/* force no inlining */
#ifdef _MSC_VER
# define FORCE_NOINLINE static __declspec(noinline)
#else
# ifdef __GNUC__
# define FORCE_NOINLINE static __attribute__((__noinline__))
# else
# define FORCE_NOINLINE static
# endif
#endif
/* prefetch */
#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) /* _mm_prefetch() is not defined outside of x86/x64 */
# include <mmintrin.h> /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
# define PREFETCH(ptr) _mm_prefetch((const char*)ptr, _MM_HINT_T0)
#elif defined(__GNUC__)
# define PREFETCH(ptr) __builtin_prefetch(ptr, 0, 0)
#else
# define PREFETCH(ptr) /* disabled */
#endif
/* disable warnings */
#ifdef _MSC_VER /* Visual Studio */
# include <intrin.h> /* For Visual 2005 */
# pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */
# pragma warning(disable : 4324) /* disable: C4324: padded structure */
#endif
#endif /* ZSTD_COMPILER_H */
lib/common/error_private.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* The purpose of this file is to have a single list of error strings embedded in binary */
lib/common/error_private.h
+7 -8
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* Note : this module is expected to remain private, do not expose it */
@@ -48,10 +48,9 @@ typedef ZSTD_ErrorCode ERR_enum;
/*-****************************************
* Error codes handling
******************************************/
#ifdef ERROR
# undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
#endif
#define ERROR(name) ((size_t)-PREFIX(name))
#undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
#define ERROR(name) ZSTD_ERROR(name)
#define ZSTD_ERROR(name) ((size_t)-PREFIX(name))
ERR_STATIC unsigned ERR_isError(size_t code) { return (code > ERROR(maxCode)); }
lib/common/fse_decompress.c
+3 -22
View File
@@ -33,35 +33,16 @@
****************************************************************** */
/* **************************************************************
* Compiler specifics
****************************************************************/
#ifdef _MSC_VER /* Visual Studio */
# define FORCE_INLINE static __forceinline
# include <intrin.h> /* For Visual 2005 */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */
#else
# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
# else
# define FORCE_INLINE static
# endif /* __STDC_VERSION__ */
#endif
/* **************************************************************
* Includes
****************************************************************/
#include <stdlib.h> /* malloc, free, qsort */
#include <string.h> /* memcpy, memset */
#include "bitstream.h"
#include "compiler.h"
#define FSE_STATIC_LINKING_ONLY
#include "fse.h"
#include "error_private.h"
/* **************************************************************
@@ -216,7 +197,7 @@ size_t FSE_buildDTable_raw (FSE_DTable* dt, unsigned nbBits)
return 0;
}
FORCE_INLINE size_t FSE_decompress_usingDTable_generic(
FORCE_INLINE_TEMPLATE size_t FSE_decompress_usingDTable_generic(
void* dst, size_t maxDstSize,
const void* cSrc, size_t cSrcSize,
const FSE_DTable* dt, const unsigned fast)
lib/common/mem.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef MEM_H_MODULE
lib/common/pool.c
+76 -28
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -25,11 +25,12 @@
/* A job is a function and an opaque argument */
typedef struct POOL_job_s {
POOL_function function;
void *opaque;
POOL_function function;
void *opaque;
} POOL_job;
struct POOL_ctx_s {
ZSTD_customMem customMem;
/* Keep track of the threads */
pthread_t *threads;
size_t numThreads;
@@ -39,6 +40,12 @@ struct POOL_ctx_s {
size_t queueHead;
size_t queueTail;
size_t queueSize;
/* The number of threads working on jobs */
size_t numThreadsBusy;
/* Indicates if the queue is empty */
int queueEmpty;
/* The mutex protects the queue */
pthread_mutex_t queueMutex;
/* Condition variable for pushers to wait on when the queue is full */
@@ -60,32 +67,47 @@ static void* POOL_thread(void* opaque) {
for (;;) {
/* Lock the mutex and wait for a non-empty queue or until shutdown */
pthread_mutex_lock(&ctx->queueMutex);
while (ctx->queueHead == ctx->queueTail && !ctx->shutdown) {
while (ctx->queueEmpty && !ctx->shutdown) {
pthread_cond_wait(&ctx->queuePopCond, &ctx->queueMutex);
}
/* empty => shutting down: so stop */
if (ctx->queueHead == ctx->queueTail) {
if (ctx->queueEmpty) {
pthread_mutex_unlock(&ctx->queueMutex);
return opaque;
}
/* Pop a job off the queue */
{ POOL_job const job = ctx->queue[ctx->queueHead];
ctx->queueHead = (ctx->queueHead + 1) % ctx->queueSize;
ctx->numThreadsBusy++;
ctx->queueEmpty = ctx->queueHead == ctx->queueTail;
/* Unlock the mutex, signal a pusher, and run the job */
pthread_mutex_unlock(&ctx->queueMutex);
pthread_cond_signal(&ctx->queuePushCond);
job.function(job.opaque);
}
}
/* If the intended queue size was 0, signal after finishing job */
if (ctx->queueSize == 1) {
pthread_mutex_lock(&ctx->queueMutex);
ctx->numThreadsBusy--;
pthread_mutex_unlock(&ctx->queueMutex);
pthread_cond_signal(&ctx->queuePushCond);
} }
} /* for (;;) */
/* Unreachable */
}
POOL_ctx *POOL_create(size_t numThreads, size_t queueSize) {
return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
}
POOL_ctx *POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
POOL_ctx *ctx;
/* Check the parameters */
if (!numThreads || !queueSize) { return NULL; }
if (!numThreads) { return NULL; }
/* Allocate the context and zero initialize */
ctx = (POOL_ctx *)calloc(1, sizeof(POOL_ctx));
ctx = (POOL_ctx *)ZSTD_calloc(sizeof(POOL_ctx), customMem);
if (!ctx) { return NULL; }
/* Initialize the job queue.
* It needs one extra space since one space is wasted to differentiate empty
@@ -95,13 +117,16 @@ POOL_ctx *POOL_create(size_t numThreads, size_t queueSize) {
ctx->queue = (POOL_job*) malloc(ctx->queueSize * sizeof(POOL_job));
ctx->queueHead = 0;
ctx->queueTail = 0;
ctx->numThreadsBusy = 0;
ctx->queueEmpty = 1;
(void)pthread_mutex_init(&ctx->queueMutex, NULL);
(void)pthread_cond_init(&ctx->queuePushCond, NULL);
(void)pthread_cond_init(&ctx->queuePopCond, NULL);
ctx->shutdown = 0;
/* Allocate space for the thread handles */
ctx->threads = (pthread_t*)malloc(numThreads * sizeof(pthread_t));
ctx->threads = (pthread_t*)ZSTD_malloc(numThreads * sizeof(pthread_t), customMem);
ctx->numThreads = 0;
ctx->customMem = customMem;
/* Check for errors */
if (!ctx->threads || !ctx->queue) { POOL_free(ctx); return NULL; }
/* Initialize the threads */
@@ -141,9 +166,9 @@ void POOL_free(POOL_ctx *ctx) {
pthread_mutex_destroy(&ctx->queueMutex);
pthread_cond_destroy(&ctx->queuePushCond);
pthread_cond_destroy(&ctx->queuePopCond);
if (ctx->queue) free(ctx->queue);
if (ctx->threads) free(ctx->threads);
free(ctx);
ZSTD_free(ctx->queue, ctx->customMem);
ZSTD_free(ctx->threads, ctx->customMem);
ZSTD_free(ctx, ctx->customMem);
}
size_t POOL_sizeof(POOL_ctx *ctx) {
@@ -153,22 +178,37 @@ size_t POOL_sizeof(POOL_ctx *ctx) {
+ ctx->numThreads * sizeof(pthread_t);
}
/**
* Returns 1 if the queue is full and 0 otherwise.
*
* If the queueSize is 1 (the pool was created with an intended queueSize of 0),
* then a queue is empty if there is a thread free and no job is waiting.
*/
static int isQueueFull(POOL_ctx const* ctx) {
if (ctx->queueSize > 1) {
return ctx->queueHead == ((ctx->queueTail + 1) % ctx->queueSize);
} else {
return ctx->numThreadsBusy == ctx->numThreads ||
!ctx->queueEmpty;
}
}
void POOL_add(void* ctxVoid, POOL_function function, void *opaque) {
POOL_ctx* const ctx = (POOL_ctx*)ctxVoid;
if (!ctx) { return; }
pthread_mutex_lock(&ctx->queueMutex);
{ POOL_job const job = {function, opaque};
/* Wait until there is space in the queue for the new job */
size_t newTail = (ctx->queueTail + 1) % ctx->queueSize;
while (ctx->queueHead == newTail && !ctx->shutdown) {
while (isQueueFull(ctx) && !ctx->shutdown) {
pthread_cond_wait(&ctx->queuePushCond, &ctx->queueMutex);
newTail = (ctx->queueTail + 1) % ctx->queueSize;
}
/* The queue is still going => there is space */
if (!ctx->shutdown) {
ctx->queueEmpty = 0;
ctx->queue[ctx->queueTail] = job;
ctx->queueTail = newTail;
ctx->queueTail = (ctx->queueTail + 1) % ctx->queueSize;
}
}
pthread_mutex_unlock(&ctx->queueMutex);
@@ -180,26 +220,34 @@ void POOL_add(void* ctxVoid, POOL_function function, void *opaque) {
/* We don't need any data, but if it is empty malloc() might return NULL. */
struct POOL_ctx_s {
int data;
int dummy;
};
static POOL_ctx g_ctx;
POOL_ctx* POOL_create(size_t numThreads, size_t queueSize) {
(void)numThreads;
(void)queueSize;
return (POOL_ctx*)malloc(sizeof(POOL_ctx));
return POOL_create_advanced(numThreads, queueSize, ZSTD_defaultCMem);
}
POOL_ctx* POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem) {
(void)numThreads;
(void)queueSize;
(void)customMem;
return &g_ctx;
}
void POOL_free(POOL_ctx* ctx) {
free(ctx);
assert(ctx == &g_ctx);
(void)ctx;
}
void POOL_add(void* ctx, POOL_function function, void* opaque) {
(void)ctx;
function(opaque);
(void)ctx;
function(opaque);
}
size_t POOL_sizeof(POOL_ctx* ctx) {
if (ctx==NULL) return 0; /* supports sizeof NULL */
assert(ctx == &g_ctx);
return sizeof(*ctx);
}
lib/common/pool.h
+9 -6
View File
@@ -1,11 +1,12 @@
/**
* Copyright (c) 2016-present, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef POOL_H
#define POOL_H
@@ -15,6 +16,7 @@ extern "C" {
#include <stddef.h> /* size_t */
#include "zstd_internal.h" /* ZSTD_customMem */
typedef struct POOL_ctx_s POOL_ctx;
@@ -22,11 +24,12 @@ typedef struct POOL_ctx_s POOL_ctx;
* Create a thread pool with at most `numThreads` threads.
* `numThreads` must be at least 1.
* The maximum number of queued jobs before blocking is `queueSize`.
* `queueSize` must be at least 1.
* @return : POOL_ctx pointer on success, else NULL.
*/
POOL_ctx *POOL_create(size_t numThreads, size_t queueSize);
POOL_ctx *POOL_create_advanced(size_t numThreads, size_t queueSize, ZSTD_customMem customMem);
/*! POOL_free() :
Free a thread pool returned by POOL_create().
*/
lib/common/threading.c
+6 -6
View File
@@ -2,9 +2,9 @@
* Copyright (c) 2016 Tino Reichardt
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*
* You can contact the author at:
* - zstdmt source repository: https://github.com/mcmilk/zstdmt
@@ -15,9 +15,9 @@
*/
/* When ZSTD_MULTITHREAD is not defined, this file would become an empty translation unit.
* Include some ISO C header code to prevent this and portably avoid related warnings.
* (Visual C++: C4206 / GCC: -Wpedantic / Clang: -Wempty-translation-unit)
*/
* Include some ISO C header code to prevent this and portably avoid related warnings.
* (Visual C++: C4206 / GCC: -Wpedantic / Clang: -Wempty-translation-unit)
*/
#include <stddef.h>
lib/common/threading.h
+7 -4
View File
@@ -1,11 +1,10 @@
/**
* Copyright (c) 2016 Tino Reichardt
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*
* You can contact the author at:
* - zstdmt source repository: https://github.com/mcmilk/zstdmt
@@ -38,7 +37,11 @@ extern "C" {
# define WIN32_LEAN_AND_MEAN
#endif
#undef ERROR /* reported already defined on VS 2015 (Rich Geldreich) */
#include <windows.h>
#undef ERROR
#define ERROR(name) ZSTD_ERROR(name)
/* mutex */
#define pthread_mutex_t CRITICAL_SECTION
lib/common/xxhash.c
+28 -22
View File
@@ -113,19 +113,25 @@ static void* XXH_memcpy(void* dest, const void* src, size_t size) { return memcp
/* *************************************
* Compiler Specific Options
***************************************/
#ifdef _MSC_VER /* Visual Studio */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# define FORCE_INLINE static __forceinline
#if defined (__GNUC__) || defined(__cplusplus) || defined(__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# define INLINE_KEYWORD inline
#else
# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
# else
# define FORCE_INLINE static
# endif /* __STDC_VERSION__ */
# define INLINE_KEYWORD
#endif
#if defined(__GNUC__)
# define FORCE_INLINE_ATTR __attribute__((always_inline))
#elif defined(_MSC_VER)
# define FORCE_INLINE_ATTR __forceinline
#else
# define FORCE_INLINE_ATTR
#endif
#define FORCE_INLINE_TEMPLATE static INLINE_KEYWORD FORCE_INLINE_ATTR
#ifdef _MSC_VER
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
#endif
@@ -248,7 +254,7 @@ typedef enum { XXH_bigEndian=0, XXH_littleEndian=1 } XXH_endianess;
*****************************/
typedef enum { XXH_aligned, XXH_unaligned } XXH_alignment;
FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
FORCE_INLINE_TEMPLATE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? XXH_read32(ptr) : XXH_swap32(XXH_read32(ptr));
@@ -256,7 +262,7 @@ FORCE_INLINE U32 XXH_readLE32_align(const void* ptr, XXH_endianess endian, XXH_a
return endian==XXH_littleEndian ? *(const U32*)ptr : XXH_swap32(*(const U32*)ptr);
}
FORCE_INLINE U32 XXH_readLE32(const void* ptr, XXH_endianess endian)
FORCE_INLINE_TEMPLATE U32 XXH_readLE32(const void* ptr, XXH_endianess endian)
{
return XXH_readLE32_align(ptr, endian, XXH_unaligned);
}
@@ -266,7 +272,7 @@ static U32 XXH_readBE32(const void* ptr)
return XXH_CPU_LITTLE_ENDIAN ? XXH_swap32(XXH_read32(ptr)) : XXH_read32(ptr);
}
FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
FORCE_INLINE_TEMPLATE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_alignment align)
{
if (align==XXH_unaligned)
return endian==XXH_littleEndian ? XXH_read64(ptr) : XXH_swap64(XXH_read64(ptr));
@@ -274,7 +280,7 @@ FORCE_INLINE U64 XXH_readLE64_align(const void* ptr, XXH_endianess endian, XXH_a
return endian==XXH_littleEndian ? *(const U64*)ptr : XXH_swap64(*(const U64*)ptr);
}
FORCE_INLINE U64 XXH_readLE64(const void* ptr, XXH_endianess endian)
FORCE_INLINE_TEMPLATE U64 XXH_readLE64(const void* ptr, XXH_endianess endian)
{
return XXH_readLE64_align(ptr, endian, XXH_unaligned);
}
@@ -335,7 +341,7 @@ static U32 XXH32_round(U32 seed, U32 input)
return seed;
}
FORCE_INLINE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_endianess endian, XXH_alignment align)
FORCE_INLINE_TEMPLATE U32 XXH32_endian_align(const void* input, size_t len, U32 seed, XXH_endianess endian, XXH_alignment align)
{
const BYTE* p = (const BYTE*)input;
const BYTE* bEnd = p + len;
@@ -435,7 +441,7 @@ static U64 XXH64_mergeRound(U64 acc, U64 val)
return acc;
}
FORCE_INLINE U64 XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_endianess endian, XXH_alignment align)
FORCE_INLINE_TEMPLATE U64 XXH64_endian_align(const void* input, size_t len, U64 seed, XXH_endianess endian, XXH_alignment align)
{
const BYTE* p = (const BYTE*)input;
const BYTE* const bEnd = p + len;
@@ -584,7 +590,7 @@ XXH_PUBLIC_API XXH_errorcode XXH64_reset(XXH64_state_t* statePtr, unsigned long
}
FORCE_INLINE XXH_errorcode XXH32_update_endian (XXH32_state_t* state, const void* input, size_t len, XXH_endianess endian)
FORCE_INLINE_TEMPLATE XXH_errorcode XXH32_update_endian (XXH32_state_t* state, const void* input, size_t len, XXH_endianess endian)
{
const BYTE* p = (const BYTE*)input;
const BYTE* const bEnd = p + len;
@@ -654,7 +660,7 @@ XXH_PUBLIC_API XXH_errorcode XXH32_update (XXH32_state_t* state_in, const void*
FORCE_INLINE U32 XXH32_digest_endian (const XXH32_state_t* state, XXH_endianess endian)
FORCE_INLINE_TEMPLATE U32 XXH32_digest_endian (const XXH32_state_t* state, XXH_endianess endian)
{
const BYTE * p = (const BYTE*)state->mem32;
const BYTE* const bEnd = (const BYTE*)(state->mem32) + state->memsize;
@@ -704,7 +710,7 @@ XXH_PUBLIC_API unsigned int XXH32_digest (const XXH32_state_t* state_in)
/* **** XXH64 **** */
FORCE_INLINE XXH_errorcode XXH64_update_endian (XXH64_state_t* state, const void* input, size_t len, XXH_endianess endian)
FORCE_INLINE_TEMPLATE XXH_errorcode XXH64_update_endian (XXH64_state_t* state, const void* input, size_t len, XXH_endianess endian)
{
const BYTE* p = (const BYTE*)input;
const BYTE* const bEnd = p + len;
@@ -771,7 +777,7 @@ XXH_PUBLIC_API XXH_errorcode XXH64_update (XXH64_state_t* state_in, const void*
FORCE_INLINE U64 XXH64_digest_endian (const XXH64_state_t* state, XXH_endianess endian)
FORCE_INLINE_TEMPLATE U64 XXH64_digest_endian (const XXH64_state_t* state, XXH_endianess endian)
{
const BYTE * p = (const BYTE*)state->mem64;
const BYTE* const bEnd = (const BYTE*)state->mem64 + state->memsize;
lib/common/zstd_common.c
+5 -6
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -15,8 +15,7 @@
#include <stdlib.h> /* malloc, calloc, free */
#include <string.h> /* memset */
#include "error_private.h"
#define ZSTD_STATIC_LINKING_ONLY
#include "zstd.h"
#include "zstd_internal.h"
/*-****************************************
lib/common/zstd_errors.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_ERRORS_H_398273423
lib/common/zstd_internal.h
+30 -50
View File
@@ -1,51 +1,20 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_CCOMMON_H_MODULE
#define ZSTD_CCOMMON_H_MODULE
/*-*******************************************************
* Compiler specifics
*********************************************************/
#ifdef _MSC_VER /* Visual Studio */
# define FORCE_INLINE static __forceinline
# include <intrin.h> /* For Visual 2005 */
# pragma warning(disable : 4100) /* disable: C4100: unreferenced formal parameter */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4204) /* disable: C4204: non-constant aggregate initializer */
# pragma warning(disable : 4324) /* disable: C4324: padded structure */
#else
# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
# else
# define FORCE_INLINE static
# endif /* __STDC_VERSION__ */
#endif
#ifdef _MSC_VER
# define FORCE_NOINLINE static __declspec(noinline)
#else
# ifdef __GNUC__
# define FORCE_NOINLINE static __attribute__((__noinline__))
# else
# define FORCE_NOINLINE static
# endif
#endif
/*-*************************************
* Dependencies
***************************************/
#include "compiler.h"
#include "mem.h"
#include "error_private.h"
#define ZSTD_STATIC_LINKING_ONLY
@@ -60,6 +29,11 @@
#include "xxhash.h" /* XXH_reset, update, digest */
#if defined (__cplusplus)
extern "C" {
#endif
/*-*************************************
* Debug
***************************************/
@@ -317,24 +291,27 @@ void ZSTD_free(void* ptr, ZSTD_customMem customMem);
MEM_STATIC U32 ZSTD_highbit32(U32 val)
{
assert(val != 0);
{
# if defined(_MSC_VER) /* Visual */
unsigned long r=0;
_BitScanReverse(&r, val);
return (unsigned)r;
unsigned long r=0;
_BitScanReverse(&r, val);
return (unsigned)r;
# elif defined(__GNUC__) && (__GNUC__ >= 3) /* GCC Intrinsic */
return 31 - __builtin_clz(val);
return 31 - __builtin_clz(val);
# else /* Software version */
static const int DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
U32 v = val;
int r;
v |= v >> 1;
v |= v >> 2;
v |= v >> 4;
v |= v >> 8;
v |= v >> 16;
r = DeBruijnClz[(U32)(v * 0x07C4ACDDU) >> 27];
return r;
static const int DeBruijnClz[32] = { 0, 9, 1, 10, 13, 21, 2, 29, 11, 14, 16, 18, 22, 25, 3, 30, 8, 12, 20, 28, 15, 17, 24, 7, 19, 27, 23, 6, 26, 5, 4, 31 };
U32 v = val;
int r;
v |= v >> 1;
v |= v >> 2;
v |= v >> 4;
v |= v >> 8;
v |= v >> 16;
r = DeBruijnClz[(U32)(v * 0x07C4ACDDU) >> 27];
return r;
# endif
}
}
@@ -395,5 +372,8 @@ typedef struct {
size_t ZSTD_getcBlockSize(const void* src, size_t srcSize,
blockProperties_t* bpPtr);
#if defined (__cplusplus)
}
#endif
#endif /* ZSTD_CCOMMON_H_MODULE */
lib/compress/fse_compress.c
+3 -21
View File
@@ -32,27 +32,6 @@
- Public forum : https://groups.google.com/forum/#!forum/lz4c
****************************************************************** */
/* **************************************************************
* Compiler specifics
****************************************************************/
#ifdef _MSC_VER /* Visual Studio */
# define FORCE_INLINE static __forceinline
# include <intrin.h> /* For Visual 2005 */
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
# pragma warning(disable : 4214) /* disable: C4214: non-int bitfields */
#else
# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
# else
# define FORCE_INLINE static
# endif /* __STDC_VERSION__ */
#endif
/* **************************************************************
* Includes
****************************************************************/
@@ -60,13 +39,16 @@
#include <string.h> /* memcpy, memset */
#include <stdio.h> /* printf (debug) */
#include "bitstream.h"
#include "compiler.h"
#define FSE_STATIC_LINKING_ONLY
#include "fse.h"
#include "error_private.h"
/* **************************************************************
* Error Management
****************************************************************/
#define FSE_isError ERR_isError
#define FSE_STATIC_ASSERT(c) { enum { FSE_static_assert = 1/(int)(!!(c)) }; } /* use only *after* variable declarations */
lib/compress/huf_compress.c
+2
View File
@@ -50,11 +50,13 @@
#include "fse.h" /* header compression */
#define HUF_STATIC_LINKING_ONLY
#include "huf.h"
#include "error_private.h"
/* **************************************************************
* Error Management
****************************************************************/
#define HUF_isError ERR_isError
#define HUF_STATIC_ASSERT(c) { enum { HUF_static_assert = 1/(int)(!!(c)) }; } /* use only *after* variable declarations */
#define CHECK_V_F(e, f) size_t const e = f; if (ERR_isError(e)) return e
#define CHECK_F(f) { CHECK_V_F(_var_err__, f); }
lib/compress/zstd_compress.c
+49 -33
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -635,7 +635,8 @@ ZSTD_compressionParameters ZSTD_adjustCParams_internal(ZSTD_compressionParameter
{ U32 const minSrcSize = (srcSize==0) ? 500 : 0;
U64 const rSize = srcSize + dictSize + minSrcSize;
if (rSize < ((U64)1<<ZSTD_WINDOWLOG_MAX)) {
U32 const srcLog = MAX(ZSTD_HASHLOG_MIN, ZSTD_highbit32((U32)(rSize)-1) + 1);
U32 const srcLog =
MAX(ZSTD_HASHLOG_MIN, (rSize==1) ? 1 : ZSTD_highbit32((U32)(rSize)-1) + 1);
if (cPar.windowLog > srcLog) cPar.windowLog = srcLog;
} }
if (cPar.hashLog > cPar.windowLog) cPar.hashLog = cPar.windowLog;
@@ -1106,7 +1107,7 @@ static size_t ZSTD_compressLiterals (ZSTD_entropyCTables_t * entropy,
else { entropy->hufCTable_repeatMode = HUF_repeat_check; } /* now have a table to reuse */
}
if ((cLitSize==0) | (cLitSize >= srcSize - minGain)) {
if ((cLitSize==0) | (cLitSize >= srcSize - minGain) | ERR_isError(cLitSize)) {
entropy->hufCTable_repeatMode = HUF_repeat_none;
return ZSTD_noCompressLiterals(dst, dstCapacity, src, srcSize);
}
@@ -1295,7 +1296,7 @@ MEM_STATIC size_t ZSTD_encodeSequences(void* dst, size_t dstCapacity,
BIT_addBits(&blockStream, sequences[n].litLength, llBits);
if (MEM_32bits() && ((llBits+mlBits)>24)) BIT_flushBits(&blockStream);
BIT_addBits(&blockStream, sequences[n].matchLength, mlBits);
if (MEM_32bits()) BIT_flushBits(&blockStream); /* (7)*/
if (MEM_32bits() || (ofBits+mlBits+llBits > 56)) BIT_flushBits(&blockStream);
if (longOffsets) {
int const extraBits = ofBits - MIN(ofBits, STREAM_ACCUMULATOR_MIN-1);
if (extraBits) {
@@ -1320,11 +1321,10 @@ MEM_STATIC size_t ZSTD_encodeSequences(void* dst, size_t dstCapacity,
}
}
MEM_STATIC size_t ZSTD_compressSequences (seqStore_t* seqStorePtr,
MEM_STATIC size_t ZSTD_compressSequences_internal(seqStore_t* seqStorePtr,
ZSTD_entropyCTables_t* entropy,
ZSTD_compressionParameters const* cParams,
void* dst, size_t dstCapacity,
size_t srcSize)
void* dst, size_t dstCapacity)
{
const int longOffsets = cParams->windowLog > STREAM_ACCUMULATOR_MIN;
U32 count[MaxSeq+1];
@@ -1359,7 +1359,7 @@ MEM_STATIC size_t ZSTD_compressSequences (seqStore_t* seqStorePtr,
if (nbSeq < 0x7F) *op++ = (BYTE)nbSeq;
else if (nbSeq < LONGNBSEQ) op[0] = (BYTE)((nbSeq>>8) + 0x80), op[1] = (BYTE)nbSeq, op+=2;
else op[0]=0xFF, MEM_writeLE16(op+1, (U16)(nbSeq - LONGNBSEQ)), op+=3;
if (nbSeq==0) goto _check_compressibility;
if (nbSeq==0) return op - ostart;
/* seqHead : flags for FSE encoding type */
seqHead = op++;
@@ -1408,23 +1408,40 @@ MEM_STATIC size_t ZSTD_compressSequences (seqStore_t* seqStorePtr,
op += streamSize;
}
return op - ostart;
}
/* check compressibility */
_check_compressibility:
{ size_t const minGain = ZSTD_minGain(srcSize);
size_t const maxCSize = srcSize - minGain;
if ((size_t)(op-ostart) >= maxCSize) {
entropy->hufCTable_repeatMode = HUF_repeat_none;
entropy->offcode_repeatMode = FSE_repeat_none;
entropy->matchlength_repeatMode = FSE_repeat_none;
entropy->litlength_repeatMode = FSE_repeat_none;
return 0;
} }
MEM_STATIC size_t ZSTD_compressSequences(seqStore_t* seqStorePtr,
ZSTD_entropyCTables_t* entropy,
ZSTD_compressionParameters const* cParams,
void* dst, size_t dstCapacity,
size_t srcSize)
{
size_t const cSize = ZSTD_compressSequences_internal(seqStorePtr, entropy, cParams,
dst, dstCapacity);
size_t const minGain = ZSTD_minGain(srcSize);
size_t const maxCSize = srcSize - minGain;
/* If the srcSize <= dstCapacity, then there is enough space to write a
* raw uncompressed block. Since we ran out of space, the block must not
* be compressible, so fall back to a raw uncompressed block.
*/
int const uncompressibleError = cSize == ERROR(dstSize_tooSmall) && srcSize <= dstCapacity;
if (ZSTD_isError(cSize) && !uncompressibleError)
return cSize;
/* Check compressibility */
if (cSize >= maxCSize || uncompressibleError) {
entropy->hufCTable_repeatMode = HUF_repeat_none;
entropy->offcode_repeatMode = FSE_repeat_none;
entropy->matchlength_repeatMode = FSE_repeat_none;
entropy->litlength_repeatMode = FSE_repeat_none;
return 0;
}
assert(!ZSTD_isError(cSize));
/* confirm repcodes */
{ int i; for (i=0; i<ZSTD_REP_NUM; i++) seqStorePtr->rep[i] = seqStorePtr->repToConfirm[i]; }
return op - ostart;
return cSize;
}
@@ -1631,7 +1648,7 @@ static void ZSTD_fillHashTable (ZSTD_CCtx* zc, const void* end, const U32 mls)
}
FORCE_INLINE
FORCE_INLINE_TEMPLATE
void ZSTD_compressBlock_fast_generic(ZSTD_CCtx* cctx,
const void* src, size_t srcSize,
const U32 mls)
@@ -1874,7 +1891,7 @@ static void ZSTD_fillDoubleHashTable (ZSTD_CCtx* cctx, const void* end, const U3
}
FORCE_INLINE
FORCE_INLINE_TEMPLATE
void ZSTD_compressBlock_doubleFast_generic(ZSTD_CCtx* cctx,
const void* src, size_t srcSize,
const U32 mls)
@@ -2432,7 +2449,7 @@ static size_t ZSTD_BtFindBestMatch_selectMLS_extDict (
/* Update chains up to ip (excluded)
Assumption : always within prefix (i.e. not within extDict) */
FORCE_INLINE
FORCE_INLINE_TEMPLATE
U32 ZSTD_insertAndFindFirstIndex (ZSTD_CCtx* zc, const BYTE* ip, U32 mls)
{
U32* const hashTable = zc->hashTable;
@@ -2456,7 +2473,7 @@ U32 ZSTD_insertAndFindFirstIndex (ZSTD_CCtx* zc, const BYTE* ip, U32 mls)
/* inlining is important to hardwire a hot branch (template emulation) */
FORCE_INLINE
FORCE_INLINE_TEMPLATE
size_t ZSTD_HcFindBestMatch_generic (
ZSTD_CCtx* zc, /* Index table will be updated */
const BYTE* const ip, const BYTE* const iLimit,
@@ -2508,7 +2525,7 @@ size_t ZSTD_HcFindBestMatch_generic (
}
FORCE_INLINE size_t ZSTD_HcFindBestMatch_selectMLS (
FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_selectMLS (
ZSTD_CCtx* zc,
const BYTE* ip, const BYTE* const iLimit,
size_t* offsetPtr,
@@ -2525,7 +2542,7 @@ FORCE_INLINE size_t ZSTD_HcFindBestMatch_selectMLS (
}
FORCE_INLINE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
FORCE_INLINE_TEMPLATE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
ZSTD_CCtx* zc,
const BYTE* ip, const BYTE* const iLimit,
size_t* offsetPtr,
@@ -2545,7 +2562,7 @@ FORCE_INLINE size_t ZSTD_HcFindBestMatch_extDict_selectMLS (
/* *******************************
* Common parser - lazy strategy
*********************************/
FORCE_INLINE
FORCE_INLINE_TEMPLATE
void ZSTD_compressBlock_lazy_generic(ZSTD_CCtx* ctx,
const void* src, size_t srcSize,
const U32 searchMethod, const U32 depth)
@@ -2707,7 +2724,7 @@ static void ZSTD_compressBlock_greedy(ZSTD_CCtx* ctx, const void* src, size_t sr
}
FORCE_INLINE
FORCE_INLINE_TEMPLATE
void ZSTD_compressBlock_lazy_extDict_generic(ZSTD_CCtx* ctx,
const void* src, size_t srcSize,
const U32 searchMethod, const U32 depth)
@@ -3156,7 +3173,6 @@ static size_t ZSTD_compressContinue_internal (ZSTD_CCtx* cctx,
return fhSize;
}
size_t ZSTD_compressContinue (ZSTD_CCtx* cctx,
void* dst, size_t dstCapacity,
const void* src, size_t srcSize)
lib/compress/zstd_opt.h
+13 -13
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -22,7 +22,7 @@
/*-*************************************
* Price functions for optimal parser
***************************************/
FORCE_INLINE void ZSTD_setLog2Prices(optState_t* optPtr)
static void ZSTD_setLog2Prices(optState_t* optPtr)
{
optPtr->log2matchLengthSum = ZSTD_highbit32(optPtr->matchLengthSum+1);
optPtr->log2litLengthSum = ZSTD_highbit32(optPtr->litLengthSum+1);
@@ -32,7 +32,7 @@ FORCE_INLINE void ZSTD_setLog2Prices(optState_t* optPtr)
}
MEM_STATIC void ZSTD_rescaleFreqs(optState_t* optPtr, const BYTE* src, size_t srcSize)
static void ZSTD_rescaleFreqs(optState_t* optPtr, const BYTE* src, size_t srcSize)
{
unsigned u;
@@ -96,7 +96,7 @@ MEM_STATIC void ZSTD_rescaleFreqs(optState_t* optPtr, const BYTE* src, size_t sr
}
FORCE_INLINE U32 ZSTD_getLiteralPrice(optState_t* optPtr, U32 litLength, const BYTE* literals)
static U32 ZSTD_getLiteralPrice(optState_t* optPtr, U32 litLength, const BYTE* literals)
{
U32 price, u;
@@ -137,7 +137,7 @@ FORCE_INLINE U32 ZSTD_getLiteralPrice(optState_t* optPtr, U32 litLength, const B
}
FORCE_INLINE U32 ZSTD_getPrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength, const int ultra)
FORCE_INLINE_TEMPLATE U32 ZSTD_getPrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength, const int ultra)
{
/* offset */
U32 price;
@@ -159,7 +159,7 @@ FORCE_INLINE U32 ZSTD_getPrice(optState_t* optPtr, U32 litLength, const BYTE* li
}
MEM_STATIC void ZSTD_updatePrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength)
static void ZSTD_updatePrice(optState_t* optPtr, U32 litLength, const BYTE* literals, U32 offset, U32 matchLength)
{
U32 u;
@@ -203,7 +203,7 @@ MEM_STATIC void ZSTD_updatePrice(optState_t* optPtr, U32 litLength, const BYTE*
/* function safe only for comparisons */
MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
static U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
{
switch (length)
{
@@ -219,7 +219,7 @@ MEM_STATIC U32 ZSTD_readMINMATCH(const void* memPtr, U32 length)
/* Update hashTable3 up to ip (excluded)
Assumption : always within prefix (i.e. not within extDict) */
FORCE_INLINE
static
U32 ZSTD_insertAndFindFirstIndexHash3 (ZSTD_CCtx* zc, const BYTE* ip)
{
U32* const hashTable3 = zc->hashTable3;
@@ -412,7 +412,7 @@ static U32 ZSTD_BtGetAllMatches_selectMLS_extDict (
/*-*******************************
* Optimal parser
*********************************/
FORCE_INLINE
FORCE_INLINE_TEMPLATE
void ZSTD_compressBlock_opt_generic(ZSTD_CCtx* ctx,
const void* src, size_t srcSize, const int ultra)
{
@@ -662,7 +662,7 @@ _storeSequence: /* cur, last_pos, best_mlen, best_off have to be set */
}
FORCE_INLINE
FORCE_INLINE_TEMPLATE
void ZSTD_compressBlock_opt_extDict_generic(ZSTD_CCtx* ctx,
const void* src, size_t srcSize, const int ultra)
{
lib/compress/zstdmt_compress.c
+7 -5
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -447,7 +447,9 @@ ZSTDMT_CCtx* ZSTDMT_createCCtx_advanced(unsigned nbThreads, ZSTD_customMem cMem)
ZSTDMT_initializeCCtxParameters(&mtctx->params, nbThreads);
mtctx->cMem = cMem;
mtctx->allJobsCompleted = 1;
mtctx->factory = POOL_create(nbThreads, 1);
mtctx->sectionSize = 0;
mtctx->overlapLog = ZSTDMT_OVERLAPLOG_DEFAULT;
mtctx->factory = POOL_create_advanced(nbThreads, 0, cMem);
mtctx->jobs = ZSTDMT_allocJobsTable(&nbJobs, cMem);
mtctx->jobIDMask = nbJobs - 1;
mtctx->bufPool = ZSTDMT_createBufferPool(nbThreads, cMem);
lib/compress/zstdmt_compress.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTDMT_COMPRESS_H
lib/decompress/huf_decompress.c
+8 -24
View File
@@ -32,38 +32,22 @@
- Public forum : https://groups.google.com/forum/#!forum/lz4c
****************************************************************** */
/* **************************************************************
* Compiler specifics
****************************************************************/
#ifdef _MSC_VER /* Visual Studio */
# define FORCE_INLINE static __forceinline
# pragma warning(disable : 4127) /* disable: C4127: conditional expression is constant */
#else
# if defined (__cplusplus) || defined (__STDC_VERSION__) && __STDC_VERSION__ >= 199901L /* C99 */
# ifdef __GNUC__
# define FORCE_INLINE static inline __attribute__((always_inline))
# else
# define FORCE_INLINE static inline
# endif
# else
# define FORCE_INLINE static
# endif /* __STDC_VERSION__ */
#endif
/* **************************************************************
* Dependencies
****************************************************************/
#include <string.h> /* memcpy, memset */
#include "bitstream.h" /* BIT_* */
#include "compiler.h"
#include "fse.h" /* header compression */
#define HUF_STATIC_LINKING_ONLY
#include "huf.h"
#include "error_private.h"
/* **************************************************************
* Error Management
****************************************************************/
#define HUF_isError ERR_isError
#define HUF_STATIC_ASSERT(c) { enum { HUF_static_assert = 1/(int)(!!(c)) }; } /* use only *after* variable declarations */
@@ -180,7 +164,7 @@ static BYTE HUF_decodeSymbolX2(BIT_DStream_t* Dstream, const HUF_DEltX2* dt, con
if (MEM_64bits()) \
HUF_DECODE_SYMBOLX2_0(ptr, DStreamPtr)
FORCE_INLINE size_t HUF_decodeStreamX2(BYTE* p, BIT_DStream_t* const bitDPtr, BYTE* const pEnd, const HUF_DEltX2* const dt, const U32 dtLog)
HINT_INLINE size_t HUF_decodeStreamX2(BYTE* p, BIT_DStream_t* const bitDPtr, BYTE* const pEnd, const HUF_DEltX2* const dt, const U32 dtLog)
{
BYTE* const pStart = p;
@@ -639,7 +623,7 @@ static U32 HUF_decodeLastSymbolX4(void* op, BIT_DStream_t* DStream, const HUF_DE
if (MEM_64bits()) \
ptr += HUF_decodeSymbolX4(ptr, DStreamPtr, dt, dtLog)
FORCE_INLINE size_t HUF_decodeStreamX4(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd, const HUF_DEltX4* const dt, const U32 dtLog)
HINT_INLINE size_t HUF_decodeStreamX4(BYTE* p, BIT_DStream_t* bitDPtr, BYTE* const pEnd, const HUF_DEltX4* const dt, const U32 dtLog)
{
BYTE* const pStart = p;
@@ -917,11 +901,11 @@ static const algo_time_t algoTime[16 /* Quantization */][3 /* single, double, qu
* Tells which decoder is likely to decode faster,
* based on a set of pre-determined metrics.
* @return : 0==HUF_decompress4X2, 1==HUF_decompress4X4 .
* Assumption : 0 < cSrcSize < dstSize <= 128 KB */
* Assumption : 0 < cSrcSize, dstSize <= 128 KB */
U32 HUF_selectDecoder (size_t dstSize, size_t cSrcSize)
{
/* decoder timing evaluation */
U32 const Q = (U32)(cSrcSize * 16 / dstSize); /* Q < 16 since dstSize > cSrcSize */
U32 const Q = cSrcSize >= dstSize ? 15 : (U32)(cSrcSize * 16 / dstSize); /* Q < 16 */
U32 const D256 = (U32)(dstSize >> 8);
U32 const DTime0 = algoTime[Q][0].tableTime + (algoTime[Q][0].decode256Time * D256);
U32 DTime1 = algoTime[Q][1].tableTime + (algoTime[Q][1].decode256Time * D256);
@@ -977,7 +961,7 @@ size_t HUF_decompress4X_hufOnly_wksp(HUF_DTable* dctx, void* dst,
{
/* validation checks */
if (dstSize == 0) return ERROR(dstSize_tooSmall);
if ((cSrcSize >= dstSize) || (cSrcSize <= 1)) return ERROR(corruption_detected); /* invalid */
if (cSrcSize == 0) return ERROR(corruption_detected);
{ U32 const algoNb = HUF_selectDecoder(dstSize, cSrcSize);
return algoNb ? HUF_decompress4X4_DCtx_wksp(dctx, dst, dstSize, cSrc, cSrcSize, workSpace, wkspSize):
lib/decompress/zstd_decompress.c
+13 -24
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -53,15 +53,6 @@
# include "zstd_legacy.h"
#endif
#if defined(_MSC_VER) && (defined(_M_X64) || defined(_M_I86)) /* _mm_prefetch() is not defined outside of x86/x64 */
# include <mmintrin.h> /* https://msdn.microsoft.com/fr-fr/library/84szxsww(v=vs.90).aspx */
# define ZSTD_PREFETCH(ptr) _mm_prefetch((const char*)ptr, _MM_HINT_T0)
#elif defined(__GNUC__)
# define ZSTD_PREFETCH(ptr) __builtin_prefetch(ptr, 0, 0)
#else
# define ZSTD_PREFETCH(ptr) /* disabled */
#endif
/*-*************************************
* Errors
@@ -953,7 +944,7 @@ static seq_t ZSTD_decodeSequence(seqState_t* seqState)
}
FORCE_INLINE
HINT_INLINE
size_t ZSTD_execSequence(BYTE* op,
BYTE* const oend, seq_t sequence,
const BYTE** litPtr, const BYTE* const litLimit,
@@ -1097,7 +1088,7 @@ static size_t ZSTD_decompressSequences(
}
FORCE_INLINE seq_t ZSTD_decodeSequenceLong_generic(seqState_t* seqState, int const longOffsets)
FORCE_INLINE_TEMPLATE seq_t ZSTD_decodeSequenceLong_generic(seqState_t* seqState, int const longOffsets)
{
seq_t seq;
@@ -1197,7 +1188,7 @@ static seq_t ZSTD_decodeSequenceLong(seqState_t* seqState, unsigned const window
}
}
FORCE_INLINE
HINT_INLINE
size_t ZSTD_execSequenceLong(BYTE* op,
BYTE* const oend, seq_t sequence,
const BYTE** litPtr, const BYTE* const litLimit,
@@ -1333,7 +1324,7 @@ static size_t ZSTD_decompressSequencesLong(
seq_t const sequence = ZSTD_decodeSequenceLong(&seqState, windowSize32);
size_t const oneSeqSize = ZSTD_execSequenceLong(op, oend, sequences[(seqNb-ADVANCED_SEQS) & STOSEQ_MASK], &litPtr, litEnd, base, vBase, dictEnd);
if (ZSTD_isError(oneSeqSize)) return oneSeqSize;
ZSTD_PREFETCH(sequence.match);
PREFETCH(sequence.match);
sequences[seqNb&STOSEQ_MASK] = sequence;
op += oneSeqSize;
}
@@ -1731,7 +1722,7 @@ size_t ZSTD_decompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, c
return 0;
}
dctx->expected = 0; /* not necessary to copy more */
/* fall-through */
case ZSTDds_decodeFrameHeader:
assert(src != NULL);
memcpy(dctx->headerBuffer + ZSTD_frameHeaderSize_prefix, src, dctx->expected);
@@ -2245,7 +2236,7 @@ size_t ZSTD_estimateDStreamSize(size_t windowSize)
size_t const blockSize = MIN(windowSize, ZSTD_BLOCKSIZE_MAX);
size_t const inBuffSize = blockSize; /* no block can be larger */
size_t const outBuffSize = windowSize + blockSize + (WILDCOPY_OVERLENGTH * 2);
return sizeof(ZSTD_DStream) + ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize;
return ZSTD_estimateDCtxSize() + inBuffSize + outBuffSize;
}
ZSTDLIB_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize)
@@ -2392,7 +2383,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
zds->outBuffSize = neededOutSize;
} }
zds->streamStage = zdss_read;
/* pass-through */
/* fall-through */
case zdss_read:
DEBUGLOG(5, "stage zdss_read");
@@ -2417,8 +2408,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
} }
if (ip==iend) { someMoreWork = 0; break; } /* no more input */
zds->streamStage = zdss_load;
/* pass-through */
/* fall-through */
case zdss_load:
{ size_t const neededInSize = ZSTD_nextSrcSizeToDecompress(zds);
size_t const toLoad = neededInSize - zds->inPos; /* should always be <= remaining space within inBuff */
@@ -2440,8 +2430,7 @@ size_t ZSTD_decompressStream(ZSTD_DStream* zds, ZSTD_outBuffer* output, ZSTD_inB
zds->outEnd = zds->outStart + decodedSize;
} }
zds->streamStage = zdss_flush;
/* pass-through */
/* fall-through */
case zdss_flush:
{ size_t const toFlushSize = zds->outEnd - zds->outStart;
size_t const flushedSize = ZSTD_limitCopy(op, oend-op, zds->outBuff + zds->outStart, toFlushSize);
lib/deprecated/zbuff.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* ***************************************************************
lib/deprecated/zbuff_common.c
+4 -5
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*-*************************************
@@ -23,4 +23,3 @@ unsigned ZBUFF_isError(size_t errorCode) { return ERR_isError(errorCode); }
/*! ZBUFF_getErrorName() :
* provides error code string from function result (useful for debugging) */
const char* ZBUFF_getErrorName(size_t errorCode) { return ERR_getErrorName(errorCode); }
lib/deprecated/zbuff_compress.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/deprecated/zbuff_decompress.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/dictBuilder/cover.c
+22 -12
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* *****************************************************************************
@@ -382,7 +382,7 @@ static void COVER_group(COVER_ctx_t *ctx, const void *group,
typedef struct {
U32 begin;
U32 end;
double score;
U32 score;
} COVER_segment_t;
/**
@@ -479,11 +479,16 @@ static COVER_segment_t COVER_selectSegment(const COVER_ctx_t *ctx, U32 *freqs,
* Check the validity of the parameters.
* Returns non-zero if the parameters are valid and 0 otherwise.
*/
static int COVER_checkParameters(ZDICT_cover_params_t parameters) {
static int COVER_checkParameters(ZDICT_cover_params_t parameters,
size_t maxDictSize) {
/* k and d are required parameters */
if (parameters.d == 0 || parameters.k == 0) {
return 0;
}
/* k <= maxDictSize */
if (parameters.k > maxDictSize) {
return 0;
}
/* d <= k */
if (parameters.d > parameters.k) {
return 0;
@@ -622,9 +627,13 @@ static size_t COVER_buildDictionary(const COVER_ctx_t *ctx, U32 *freqs,
/* Select a segment */
COVER_segment_t segment = COVER_selectSegment(
ctx, freqs, activeDmers, epochBegin, epochEnd, parameters);
/* Trim the segment if necessary and if it is empty then we are done */
/* If the segment covers no dmers, then we are out of content */
if (segment.score == 0) {
break;
}
/* Trim the segment if necessary and if it is too small then we are done */
segmentSize = MIN(segment.end - segment.begin + parameters.d - 1, tail);
if (segmentSize == 0) {
if (segmentSize < parameters.d) {
break;
}
/* We fill the dictionary from the back to allow the best segments to be
@@ -648,7 +657,7 @@ ZDICTLIB_API size_t ZDICT_trainFromBuffer_cover(
COVER_ctx_t ctx;
COVER_map_t activeDmers;
/* Checks */
if (!COVER_checkParameters(parameters)) {
if (!COVER_checkParameters(parameters, dictBufferCapacity)) {
DISPLAYLEVEL(1, "Cover parameters incorrect\n");
return ERROR(GENERIC);
}
@@ -884,7 +893,7 @@ static void COVER_tryParameters(void *opaque) {
goto _compressCleanup;
}
/* Compress each sample and sum their sizes (or error) */
totalCompressedSize = 0;
totalCompressedSize = dictBufferCapacity;
for (i = 0; i < ctx->nbSamples; ++i) {
const size_t size = ZSTD_compress_usingCDict(
cctx, dst, dstCapacity, ctx->samples + ctx->offsets[i],
@@ -960,7 +969,7 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
/* Initialization */
COVER_best_init(&best);
/* Turn down global display level to clean up display at level 2 and below */
g_displayLevel = parameters->zParams.notificationLevel - 1;
g_displayLevel = displayLevel == 0 ? 0 : displayLevel - 1;
/* Loop through d first because each new value needs a new context */
LOCALDISPLAYLEVEL(displayLevel, 2, "Trying %u different sets of parameters\n",
kIterations);
@@ -994,8 +1003,9 @@ ZDICTLIB_API size_t ZDICT_optimizeTrainFromBuffer_cover(
data->parameters.k = k;
data->parameters.d = d;
data->parameters.steps = kSteps;
data->parameters.zParams.notificationLevel = g_displayLevel;
/* Check the parameters */
if (!COVER_checkParameters(data->parameters)) {
if (!COVER_checkParameters(data->parameters, dictBufferCapacity)) {
DISPLAYLEVEL(1, "Cover parameters incorrect\n");
free(data);
continue;
lib/dictBuilder/zdict.c
+8 -6
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -60,7 +60,7 @@
#define NOISELENGTH 32
static const int g_compressionLevel_default = 6;
static const int g_compressionLevel_default = 3;
static const U32 g_selectivity_default = 9;
@@ -703,7 +703,7 @@ static size_t ZDICT_analyzeEntropy(void* dstBuffer, size_t maxDstSize,
memset(repOffset, 0, sizeof(repOffset));
repOffset[1] = repOffset[4] = repOffset[8] = 1;
memset(bestRepOffset, 0, sizeof(bestRepOffset));
if (compressionLevel==0) compressionLevel = g_compressionLevel_default;
if (compressionLevel<=0) compressionLevel = g_compressionLevel_default;
params = ZSTD_getParams(compressionLevel, averageSampleSize, dictBufferSize);
{ size_t const beginResult = ZSTD_compressBegin_advanced(esr.ref, dictBuffer, dictBufferSize, params, 0);
if (ZSTD_isError(beginResult)) {
@@ -1056,6 +1056,8 @@ size_t ZDICT_trainFromBuffer(void* dictBuffer, size_t dictBufferCapacity,
memset(&params, 0, sizeof(params));
params.d = 8;
params.steps = 4;
/* Default to level 6 since no compression level information is avaialble */
params.zParams.compressionLevel = 6;
return ZDICT_optimizeTrainFromBuffer_cover(dictBuffer, dictBufferCapacity,
samplesBuffer, samplesSizes,
nbSamples, &params);
lib/dictBuilder/zdict.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef DICTBUILDER_H_001
lib/dll/example/Makefile
+5 -5
View File
@@ -1,11 +1,11 @@
# ##########################################################################
# ################################################################
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# ##########################################################################
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
VOID := /dev/null
ZSTDDIR := ../include
lib/legacy/zstd_legacy.h
+8 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_LEGACY_H
@@ -123,6 +123,7 @@ MEM_STATIC size_t ZSTD_decompressLegacy(
const void* dict,size_t dictSize)
{
U32 const version = ZSTD_isLegacy(src, compressedSize);
(void)dst; (void)dstCapacity; (void)dict; (void)dictSize; /* unused when ZSTD_LEGACY_SUPPORT >= 8 */
switch(version)
{
#if (ZSTD_LEGACY_SUPPORT <= 1)
@@ -223,6 +224,7 @@ MEM_STATIC size_t ZSTD_freeLegacyStreamContext(void* legacyContext, U32 version)
case 1 :
case 2 :
case 3 :
(void)legacyContext;
return ERROR(version_unsupported);
#if (ZSTD_LEGACY_SUPPORT <= 4)
case 4 : return ZBUFFv04_freeDCtx((ZBUFFv04_DCtx*)legacyContext);
@@ -250,6 +252,7 @@ MEM_STATIC size_t ZSTD_initLegacyStream(void** legacyContext, U32 prevVersion, U
case 1 :
case 2 :
case 3 :
(void)dict; (void)dictSize;
return 0;
#if (ZSTD_LEGACY_SUPPORT <= 4)
case 4 :
@@ -306,6 +309,7 @@ MEM_STATIC size_t ZSTD_decompressLegacyStream(void* legacyContext, U32 version,
case 1 :
case 2 :
case 3 :
(void)legacyContext; (void)output; (void)input;
return ERROR(version_unsupported);
#if (ZSTD_LEGACY_SUPPORT <= 4)
case 4 :
lib/legacy/zstd_v01.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v01.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_V01_H_28739879432
lib/legacy/zstd_v02.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v02.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_V02_H_4174539423
lib/legacy/zstd_v03.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v03.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_V03_H_298734209782
lib/legacy/zstd_v04.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v04.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_V04_H_91868324769238
lib/legacy/zstd_v05.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v05.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTDv05_H
lib/legacy/zstd_v06.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v06.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTDv06_H
lib/legacy/zstd_v07.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
lib/legacy/zstd_v07.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTDv07_H_235446
lib/zstd.h
+3 -5
View File
@@ -2,11 +2,10 @@
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#if defined (__cplusplus)
extern "C" {
#endif
@@ -804,7 +803,6 @@ ZSTDLIB_API size_t ZSTD_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstC
ZSTDLIB_API size_t ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
/*-
Buffer-less streaming decompression (synchronous mode)
programs/.gitignore
+4
View File
@@ -1,8 +1,12 @@
# local binary (Makefile)
zstd
zstd32
zstd4
zstd-compress
zstd-decompress
zstd-frugal
zstd-small
zstd-nolegacy
# Object files
*.o
programs/Makefile
+31 -20
View File
@@ -1,12 +1,10 @@
# ##########################################################################
# ################################################################
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This Makefile is validated for Linux, macOS, *BSD, Hurd, Solaris, MSYS2 targets
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ##########################################################################
# zstd : Command Line Utility, supporting gzip-like arguments
# zstd32 : Same as zstd, but forced to compile in 32-bits mode
@@ -42,7 +40,7 @@ CPPFLAGS+= -I$(ZSTDDIR) -I$(ZSTDDIR)/common -I$(ZSTDDIR)/compress \
-DZSTD_NEWAPI \
-DXXH_NAMESPACE=ZSTD_ # because xxhash.o already compiled with this macro from library
CFLAGS ?= -O3
DEBUGFLAGS = -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
DEBUGFLAGS= -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow \
-Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement \
-Wstrict-prototypes -Wundef -Wpointer-arith -Wformat-security \
-Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings \
@@ -68,8 +66,7 @@ endif
else
endif
ZSTDLIB_FILES := $(wildcard $(ZSTD_FILES)) $(wildcard $(ZSTDLEGACY_FILES)) $(wildcard $(ZDICT_FILES))
ZSTDLIB_OBJ := $(patsubst %.c,%.o,$(ZSTDLIB_FILES))
ZSTDLIB_FILES := $(sort $(wildcard $(ZSTD_FILES)) $(wildcard $(ZSTDLEGACY_FILES)) $(wildcard $(ZDICT_FILES)))
# Define *.exe as extension for Windows systems
ifneq (,$(filter Windows%,$(OS)))
@@ -132,12 +129,15 @@ else
LZ4_MSG := $(NO_LZ4_MSG)
endif
.PHONY: default all clean clean_decomp_o install uninstall generate_res
.PHONY: default
default: zstd-release
.PHONY: all
all: zstd
.PHONY: allVariants
allVariants: zstd zstd-compress zstd-decompress zstd-small zstd-nolegacy
$(ZSTDDECOMP_O): CFLAGS += $(ALIGN_LOOP)
zstd zstd4 : CPPFLAGS += $(THREAD_CPP) $(ZLIBCPP) $(LZMACPP)
@@ -154,8 +154,9 @@ zstd zstd4 : $(ZSTDLIB_FILES) zstdcli.o fileio.o bench.o datagen.o dibio.o
ifneq (,$(filter Windows%,$(OS)))
windres/generate_res.bat
endif
$(CC) $(FLAGS) $^ $(RES_FILE) -o zstd$(EXT) $(LDFLAGS)
$(CC) $(FLAGS) $^ $(RES_FILE) -o $@$(EXT) $(LDFLAGS)
.PHONY: zstd-release
zstd-release: DEBUGFLAGS :=
zstd-release: zstd
@@ -166,8 +167,8 @@ ifneq (,$(filter Windows%,$(OS)))
endif
$(CC) -m32 $(FLAGS) $^ $(RES32_FILE) -o $@$(EXT)
zstd-nolegacy : clean_decomp_o
$(MAKE) zstd ZSTD_LEGACY_SUPPORT=0
zstd-nolegacy : $(ZSTD_FILES) $(ZDICT_FILES) zstdcli.o fileio.c bench.o datagen.o dibio.o
$(CC) $(FLAGS) $^ -o $@$(EXT) $(LDFLAGS)
zstd-nomt : THREAD_CPP :=
zstd-nomt : THREAD_LD :=
@@ -198,9 +199,9 @@ zstd-pgo : clean zstd
$(MAKE) zstd MOREFLAGS=-fprofile-use
# minimal target, with only zstd compression and decompression. no bench. no legacy.
zstd-small: CFLAGS = "-Os -s"
zstd-small: CFLAGS = -Os -s
zstd-frugal zstd-small: $(ZSTD_FILES) zstdcli.c fileio.c
$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODICT $^ -o zstd$(EXT)
$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODICT $^ -o $@$(EXT)
zstd-decompress: $(ZSTDCOMMON_FILES) $(ZSTDDECOMP_FILES) zstdcli.c fileio.c
$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODICT -DZSTD_NOCOMPRESS $^ -o $@$(EXT)
@@ -208,34 +209,37 @@ zstd-decompress: $(ZSTDCOMMON_FILES) $(ZSTDDECOMP_FILES) zstdcli.c fileio.c
zstd-compress: $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES) zstdcli.c fileio.c
$(CC) $(FLAGS) -DZSTD_NOBENCH -DZSTD_NODICT -DZSTD_NODECOMPRESS $^ -o $@$(EXT)
# zstd is now built with Multi-threading by default
# zstd is now built with multithreading enabled y default
zstdmt: zstd
.PHONY: generate_res
generate_res:
windres/generate_res.bat
.PHONY: clean
clean:
$(MAKE) -C $(ZSTDDIR) clean
@$(RM) $(ZSTDDIR)/decompress/*.o $(ZSTDDIR)/decompress/zstd_decompress.gcda
@$(RM) core *.o tmp* result* *.gcda dictionary *.zst \
zstd$(EXT) zstd32$(EXT) zstd-compress$(EXT) zstd-decompress$(EXT) \
zstd-small$(EXT) zstd-frugal$(EXT) zstd-nolegacy$(EXT) zstd4$(EXT) \
*.gcda default.profraw have_zlib$(EXT)
@echo Cleaning completed
clean_decomp_o:
@$(RM) $(ZSTDDECOMP_O)
MD2ROFF = ronn
MD2ROFF_FLAGS = --roff --warnings --manual="User Commands" --organization="zstd $(ZSTD_VERSION)"
zstd.1: zstd.1.md
cat $^ | $(MD2ROFF) $(MD2ROFF_FLAGS) | sed -n '/^\.\\\".*/!p' > $@
.PHONY: man
man: zstd.1
.PHONY: clean-man
clean-man:
rm zstd.1
.PHONY: preview-man
preview-man: clean-man man
man ./zstd.1
@@ -244,6 +248,10 @@ preview-man: clean-man man
#-----------------------------------------------------------------------------
ifneq (,$(filter $(shell uname),Linux Darwin GNU/kFreeBSD GNU OpenBSD FreeBSD NetBSD DragonFly SunOS))
.PHONY: list
list:
@$(MAKE) -pRrq -f $(lastword $(MAKEFILE_LIST)) : 2>/dev/null | awk -v RS= -F: '/^# File/,/^# Finished Make data base/ {if ($$1 !~ "^[#.]") {print $$1}}' | sort | egrep -v -e '^[^[:alnum:]]' -e '^$@$$' | xargs
ifneq (,$(filter $(shell uname),SunOS))
INSTALL ?= ginstall
else
@@ -264,6 +272,7 @@ INSTALL_PROGRAM ?= $(INSTALL) -m 755
INSTALL_SCRIPT ?= $(INSTALL) -m 755
INSTALL_MAN ?= $(INSTALL) -m 644
.PHONY: install
install: zstd
@echo Installing binaries
@$(INSTALL) -d -m 755 $(DESTDIR)$(BINDIR)/ $(DESTDIR)$(MANDIR)/
@@ -279,6 +288,7 @@ install: zstd
@ln -sf zstd.1 $(DESTDIR)$(MANDIR)/unzstd.1
@echo zstd installation completed
.PHONY: uninstall
uninstall:
@$(RM) $(DESTDIR)$(BINDIR)/zstdgrep
@$(RM) $(DESTDIR)$(BINDIR)/zstdless
@@ -289,4 +299,5 @@ uninstall:
@$(RM) $(DESTDIR)$(MANDIR)/unzstd.1
@$(RM) $(DESTDIR)$(MANDIR)/zstd.1
@echo zstd programs successfully uninstalled
endif
programs/README.md
+28 -17
View File
@@ -3,44 +3,54 @@ Command Line Interface for Zstandard library
Command Line Interface (CLI) can be created using the `make` command without any additional parameters.
There are however other Makefile targets that create different variations of CLI:
- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd versions
- `zstd32` : Same as `zstd`, but forced to compile in 32-bits mode
- `zstd_nolegacy` : Same as `zstd` except of support for decompression of legacy zstd versions
- `zstd-small` : CLI optimized for minimal size; without dictionary builder, benchmark, and support for decompression of legacy zstd versions
- `zstd-compress` : compressor-only version of CLI; without dictionary builder, benchmark, and support for decompression of legacy zstd versions
- `zstd-decompress` : decompressor-only version of CLI; without dictionary builder, benchmark, and support for decompression of legacy zstd versions
- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd formats
- `zstd_nolegacy` : Same as `zstd` but without support for legacy zstd formats
- `zstd-small` : CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats
- `zstd-compress` : version of CLI which can only compress into zstd format
- `zstd-decompress` : version of CLI which can only decompress zstd format
#### Compilation variables
`zstd` tries to detect and use the following features automatically :
`zstd` scope can be altered by modifying the following compilation variables :
- __HAVE_THREAD__ : multithreading is automatically enabled when `pthread` is detected.
It's possible to disable multithread support, by either compiling `zstd-nomt` target or using HAVE_THREAD=0 variable.
It's possible to disable multithread support, by setting HAVE_THREAD=0 .
Example : make zstd HAVE_THREAD=0
It's also possible to force compilation with multithread support, using HAVE_THREAD=1.
In which case, linking stage will fail if `pthread` library cannot be found.
This might be useful to prevent silent feature disabling.
- __HAVE_ZLIB__ : `zstd` can compress and decompress files in `.gz` format.
This is done through command `--format=gzip`.
This is ordered through command `--format=gzip`.
Alternatively, symlinks named `gzip` or `gunzip` will mimic intended behavior.
`.gz` support is automatically enabled when `zlib` library is detected at build time.
It's possible to disable `.gz` support, by either compiling `zstd-nogz` target or using HAVE_ZLIB=0 variable.
It's possible to disable `.gz` support, by setting HAVE_ZLIB=0.
Example : make zstd HAVE_ZLIB=0
It's also possible to force compilation with zlib support, using HAVE_ZLIB=1.
In which case, linking stage will fail if `zlib` library cannot be found.
This might be useful to prevent silent feature disabling.
- __HAVE_LZMA__ : `zstd` can compress and decompress files in `.xz` and `.lzma` formats.
This is done through commands `--format=xz` and `--format=lzma` respectively.
This is ordered through commands `--format=xz` and `--format=lzma` respectively.
Alternatively, symlinks named `xz`, `unxz`, `lzma`, or `unlzma` will mimic intended behavior.
`.xz` and `.lzma` support is automatically enabled when `lzma` library is detected at build time.
It's possible to disable `.xz` and `.lzma` support, by either compiling `zstd-noxz` target or using HAVE_LZMA=0 variable.
It's possible to disable `.xz` and `.lzma` support, by setting HAVE_LZMA=0 .
Example : make zstd HAVE_LZMA=0
It's also possible to force compilation with lzma support, using HAVE_LZMA=1.
In which case, linking stage will fail if `lzma` library cannot be found.
This might be useful to prevent silent feature disabling.
- __ZSTD_LEGACY_SUPPORT__ : `zstd` can decompress files compressed by older versions of `zstd`.
Starting v0.8.0, all versions of `zstd` produce frames compliant with the [specification](../doc/zstd_compression_format.md), and are therefore compatible.
But older versions (< v0.8.0) produced different, incompatible, frames.
By default, `zstd` supports decoding legacy formats >= v0.4.0 (`ZSTD_LEGACY_SUPPORT=4`).
This can be altered by modifying this compilation variable.
`ZSTD_LEGACY_SUPPORT=1` means "support all formats >= v0.1.0".
`ZSTD_LEGACY_SUPPORT=2` means "support all formats >= v0.2.0", and so on.
`ZSTD_LEGACY_SUPPORT=0` means _DO NOT_ support any legacy format.
if `ZSTD_LEGACY_SUPPORT >= 8`, it's the same as `0`, since there is no legacy format after `7`.
Note : `zstd` only supports decoding older formats, and cannot generate any legacy format.
#### Aggregation of parameters
CLI supports aggregation of parameters i.e. `-b1`, `-e18`, and `-i1` can be joined into `-b1e18i1`.
@@ -61,7 +71,7 @@ will rely more and more on previously decoded content to compress the rest of th
Usage of the dictionary builder and created dictionaries with CLI:
1. Create the dictionary : `zstd --train FullPathToTrainingSet/* -o dictionaryName`
1. Create the dictionary : `zstd --train PathToTrainingSet/* -o dictionaryName`
2. Compress with the dictionary: `zstd FILE -D dictionaryName`
3. Decompress with the dictionary: `zstd --decompress FILE.zst -D dictionaryName`
@@ -70,8 +80,8 @@ Usage of the dictionary builder and created dictionaries with CLI:
CLI includes in-memory compression benchmark module for zstd.
The benchmark is conducted using given filenames. The files are read into memory and joined together.
It makes benchmark more precise as it eliminates I/O overhead.
Many filenames can be supplied as multiple parameters, parameters with wildcards or
names of directories can be used as parameters with the `-r` option.
Multiple filenames can be supplied, as multiple parameters, with wildcards,
or names of directories can be used as parameters with `-r` option.
The benchmark measures ratio, compressed size, compression and decompression speed.
One can select compression levels starting from `-b` and ending with `-e`.
@@ -101,13 +111,14 @@ Advanced arguments :
-v : verbose mode; specify multiple times to increase verbosity
-q : suppress warnings; specify twice to suppress errors too
-c : force write to standard output, even if it is the console
-l : print information about zstd compressed files
--ultra : enable levels beyond 19, up to 22 (requires more memory)
-T# : use # threads for compression (default:1)
-B# : select size of each job (default:0==automatic)
--no-dictID : don't write dictID into header (dictionary compression)
--[no-]check : integrity check (default:enabled)
-r : operate recursively on directories
--format=gzip : compress files to the .gz format
--format=xz : compress files to the .xz format
--format=lzma : compress files to the .lzma format
--test : test compressed file integrity
--[no-]sparse : sparse mode (default:disabled)
-M# : Set a memory usage limit for decompression
programs/bench.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
programs/bench.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
programs/datagen.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
programs/datagen.h
+6 -4
View File
@@ -1,11 +1,13 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef DATAGEN_H
#define DATAGEN_H
programs/dibio.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
programs/dibio.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/* This library is designed for a single-threaded console application.
programs/fileio.c
+234 -233
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -925,230 +925,6 @@ int FIO_compressFilename(const char* dstFileName, const char* srcFileName,
return result;
}
typedef struct {
int numActualFrames;
int numSkippableFrames;
unsigned long long decompressedSize;
int decompUnavailable;
unsigned long long compressedSize;
int usesCheck;
} fileInfo_t;
/*
* Reads information from file, stores in *info
* if successful, returns 0, returns 1 for frame analysis error, returns 2 for file not compressed with zstd
* returns 3 for cases in which file could not be opened.
*/
static int getFileInfo(fileInfo_t* info, const char* inFileName){
int detectError = 0;
FILE* const srcFile = FIO_openSrcFile(inFileName);
if (srcFile == NULL) {
DISPLAY("Error: could not open source file %s\n", inFileName);
return 3;
}
info->compressedSize = (unsigned long long)UTIL_getFileSize(inFileName);
/* begin analyzing frame */
for ( ; ; ) {
BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
size_t const numBytesRead = fread(headerBuffer, 1, sizeof(headerBuffer), srcFile);
if (numBytesRead < ZSTD_frameHeaderSize_min) {
if (feof(srcFile) && numBytesRead == 0 && info->compressedSize > 0) {
break;
}
else if (feof(srcFile)) {
DISPLAY("Error: reached end of file with incomplete frame\n");
detectError = 2;
break;
}
else {
DISPLAY("Error: did not reach end of file but ran out of frames\n");
detectError = 1;
break;
}
}
{
U32 const magicNumber = MEM_readLE32(headerBuffer);
if (magicNumber == ZSTD_MAGICNUMBER) {
U64 const frameContentSize = ZSTD_getFrameContentSize(headerBuffer, numBytesRead);
if (frameContentSize == ZSTD_CONTENTSIZE_ERROR || frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN) {
info->decompUnavailable = 1;
}
else {
info->decompressedSize += frameContentSize;
}
{
/* move to the end of the frame header */
size_t const headerSize = ZSTD_frameHeaderSize(headerBuffer, numBytesRead);
if (ZSTD_isError(headerSize)) {
DISPLAY("Error: could not determine frame header size\n");
detectError = 1;
break;
}
{
int const ret = fseek(srcFile, ((long)headerSize)-((long)numBytesRead), SEEK_CUR);
if (ret != 0) {
DISPLAY("Error: could not move to end of frame header\n");
detectError = 1;
break;
}
}
}
/* skip the rest of the blocks in the frame */
{
int lastBlock = 0;
do {
BYTE blockHeaderBuffer[3];
U32 blockHeader;
int blockSize;
size_t const readBytes = fread(blockHeaderBuffer, 1, 3, srcFile);
if (readBytes != 3) {
DISPLAY("There was a problem reading the block header\n");
detectError = 1;
break;
}
blockHeader = MEM_readLE24(blockHeaderBuffer);
lastBlock = blockHeader & 1;
blockSize = blockHeader >> 3;
{
int const ret = fseek(srcFile, blockSize, SEEK_CUR);
if (ret != 0) {
DISPLAY("Error: could not skip to end of block\n");
detectError = 1;
break;
}
}
} while (lastBlock != 1);
if (detectError) {
break;
}
}
{
/* check if checksum is used */
BYTE const frameHeaderDescriptor = headerBuffer[4];
int const contentChecksumFlag = (frameHeaderDescriptor & (1 << 2)) >> 2;
if (contentChecksumFlag) {
int const ret = fseek(srcFile, 4, SEEK_CUR);
info->usesCheck = 1;
if (ret != 0) {
DISPLAY("Error: could not skip past checksum\n");
detectError = 1;
break;
}
}
}
info->numActualFrames++;
}
else if (magicNumber == ZSTD_MAGIC_SKIPPABLE_START) {
BYTE frameSizeBuffer[4];
size_t const readBytes = fread(frameSizeBuffer, 1, 4, srcFile);
if (readBytes != 4) {
DISPLAY("There was an error reading skippable frame size");
detectError = 1;
break;
}
{
U32 const frameSize = MEM_readLE32(frameSizeBuffer);
int const ret = LONG_SEEK(srcFile, frameSize, SEEK_CUR);
if (ret != 0) {
DISPLAY("Error: could not find end of skippable frame\n");
detectError = 1;
break;
}
}
info->numSkippableFrames++;
}
else {
detectError = 2;
break;
}
}
}
fclose(srcFile);
return detectError;
}
static void displayInfo(const char* inFileName, fileInfo_t* info, int displayLevel){
double const compressedSizeMB = (double)info->compressedSize/(1 MB);
double const decompressedSizeMB = (double)info->decompressedSize/(1 MB);
double const ratio = (info->compressedSize == 0) ? 0 : ((double)info->decompressedSize)/info->compressedSize;
const char* const checkString = (info->usesCheck ? "XXH64" : "None");
if (displayLevel <= 2) {
if (!info->decompUnavailable) {
DISPLAYOUT("Skippable Non-Skippable Compressed Uncompressed Ratio Check Filename\n");
DISPLAYOUT("%9d %13d %7.2f MB %9.2f MB %5.3f %5s %s\n",
info->numSkippableFrames, info->numActualFrames, compressedSizeMB, decompressedSizeMB,
ratio, checkString, inFileName);
}
else {
DISPLAYOUT("Skippable Non-Skippable Compressed Check Filename\n");
DISPLAYOUT("%9d %13d %7.2f MB %5s %s\n",
info->numSkippableFrames, info->numActualFrames, compressedSizeMB, checkString, inFileName);
}
}
else{
DISPLAYOUT("# Zstandard Frames: %d\n", info->numActualFrames);
DISPLAYOUT("# Skippable Frames: %d\n", info->numSkippableFrames);
DISPLAYOUT("Compressed Size: %.2f MB (%llu B)\n", compressedSizeMB, info->compressedSize);
if (!info->decompUnavailable) {
DISPLAYOUT("Decompressed Size: %.2f MB (%llu B)\n", decompressedSizeMB, info->decompressedSize);
DISPLAYOUT("Ratio: %.4f\n", ratio);
}
DISPLAYOUT("Check: %s\n", checkString);
DISPLAYOUT("\n");
}
}
static int FIO_listFile(const char* inFileName, int displayLevel, unsigned fileNo, unsigned numFiles){
/* initialize info to avoid warnings */
fileInfo_t info;
memset(&info, 0, sizeof(info));
DISPLAYOUT("%s (%u/%u):\n", inFileName, fileNo, numFiles);
{
int const error = getFileInfo(&info, inFileName);
if (error == 1) {
/* display error, but provide output */
DISPLAY("An error occurred with getting file info\n");
}
else if (error == 2) {
DISPLAYOUT("File %s not compressed with zstd\n", inFileName);
if (displayLevel > 2) {
DISPLAYOUT("\n");
}
return 1;
}
else if (error == 3) {
/* error occurred with opening the file */
if (displayLevel > 2) {
DISPLAYOUT("\n");
}
return 1;
}
displayInfo(inFileName, &info, displayLevel);
return error;
}
}
int FIO_listMultipleFiles(unsigned numFiles, const char** filenameTable, int displayLevel){
if (numFiles == 0) {
DISPLAYOUT("No files given\n");
return 0;
}
DISPLAYOUT("===========================================\n");
DISPLAYOUT("Printing information about compressed files\n");
DISPLAYOUT("===========================================\n");
DISPLAYOUT("Number of files listed: %u\n", numFiles);
{
int error = 0;
unsigned u;
for (u=0; u<numFiles;u++) {
error |= FIO_listFile(filenameTable[u], displayLevel, u+1, numFiles);
}
return error;
}
}
int FIO_compressMultipleFilenames(const char** inFileNamesTable, unsigned nbFiles,
const char* suffix,
@@ -1194,10 +970,8 @@ int FIO_compressMultipleFilenames(const char** inFileNamesTable, unsigned nbFile
missed_files += FIO_compressFilename_dstFile(ress, dstFileName, inFileNamesTable[u], compressionLevel);
} }
/* Close & Free */
FIO_freeCResources(ress);
free(dstFileName);
return missed_files;
}
@@ -1208,8 +982,8 @@ int FIO_compressMultipleFilenames(const char** inFileNamesTable, unsigned nbFile
#ifndef ZSTD_NODECOMPRESS
/* **************************************************************************
* Decompression
****************************************************************************/
* Decompression
***************************************************************************/
typedef struct {
void* srcBuffer;
size_t srcBufferLoaded;
@@ -1616,7 +1390,7 @@ static unsigned long long FIO_decompressLz4Frame(dRess_t* ress,
/* Write Block */
if (decodedBytes) {
if (fwrite(ress->dstBuffer, 1, decodedBytes, ress->dstFile) != decodedBytes) {
DISPLAYLEVEL(1, "zstd: %s \n", strerr(errno));
DISPLAYLEVEL(1, "zstd: %s \n", strerror(errno));
decodingError = 1; break;
}
filesize += decodedBytes;
@@ -1878,4 +1652,231 @@ int FIO_decompressMultipleFilenames(const char** srcNamesTable, unsigned nbFiles
return missingFiles + skippedFiles;
}
/* **************************************************************************
* .zst file info (--list command)
***************************************************************************/
typedef struct {
int numActualFrames;
int numSkippableFrames;
unsigned long long decompressedSize;
int decompUnavailable;
unsigned long long compressedSize;
int usesCheck;
} fileInfo_t;
/** getFileInfo() :
* Reads information from file, stores in *info
* @return : 0 if successful
* 1 for frame analysis error
* 2 for file not compressed with zstd
* 3 for cases in which file could not be opened.
*/
static int getFileInfo(fileInfo_t* info, const char* inFileName){
int detectError = 0;
FILE* const srcFile = FIO_openSrcFile(inFileName);
if (srcFile == NULL) {
DISPLAY("Error: could not open source file %s\n", inFileName);
return 3;
}
info->compressedSize = (unsigned long long)UTIL_getFileSize(inFileName);
/* begin analyzing frame */
for ( ; ; ) {
BYTE headerBuffer[ZSTD_FRAMEHEADERSIZE_MAX];
size_t const numBytesRead = fread(headerBuffer, 1, sizeof(headerBuffer), srcFile);
if (numBytesRead < ZSTD_frameHeaderSize_min) {
if (feof(srcFile) && numBytesRead == 0 && info->compressedSize > 0) {
break;
}
else if (feof(srcFile)) {
DISPLAY("Error: reached end of file with incomplete frame\n");
detectError = 2;
break;
}
else {
DISPLAY("Error: did not reach end of file but ran out of frames\n");
detectError = 1;
break;
}
}
{ U32 const magicNumber = MEM_readLE32(headerBuffer);
/* Zstandard frame */
if (magicNumber == ZSTD_MAGICNUMBER) {
U64 const frameContentSize = ZSTD_getFrameContentSize(headerBuffer, numBytesRead);
if (frameContentSize == ZSTD_CONTENTSIZE_ERROR || frameContentSize == ZSTD_CONTENTSIZE_UNKNOWN) {
info->decompUnavailable = 1;
} else {
info->decompressedSize += frameContentSize;
}
/* move to the end of the frame header */
{ size_t const headerSize = ZSTD_frameHeaderSize(headerBuffer, numBytesRead);
if (ZSTD_isError(headerSize)) {
DISPLAY("Error: could not determine frame header size\n");
detectError = 1;
break;
}
{ int const ret = fseek(srcFile, ((long)headerSize)-((long)numBytesRead), SEEK_CUR);
if (ret != 0) {
DISPLAY("Error: could not move to end of frame header\n");
detectError = 1;
break;
} } }
/* skip the rest of the blocks in the frame */
{ int lastBlock = 0;
do {
BYTE blockHeaderBuffer[3];
size_t const readBytes = fread(blockHeaderBuffer, 1, 3, srcFile);
if (readBytes != 3) {
DISPLAY("There was a problem reading the block header\n");
detectError = 1;
break;
}
{ U32 const blockHeader = MEM_readLE24(blockHeaderBuffer);
U32 const blockTypeID = (blockHeader >> 1) & 3;
U32 const isRLE = (blockTypeID == 1);
U32 const isWrongBlock = (blockTypeID == 3);
long const blockSize = isRLE ? 1 : (long)(blockHeader >> 3);
if (isWrongBlock) {
DISPLAY("Error: unsupported block type \n");
detectError = 1;
break;
}
lastBlock = blockHeader & 1;
{ int const ret = fseek(srcFile, blockSize, SEEK_CUR);
if (ret != 0) {
DISPLAY("Error: could not skip to end of block\n");
detectError = 1;
break;
} } }
} while (lastBlock != 1);
if (detectError) break;
}
/* check if checksum is used */
{ BYTE const frameHeaderDescriptor = headerBuffer[4];
int const contentChecksumFlag = (frameHeaderDescriptor & (1 << 2)) >> 2;
if (contentChecksumFlag) {
int const ret = fseek(srcFile, 4, SEEK_CUR);
info->usesCheck = 1;
if (ret != 0) {
DISPLAY("Error: could not skip past checksum\n");
detectError = 1;
break;
} } }
info->numActualFrames++;
}
/* Skippable frame */
else if ((magicNumber & 0xFFFFFFF0U) == ZSTD_MAGIC_SKIPPABLE_START) {
U32 const frameSize = MEM_readLE32(headerBuffer + 4);
long const seek = (long)(8 + frameSize - numBytesRead);
int const ret = LONG_SEEK(srcFile, seek, SEEK_CUR);
if (ret != 0) {
DISPLAY("Error: could not find end of skippable frame\n");
detectError = 1;
break;
}
info->numSkippableFrames++;
}
/* unknown content */
else {
detectError = 2;
break;
}
}
} /* end analyzing frame */
fclose(srcFile);
return detectError;
}
static void displayInfo(const char* inFileName, fileInfo_t* info, int displayLevel){
unsigned const unit = info->compressedSize < (1 MB) ? (1 KB) : (1 MB);
const char* const unitStr = info->compressedSize < (1 MB) ? "KB" : "MB";
double const compressedSizeUnit = (double)info->compressedSize / unit;
double const decompressedSizeUnit = (double)info->decompressedSize / unit;
double const ratio = (info->compressedSize == 0) ? 0 : ((double)info->decompressedSize)/info->compressedSize;
const char* const checkString = (info->usesCheck ? "XXH64" : "None");
if (displayLevel <= 2) {
if (!info->decompUnavailable) {
DISPLAYOUT("Skippable Non-Skippable Compressed Uncompressed Ratio Check Filename\n");
DISPLAYOUT("%9d %13d %7.2f %2s %9.2f %2s %5.3f %5s %s\n",
info->numSkippableFrames, info->numActualFrames,
compressedSizeUnit, unitStr, decompressedSizeUnit, unitStr,
ratio, checkString, inFileName);
} else {
DISPLAYOUT("Skippable Non-Skippable Compressed Check Filename\n");
DISPLAYOUT("%9d %13d %7.2f MB %5s %s\n",
info->numSkippableFrames, info->numActualFrames,
compressedSizeUnit, checkString, inFileName);
}
} else {
DISPLAYOUT("# Zstandard Frames: %d\n", info->numActualFrames);
DISPLAYOUT("# Skippable Frames: %d\n", info->numSkippableFrames);
DISPLAYOUT("Compressed Size: %.2f %2s (%llu B)\n",
compressedSizeUnit, unitStr, info->compressedSize);
if (!info->decompUnavailable) {
DISPLAYOUT("Decompressed Size: %.2f %2s (%llu B)\n",
decompressedSizeUnit, unitStr, info->decompressedSize);
DISPLAYOUT("Ratio: %.4f\n", ratio);
}
DISPLAYOUT("Check: %s\n", checkString);
DISPLAYOUT("\n");
}
}
static int FIO_listFile(const char* inFileName, int displayLevel, unsigned fileNo, unsigned numFiles){
/* initialize info to avoid warnings */
fileInfo_t info;
memset(&info, 0, sizeof(info));
DISPLAYOUT("%s (%u/%u):\n", inFileName, fileNo, numFiles);
{
int const error = getFileInfo(&info, inFileName);
if (error == 1) {
/* display error, but provide output */
DISPLAY("An error occurred with getting file info\n");
}
else if (error == 2) {
DISPLAYOUT("File %s not compressed with zstd\n", inFileName);
if (displayLevel > 2) {
DISPLAYOUT("\n");
}
return 1;
}
else if (error == 3) {
/* error occurred with opening the file */
if (displayLevel > 2) {
DISPLAYOUT("\n");
}
return 1;
}
displayInfo(inFileName, &info, displayLevel);
return error;
}
}
int FIO_listMultipleFiles(unsigned numFiles, const char** filenameTable, int displayLevel){
if (numFiles == 0) {
DISPLAYOUT("No files given\n");
return 0;
}
DISPLAYOUT("===========================================\n");
DISPLAYOUT("Printing information about compressed files\n");
DISPLAYOUT("===========================================\n");
DISPLAYOUT("Number of files listed: %u\n", numFiles);
{
int error = 0;
unsigned u;
for (u=0; u<numFiles;u++) {
error |= FIO_listFile(filenameTable[u], displayLevel, u+1, numFiles);
}
return error;
}
}
#endif /* #ifndef ZSTD_NODECOMPRESS */
programs/fileio.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
programs/platform.h
+4 -6
View File
@@ -1,12 +1,10 @@
/**
* platform.h - compiler and OS detection
*
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef PLATFORM_H_MODULE
programs/util.h
+17 -15
View File
@@ -1,12 +1,10 @@
/**
* util.h - utility functions
*
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef UTIL_H_MODULE
@@ -107,13 +105,21 @@ extern "C" {
#endif
/*-****************************************
* Console log
******************************************/
static int g_utilDisplayLevel;
#define UTIL_DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define UTIL_DISPLAYLEVEL(l, ...) { if (g_utilDisplayLevel>=l) { UTIL_DISPLAY(__VA_ARGS__); } }
/*-****************************************
* Time functions
******************************************/
#if defined(_WIN32) /* Windows */
typedef LARGE_INTEGER UTIL_freq_t;
typedef LARGE_INTEGER UTIL_time_t;
UTIL_STATIC void UTIL_initTimer(UTIL_freq_t* ticksPerSecond) { if (!QueryPerformanceFrequency(ticksPerSecond)) fprintf(stderr, "ERROR: QueryPerformance not present\n"); }
UTIL_STATIC void UTIL_initTimer(UTIL_freq_t* ticksPerSecond) { if (!QueryPerformanceFrequency(ticksPerSecond)) UTIL_DISPLAYLEVEL(1, "ERROR: QueryPerformance not present\n"); }
UTIL_STATIC void UTIL_getTime(UTIL_time_t* x) { QueryPerformanceCounter(x); }
UTIL_STATIC U64 UTIL_getSpanTimeMicro(UTIL_freq_t ticksPerSecond, UTIL_time_t clockStart, UTIL_time_t clockEnd) { return 1000000ULL*(clockEnd.QuadPart - clockStart.QuadPart)/ticksPerSecond.QuadPart; }
UTIL_STATIC U64 UTIL_getSpanTimeNano(UTIL_freq_t ticksPerSecond, UTIL_time_t clockStart, UTIL_time_t clockEnd) { return 1000000000ULL*(clockEnd.QuadPart - clockStart.QuadPart)/ticksPerSecond.QuadPart; }
@@ -286,10 +292,6 @@ UTIL_STATIC void *UTIL_realloc(void *ptr, size_t size)
return NULL;
}
static int g_utilDisplayLevel;
#define UTIL_DISPLAY(...) fprintf(stderr, __VA_ARGS__)
#define UTIL_DISPLAYLEVEL(l, ...) { if (g_utilDisplayLevel>=l) { UTIL_DISPLAY(__VA_ARGS__); } }
#ifdef _WIN32
# define UTIL_HAS_CREATEFILELIST
@@ -311,7 +313,7 @@ UTIL_STATIC int UTIL_prepareFileList(const char *dirName, char** bufStart, size_
hFile=FindFirstFileA(path, &cFile);
if (hFile == INVALID_HANDLE_VALUE) {
fprintf(stderr, "Cannot open directory '%s'\n", dirName);
UTIL_DISPLAYLEVEL(1, "Cannot open directory '%s'\n", dirName);
return 0;
}
free(path);
@@ -365,7 +367,7 @@ UTIL_STATIC int UTIL_prepareFileList(const char *dirName, char** bufStart, size_
int dirLength, fnameLength, pathLength, nbFiles = 0;
if (!(dir = opendir(dirName))) {
fprintf(stderr, "Cannot open directory '%s': %s\n", dirName, strerror(errno));
UTIL_DISPLAYLEVEL(1, "Cannot open directory '%s': %s\n", dirName, strerror(errno));
return 0;
}
@@ -410,7 +412,7 @@ UTIL_STATIC int UTIL_prepareFileList(const char *dirName, char** bufStart, size_
}
if (errno != 0) {
fprintf(stderr, "readdir(%s) error: %s\n", dirName, strerror(errno));
UTIL_DISPLAYLEVEL(1, "readdir(%s) error: %s\n", dirName, strerror(errno));
free(*bufStart);
*bufStart = NULL;
}
@@ -423,7 +425,7 @@ UTIL_STATIC int UTIL_prepareFileList(const char *dirName, char** bufStart, size_
UTIL_STATIC int UTIL_prepareFileList(const char *dirName, char** bufStart, size_t* pos, char** bufEnd, int followLinks)
{
(void)bufStart; (void)bufEnd; (void)pos;
fprintf(stderr, "Directory %s ignored (compiled without _WIN32 or _POSIX_C_SOURCE)\n", dirName);
UTIL_DISPLAYLEVEL(1, "Directory %s ignored (compiled without _WIN32 or _POSIX_C_SOURCE)\n", dirName);
return 0;
}
programs/zstd.1
+2 -2
View File
@@ -1,5 +1,5 @@
.
.TH "ZSTD" "1" "July 2017" "zstd 1.3.1" "User Commands"
.TH "ZSTD" "1" "August 2017" "zstd 1.3.1" "User Commands"
.
.SH "NAME"
\fBzstd\fR \- zstd, zstdmt, unzstd, zstdcat \- Compress or decompress \.zst files
@@ -149,7 +149,7 @@ display help/long help and exit
.
.TP
\fB\-V\fR, \fB\-\-version\fR
display version number and exit
display version number and exit\. Advanced : \fB\-vV\fR also displays supported formats\. \fB\-vvV\fR also displays POSIX support\.
.
.TP
\fB\-v\fR
programs/zstd.1.md
+3 -1
View File
@@ -140,7 +140,9 @@ the last one takes effect.
* `-h`/`-H`, `--help`:
display help/long help and exit
* `-V`, `--version`:
display version number and exit
display version number and exit.
Advanced : `-vV` also displays supported formats.
`-vvV` also displays POSIX support.
* `-v`:
verbose mode
* `-q`, `--quiet`:
programs/zstdcli.c
+66 -35
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -16,7 +16,7 @@
#endif
#ifndef ZSTDCLI_CLEVEL_MAX
# define ZSTDCLI_CLEVEL_MAX 19 /* when not using --ultra */
# define ZSTDCLI_CLEVEL_MAX 19 /* without using --ultra */
#endif
@@ -26,14 +26,15 @@
**************************************/
#include "platform.h" /* IS_CONSOLE, PLATFORM_POSIX_VERSION */
#include "util.h" /* UTIL_HAS_CREATEFILELIST, UTIL_createFileList */
#include <stdio.h> /* fprintf(), stdin, stdout, stderr */
#include <string.h> /* strcmp, strlen */
#include <errno.h> /* errno */
#include "fileio.h"
#include "fileio.h" /* stdinmark, stdoutmark, ZSTD_EXTENSION */
#ifndef ZSTD_NOBENCH
# include "bench.h" /* BMK_benchFiles, BMK_SetNbSeconds */
#endif
#ifndef ZSTD_NODICT
# include "dibio.h"
# include "dibio.h" /* ZDICT_cover_params_t, DiB_trainFromFiles() */
#endif
#define ZSTD_STATIC_LINKING_ONLY /* ZSTD_maxCLevel */
#include "zstd.h" /* ZSTD_VERSION_STRING */
@@ -64,7 +65,7 @@
#define MB *(1 <<20)
#define GB *(1U<<30)
#define DEFAULT_DISPLAY_LEVEL 2
#define DISPLAY_LEVEL_DEFAULT 2
static const char* g_defaultDictName = "dictionary";
static const unsigned g_defaultMaxDictSize = 110 KB;
@@ -79,7 +80,7 @@ static U32 g_overlapLog = OVERLAP_LOG_DEFAULT;
**************************************/
#define DISPLAY(...) fprintf(g_displayOut, __VA_ARGS__)
#define DISPLAYLEVEL(l, ...) { if (g_displayLevel>=l) { DISPLAY(__VA_ARGS__); } }
static int g_displayLevel = DEFAULT_DISPLAY_LEVEL; /* 0 : no display, 1: errors, 2 : + result + interaction + warnings, 3 : + progression, 4 : + information */
static int g_displayLevel = DISPLAY_LEVEL_DEFAULT; /* 0 : no display, 1: errors, 2 : + result + interaction + warnings, 3 : + progression, 4 : + information */
static FILE* g_displayOut;
@@ -88,12 +89,12 @@ static FILE* g_displayOut;
**************************************/
static int usage(const char* programName)
{
DISPLAY( "Usage :\n");
DISPLAY( " %s [args] [FILE(s)] [-o file]\n", programName);
DISPLAY( "Usage : \n");
DISPLAY( " %s [args] [FILE(s)] [-o file] \n", programName);
DISPLAY( "\n");
DISPLAY( "FILE : a filename\n");
DISPLAY( "FILE : a filename \n");
DISPLAY( " with no FILE, or when FILE is - , read standard input\n");
DISPLAY( "Arguments :\n");
DISPLAY( "Arguments : \n");
#ifndef ZSTD_NOCOMPRESS
DISPLAY( " -# : # compression level (1-%d, default:%d) \n", ZSTDCLI_CLEVEL_MAX, ZSTDCLI_CLEVEL_DEFAULT);
#endif
@@ -105,7 +106,7 @@ static int usage(const char* programName)
DISPLAY( " -f : overwrite output without prompting and (de)compress links \n");
DISPLAY( "--rm : remove source file(s) after successful de/compression \n");
DISPLAY( " -k : preserve source file(s) (default) \n");
DISPLAY( " -h/-H : display help/long help and exit\n");
DISPLAY( " -h/-H : display help/long help and exit \n");
return 0;
}
@@ -114,12 +115,12 @@ static int usage_advanced(const char* programName)
DISPLAY(WELCOME_MESSAGE);
usage(programName);
DISPLAY( "\n");
DISPLAY( "Advanced arguments :\n");
DISPLAY( " -V : display Version number and exit\n");
DISPLAY( "Advanced arguments : \n");
DISPLAY( " -V : display Version number and exit \n");
DISPLAY( " -v : verbose mode; specify multiple times to increase verbosity\n");
DISPLAY( " -q : suppress warnings; specify twice to suppress errors too\n");
DISPLAY( " -c : force write to standard output, even if it is the console\n");
DISPLAY( " -l : print information about zstd compressed files.\n");
DISPLAY( " -l : print information about zstd compressed files \n");
#ifndef ZSTD_NOCOMPRESS
DISPLAY( "--ultra : enable levels beyond %i, up to %i (requires more memory)\n", ZSTDCLI_CLEVEL_MAX, ZSTD_maxCLevel());
#ifdef ZSTD_MULTITHREAD
@@ -151,11 +152,10 @@ static int usage_advanced(const char* programName)
#endif
#endif
DISPLAY( " -M# : Set a memory usage limit for decompression \n");
DISPLAY( "--list : list information about a zstd compressed file \n");
DISPLAY( "-- : All arguments after \"--\" are treated as files \n");
#ifndef ZSTD_NODICT
DISPLAY( "\n");
DISPLAY( "Dictionary builder :\n");
DISPLAY( "Dictionary builder : \n");
DISPLAY( "--train ## : create a dictionary from a training set of files \n");
DISPLAY( "--train-cover[=k=#,d=#,steps=#] : use the cover algorithm with optional args\n");
DISPLAY( "--train-legacy[=s=#] : use the legacy algorithm with selectivity (default: %u)\n", g_defaultSelectivityLevel);
@@ -165,12 +165,12 @@ static int usage_advanced(const char* programName)
#endif
#ifndef ZSTD_NOBENCH
DISPLAY( "\n");
DISPLAY( "Benchmark arguments :\n");
DISPLAY( "Benchmark arguments : \n");
DISPLAY( " -b# : benchmark file(s), using # compression level (default : 1) \n");
DISPLAY( " -e# : test all compression levels from -bX to # (default: 1)\n");
DISPLAY( " -i# : minimum evaluation time in seconds (default : 3s)\n");
DISPLAY( " -i# : minimum evaluation time in seconds (default : 3s) \n");
DISPLAY( " -B# : cut file into independent blocks of size # (default: no block)\n");
DISPLAY( "--priority=rt : set process priority to real-time\n");
DISPLAY( "--priority=rt : set process priority to real-time \n");
#endif
return 0;
}
@@ -313,6 +313,35 @@ static unsigned parseCompressionParameters(const char* stringPtr, ZSTD_compressi
return 1;
}
static void printVersion(void)
{
DISPLAY(WELCOME_MESSAGE);
/* format support */
DISPLAYLEVEL(3, "*** supports: zstd");
#if defined(ZSTD_LEGACY_SUPPORT) && (ZSTD_LEGACY_SUPPORT>0) && (ZSTD_LEGACY_SUPPORT<8)
DISPLAYLEVEL(3, ", zstd legacy v0.%d+", ZSTD_LEGACY_SUPPORT);
#endif
#ifdef ZSTD_GZCOMPRESS
DISPLAYLEVEL(3, ", gzip");
#endif
#ifdef ZSTD_LZ4COMPRESS
DISPLAYLEVEL(3, ", lz4");
#endif
#ifdef ZSTD_LZMACOMPRESS
DISPLAYLEVEL(3, ", lzma, xz ");
#endif
DISPLAYLEVEL(3, "\n");
/* posix support */
#ifdef _POSIX_C_SOURCE
DISPLAYLEVEL(4, "_POSIX_C_SOURCE defined: %ldL\n", (long) _POSIX_C_SOURCE);
#endif
#ifdef _POSIX_VERSION
DISPLAYLEVEL(4, "_POSIX_VERSION defined: %ldL \n", (long) _POSIX_VERSION);
#endif
#ifdef PLATFORM_POSIX_VERSION
DISPLAYLEVEL(4, "PLATFORM_POSIX_VERSION defined: %ldL\n", (long) PLATFORM_POSIX_VERSION);
#endif
}
typedef enum { zom_compress, zom_decompress, zom_test, zom_bench, zom_train, zom_list } zstd_operation_mode;
@@ -492,7 +521,7 @@ int main(int argCount, const char* argv[])
switch(argument[0])
{
/* Display help */
case 'V': g_displayOut=stdout; DISPLAY(WELCOME_MESSAGE); CLEAN_RETURN(0); /* Version Only */
case 'V': g_displayOut=stdout; printVersion(); CLEAN_RETURN(0); /* Version Only */
case 'H':
case 'h': g_displayOut=stdout; CLEAN_RETURN(usage_advanced(programName));
@@ -635,24 +664,18 @@ int main(int argCount, const char* argv[])
filenameTable[filenameIdx++] = argument;
}
if (lastCommand) { DISPLAY("error : command must be followed by argument \n"); CLEAN_RETURN(1); } /* forgotten argument */
if (lastCommand) { /* forgotten argument */
DISPLAY("error : command must be followed by argument \n");
CLEAN_RETURN(1);
}
/* Welcome message (if verbose) */
DISPLAYLEVEL(3, WELCOME_MESSAGE);
#ifdef _POSIX_C_SOURCE
DISPLAYLEVEL(4, "_POSIX_C_SOURCE defined: %ldL\n", (long) _POSIX_C_SOURCE);
#endif
#ifdef _POSIX_VERSION
DISPLAYLEVEL(4, "_POSIX_VERSION defined: %ldL\n", (long) _POSIX_VERSION);
#endif
#ifdef PLATFORM_POSIX_VERSION
DISPLAYLEVEL(4, "PLATFORM_POSIX_VERSION defined: %ldL\n", (long) PLATFORM_POSIX_VERSION);
#endif
if (nbThreads == 0) {
/* try to guess */
nbThreads = UTIL_countPhysicalCores();
DISPLAYLEVEL(3, "Note: %d physical core(s) detected\n", nbThreads);
DISPLAYLEVEL(3, "Note: %d physical core(s) detected \n", nbThreads);
}
g_utilDisplayLevel = g_displayLevel;
@@ -679,10 +702,17 @@ int main(int argCount, const char* argv[])
}
}
#endif
if (operation == zom_list) {
#ifndef ZSTD_NODECOMPRESS
int const ret = FIO_listMultipleFiles(filenameIdx, filenameTable, g_displayLevel);
CLEAN_RETURN(ret);
#else
DISPLAY("file information is not supported \n");
CLEAN_RETURN(1);
#endif
}
/* Check if benchmark is selected */
if (operation==zom_bench) {
#ifndef ZSTD_NOBENCH
@@ -692,7 +722,7 @@ int main(int argCount, const char* argv[])
BMK_setNbSeconds(bench_nbSeconds);
BMK_benchFiles(filenameTable, filenameIdx, dictFileName, cLevel, cLevelLast, &compressionParams, setRealTimePrio);
#endif
(void)bench_nbSeconds;
(void)bench_nbSeconds; (void)blockSize; (void)setRealTimePrio;
goto _end;
}
@@ -763,6 +793,7 @@ int main(int argCount, const char* argv[])
else
operationResult = FIO_compressMultipleFilenames(filenameTable, filenameIdx, outFileName ? outFileName : suffix, dictFileName, cLevel, &compressionParams);
#else
(void)suffix;
DISPLAY("Compression not supported\n");
#endif
} else { /* decompression or test */
tests/.gitignore
+2
View File
@@ -21,6 +21,7 @@ symbols
legacy
decodecorpus
pool
poolTests
invalidDictionaries
# Tmp test directory
@@ -48,6 +49,7 @@ grillResults.txt
_*
tmp*
*.zst
*.gz
result
out
tests/Makefile
+17 -22
View File
@@ -1,13 +1,11 @@
# ##########################################################################
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# ################################################################
# Copyright (c) 2015-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This Makefile is validated for Linux, macOS, *BSD, Hurd, Solaris, MSYS2 targets
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# ##########################################################################
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
# datagen : Synthetic and parametrable data generator, for tests
# fullbench : Precisely measure speed for each zstd inner functions
# fullbench32: Same as fullbench, but forced to compile in 32-bits mode
@@ -46,10 +44,6 @@ ZSTD_FILES := $(ZSTDDECOMP_FILES) $(ZSTDCOMMON_FILES) $(ZSTDCOMP_FILES)
ZBUFF_FILES := $(ZSTDDIR)/deprecated/*.c
ZDICT_FILES := $(ZSTDDIR)/dictBuilder/*.c
ZSTD_OBJ := $(patsubst %.c,%.o, $(wildcard $(ZSTD_FILES)) )
ZBUFF_OBJ := $(patsubst %.c,%.o, $(wildcard $(ZBUFF_FILES)) )
ZDICT_OBJ := $(patsubst %.c,%.o, $(wildcard $(ZDICT_FILES)) )
# Define *.exe as extension for Windows systems
ifneq (,$(filter Windows%,$(OS)))
@@ -77,7 +71,7 @@ all: fullbench fuzzer zstreamtest paramgrill datagen decodecorpus
all32: fullbench32 fuzzer32 zstreamtest32
allnothread: fullbench fuzzer paramgrill datagen decodecorpus
allnothread: fullbench fuzzer paramgrill datagen decodecorpus
dll: fuzzer-dll zstreamtest-dll
@@ -91,7 +85,7 @@ zstd-nolegacy:
$(MAKE) -C $(PRGDIR) $@
gzstd:
$(MAKE) -C $(PRGDIR) $@
$(MAKE) -C $(PRGDIR) zstd HAVE_ZLIB=1
fullbench32: CPPFLAGS += -m32
fullbench fullbench32 : CPPFLAGS += $(MULTITHREAD_CPP)
@@ -192,8 +186,8 @@ else
$(CC) $(FLAGS) $^ -o $@$(EXT) -Wl,-rpath=$(ZSTDDIR) $(ZSTDDIR)/libzstd.so
endif
poolTests : poolTests.c $(ZSTDDIR)/common/pool.c $(ZSTDDIR)/common/threading.c
$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
poolTests : poolTests.c $(ZSTDDIR)/common/pool.c $(ZSTDDIR)/common/threading.c $(ZSTDDIR)/common/zstd_common.c $(ZSTDDIR)/common/error_private.c
$(CC) $(FLAGS) $(MULTITHREAD) $^ -o $@$(EXT)
namespaceTest:
if $(CC) namespaceTest.c ../lib/common/xxhash.c -o $@ ; then echo compilation should fail; exit 1 ; fi
@@ -280,17 +274,17 @@ test-zstd: zstd zstd-playTests
test-zstd32: ZSTD = $(PRGDIR)/zstd32
test-zstd32: zstd32 zstd-playTests
test-zstd-nolegacy: ZSTD = $(PRGDIR)/zstd
test-zstd-nolegacy: ZSTD = $(PRGDIR)/zstd-nolegacy
test-zstd-nolegacy: zstd-nolegacy zstd-playTests
test-gzstd: gzstd
$(PRGDIR)/zstd README.md test-zstd-speed.py
gzip README.md test-zstd-speed.py
$(PRGDIR)/zstd -f README.md test-zstd-speed.py
gzip -f README.md test-zstd-speed.py
cat README.md.zst test-zstd-speed.py.gz >zstd_gz.zst
cat README.md.gz test-zstd-speed.py.zst >gz_zstd.gz
$(PRGDIR)/zstd -d README.md.gz -o README2.md
$(PRGDIR)/zstd -d README.md.gz test-zstd-speed.py.gz
$(PRGDIR)/zstd -d zstd_gz.zst gz_zstd.gz
$(PRGDIR)/zstd -df README.md.gz -o README2.md
$(PRGDIR)/zstd -df README.md.gz test-zstd-speed.py.gz
$(PRGDIR)/zstd -df zstd_gz.zst gz_zstd.gz
$(DIFF) -q zstd_gz gz_zstd
echo Hello World ZSTD | $(PRGDIR)/zstd -c - >hello.zst
echo Hello World GZIP | gzip -c - >hello.gz
@@ -298,6 +292,7 @@ test-gzstd: gzstd
cat hello.zst hello.gz hello.txt >hello_zst_gz_txt.gz
$(PRGDIR)/zstd -dcf hello.*
$(PRGDIR)/zstd -dcf - <hello_zst_gz_txt.gz
$(RM) *.gz *.zst README2.md gz_zstd zstd_gz hello.txt
test-fullbench: fullbench datagen
$(QEMU_SYS) ./fullbench -i1
tests/datagencli.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
tests/decodecorpus.c
+5 -5
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2017-present, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <limits.h>
tests/files/huffman-compressed-larger
BIN
View File
Binary file not shown.
tests/fullbench.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
tests/fuzz/Makefile
+5 -7
View File
@@ -1,13 +1,11 @@
# ##########################################################################
# ################################################################
# Copyright (c) 2016-present, Facebook, Inc.
# All rights reserved.
#
# This Makefile is validated for Linux, and macOS targets
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# ##########################################################################
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
CFLAGS ?= -O3
CXXFLAGS ?= -O3
tests/fuzz/fuzz.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
tests/fuzz/fuzz_helpers.h
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
tests/fuzz/regression_driver.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "fuzz.h"
tests/fuzz/simple_decompress.c
+5 -5
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
tests/fuzz/simple_round_trip.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
tests/fuzz/stream_decompress.c
+5 -5
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
tests/fuzz/stream_round_trip.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/**
tests/fuzzer.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
tests/gzip/Makefile
+5 -5
View File
@@ -1,10 +1,10 @@
# ################################################################
# Copyright (c) 2017-present, Yann Collet, Facebook, Inc.
# Copyright (c) 2017-present, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
PRGDIR = ../../programs
@@ -12,7 +12,7 @@ VOID = /dev/null
export PATH := .:$(PATH)
.PHONY: all
#all: test-gzip-env
#all: test-gzip-env
all: test-helin-segv test-hufts test-keep test-list test-memcpy-abuse test-mixed
all: test-null-suffix-clobber test-stdin test-trailing-nul test-unpack-invalid
all: test-zdiff test-zgrep-context test-zgrep-f test-zgrep-signal test-znew-k test-z-suffix
tests/invalidDictionaries.c
+9
View File
@@ -1,3 +1,12 @@
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stddef.h>
#include "zstd.h"
tests/legacy.c
+5 -6
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2017-present, Facebook, Inc.
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*
@@ -226,4 +226,3 @@ const char* const EXPECTED =
"snowden is snowed in / he's now then in his snow den / when does the snow end?\n"
"goodbye little dog / you dug some holes in your day / they'll be hard to fill.\n"
"when life shuts a door, / just open it. it’s a door. / that is how doors work.\n";
tests/longmatch.c
+10
View File
@@ -1,3 +1,13 @@
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdio.h>
#include <stddef.h>
#include <stdlib.h>
tests/namespaceTest.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
tests/paramgrill.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
tests/playTests.sh
+10 -1
View File
@@ -291,8 +291,10 @@ $ECHO "- Create dictionary with wrong dictID parameter order (must fail)"
$ZSTD --train *.c ../programs/*.c --dictID -o 1 tmpDict1 && die "wrong order : --dictID must be followed by argument "
$ECHO "- Create dictionary with size limit"
$ZSTD --train *.c ../programs/*.c -o tmpDict2 --maxdict=4K -v
$ECHO "- Create dictionary with small size limit"
$ZSTD --train *.c ../programs/*.c -o tmpDict3 --maxdict=1K -v
$ECHO "- Create dictionary with wrong parameter order (must fail)"
$ZSTD --train *.c ../programs/*.c -o tmpDict2 --maxdict -v 4K && die "wrong order : --maxdict must be followed by argument "
$ZSTD --train *.c ../programs/*.c -o tmpDict3 --maxdict -v 4K && die "wrong order : --maxdict must be followed by argument "
$ECHO "- Compress without dictID"
$ZSTD -f tmp -D tmpDict1 --no-dictID
$ZSTD -d tmp.zst -D tmpDict -fo result
@@ -386,6 +388,13 @@ $ZSTD -t tmpSplit.* && die "bad file not detected !"
./datagen | $ZSTD -c | $ZSTD -t
$ECHO "\n**** golden files tests **** "
$ZSTD -t -r files
$ZSTD -c -r files | $ZSTD -t
$ECHO "\n**** benchmark mode tests **** "
$ECHO "bench one file"
tests/poolTests.c
+38 -3
View File
@@ -1,5 +1,16 @@
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include "pool.h"
#include "threading.h"
#include "util.h"
#include <stddef.h>
#include <stdio.h>
@@ -50,21 +61,45 @@ int testOrder(size_t numThreads, size_t queueSize) {
return 0;
}
void waitFn(void *opaque) {
(void)opaque;
UTIL_sleepMilli(1);
}
/* Tests for deadlock */
int testWait(size_t numThreads, size_t queueSize) {
struct data data;
POOL_ctx *ctx = POOL_create(numThreads, queueSize);
ASSERT_TRUE(ctx);
{
size_t i;
for (i = 0; i < 16; ++i) {
POOL_add(ctx, &waitFn, &data);
}
}
POOL_free(ctx);
return 0;
}
int main(int argc, const char **argv) {
size_t numThreads;
for (numThreads = 1; numThreads <= 4; ++numThreads) {
size_t queueSize;
for (queueSize = 1; queueSize <= 2; ++queueSize) {
for (queueSize = 0; queueSize <= 2; ++queueSize) {
if (testOrder(numThreads, queueSize)) {
printf("FAIL: testOrder\n");
return 1;
}
if (testWait(numThreads, queueSize)) {
printf("FAIL: testWait\n");
return 1;
}
}
}
printf("PASS: testOrder\n");
(void)argc;
(void)argv;
return (POOL_create(0, 1) || POOL_create(1, 0)) ? printf("FAIL: testInvalid\n"), 1
: printf("PASS: testInvalid\n"), 0;
return (POOL_create(0, 1)) ? printf("FAIL: testInvalid\n"), 1
: printf("PASS: testInvalid\n"), 0;
return 0;
}
tests/roundTripCrash.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
/*
tests/symbols.c
+10
View File
@@ -1,3 +1,13 @@
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#include <stdio.h>
#include "zstd_errors.h"
#define ZSTD_STATIC_LINKING_ONLY
tests/test-zstd-speed.py
+5 -5
View File
@@ -1,13 +1,13 @@
#! /usr/bin/env python3
#
# ################################################################
# Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
#
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ##########################################################################
# Limitations:
# - doesn't support filenames with spaces
tests/test-zstd-versions.py
+5 -5
View File
@@ -1,14 +1,14 @@
#!/usr/bin/env python3
"""Test zstd interoperability between versions"""
#
# ################################################################
# Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
# All rights reserved.
#
# This source code is licensed under the BSD-style license found in the
# LICENSE file in the root directory of this source tree. An additional grant
# of patent rights can be found in the PATENTS file in the same directory.
#
# This source code is licensed under both the BSD-style license (found in the
# LICENSE file in the root directory of this source tree) and the GPLv2 (found
# in the COPYING file in the root directory of this source tree).
# ################################################################
import filecmp
import glob
tests/zbufftest.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
tests/zstreamtest.c
+4 -4
View File
@@ -1,10 +1,10 @@
/**
/*
* Copyright (c) 2016-present, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
zlibWrapper/examples/zwrapbench.c
+5 -6
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Yann Collet, Przemyslaw Skibinski, Facebook, Inc.
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -36,7 +36,6 @@
#endif
/*-************************************
* Constants
**************************************/
zlibWrapper/gzcompatibility.h
+8 -8
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Przemyslaw Skibinski, Facebook, Inc.
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
@@ -12,9 +12,9 @@
#if ZLIB_VERNUM <= 0x1240
ZEXTERN int ZEXPORT gzclose_r OF((gzFile file));
ZEXTERN int ZEXPORT gzclose_w OF((gzFile file));
ZEXTERN int ZEXPORT gzbuffer OF((gzFile file, unsigned size));
ZEXTERN int ZEXPORT gzbuffer OF((gzFile file, unsigned size));
ZEXTERN z_off_t ZEXPORT gzoffset OF((gzFile file));
#if !defined(_WIN32) && defined(Z_LARGE64)
# define z_off64_t off64_t
#else
@@ -38,7 +38,7 @@ struct gzFile_s {
#if ZLIB_VERNUM <= 0x1270
#if defined(_WIN32) && !defined(Z_SOLO)
# include <stddef.h> /* for wchar_t */
# include <stddef.h> /* for wchar_t */
ZEXTERN gzFile ZEXPORT gzopen_w OF((const wchar_t *path,
const char *mode));
#endif
zlibWrapper/gzlib.c
+1 -1
View File
@@ -1,5 +1,5 @@
/* gzlib.c contains minimal changes required to be compiled with zlibWrapper:
* - gz_statep was converted to union to work with -Wstrict-aliasing=1 */
* - gz_statep was converted to union to work with -Wstrict-aliasing=1 */
/* gzlib.c -- zlib functions common to reading and writing gzip files
* Copyright (C) 2004-2017 Mark Adler
zlibWrapper/gzread.c
+2 -2
View File
@@ -1,6 +1,6 @@
/* gzread.c contains minimal changes required to be compiled with zlibWrapper:
* - gz_statep was converted to union to work with -Wstrict-aliasing=1 */
* - gz_statep was converted to union to work with -Wstrict-aliasing=1 */
/* gzread.c -- zlib functions for reading gzip files
* Copyright (C) 2004, 2005, 2010, 2011, 2012, 2013, 2016 Mark Adler
* For conditions of distribution and use, see http://www.zlib.net/zlib_license.html
zlibWrapper/gzwrite.c
+2 -2
View File
@@ -1,6 +1,6 @@
/* gzwrite.c contains minimal changes required to be compiled with zlibWrapper:
* - gz_statep was converted to union to work with -Wstrict-aliasing=1 */
* - gz_statep was converted to union to work with -Wstrict-aliasing=1 */
/* gzwrite.c -- zlib functions for writing gzip files
* Copyright (C) 2004-2017 Mark Adler
* For conditions of distribution and use, see http://www.zlib.net/zlib_license.html
zlibWrapper/zstd_zlibwrapper.c
+5 -5
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Przemyslaw Skibinski, Facebook, Inc.
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
zlibWrapper/zstd_zlibwrapper.h
+9 -9
View File
@@ -1,10 +1,10 @@
/**
* Copyright (c) 2016-present, Przemyslaw Skibinski, Facebook, Inc.
/*
* Copyright (c) 2016-present, Przemyslaw Skibinski, Yann Collet, Facebook, Inc.
* All rights reserved.
*
* This source code is licensed under the BSD-style license found in the
* LICENSE file in the root directory of this source tree. An additional grant
* of patent rights can be found in the PATENTS file in the same directory.
* This source code is licensed under both the BSD-style license (found in the
* LICENSE file in the root directory of this source tree) and the GPLv2 (found
* in the COPYING file in the root directory of this source tree).
*/
#ifndef ZSTD_ZLIBWRAPPER_H
@@ -32,7 +32,7 @@ const char * zstdVersion(void);
/*** COMPRESSION ***/
/* ZWRAP_useZSTDcompression() enables/disables zstd compression during runtime.
By default zstd compression is disabled. To enable zstd compression please use one of the methods:
- compilation with the additional option -DZWRAP_USE_ZSTD=1
- compilation with the additional option -DZWRAP_USE_ZSTD=1
- using '#define ZWRAP_USE_ZSTD 1' in source code before '#include "zstd_zlibwrapper.h"'
- calling ZWRAP_useZSTDcompression(1)
All above-mentioned methods will enable zstd compression for all threads.
@@ -45,13 +45,13 @@ int ZWRAP_isUsingZSTDcompression(void);
/* Changes a pledged source size for a given compression stream.
It will change ZSTD compression parameters what may improve compression speed and/or ratio.
The function should be called just after deflateInit() or deflateReset() and before deflate() or deflateSetDictionary().
It's only helpful when data is compressed in blocks.
There will be no change in case of deflateInit() or deflateReset() immediately followed by deflate(strm, Z_FINISH)
It's only helpful when data is compressed in blocks.
There will be no change in case of deflateInit() or deflateReset() immediately followed by deflate(strm, Z_FINISH)
as this case is automatically detected. */
int ZWRAP_setPledgedSrcSize(z_streamp strm, unsigned long long pledgedSrcSize);
/* Similar to deflateReset but preserves dictionary set using deflateSetDictionary.
It should improve compression speed because there will be less calls to deflateSetDictionary
It should improve compression speed because there will be less calls to deflateSetDictionary
When using zlib compression this method redirects to deflateReset. */
int ZWRAP_deflateReset_keepDict(z_streamp strm);