facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,379 Commits 2 Branches 0 Tags
Commit Graph

9379 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Jonathan Ringer
e1f85dbca3
pzstd: fix linking for static builds 2021-07-09 13:21:15 -07:00
binhdvo
b3e372c171
Merge pull request #2717 from binhdvo/bootcamp 
Proactively skip huffman compression based on sampling where non-comp…
 2021-07-01 10:39:58 -04:00
sen
d90bc0e0b6
Merge pull request #2720 from senhuang42/remove_folder 
Remove folder when done with test
 2021-06-30 14:37:21 -04:00
senhuang42
f5f6cc2e48 Remove folder when done with test 2021-06-30 13:09:00 -04:00
Binh Vo
dc5b693f1e Proactively skip huffman compression based on sampling where non-compressibility is suspected 2021-06-30 11:02:47 -04:00
Nick Terrell
7b43e69b45
Merge pull request #2655 from heitbaum/patch-1 
[trace] remove zstd_trace.c reference from freestanding
 2021-06-29 18:33:40 -07:00
Nick Terrell
609be382ac
Merge pull request #2719 from danlark1/danlark_iwyu 
Include what you use in zstd_ldm_geartab
 2021-06-29 16:53:10 -07:00
Nick Terrell
094b26081f
Merge pull request #2689 from danlark1/dev 
Optimize zstd decompression by another x%
 2021-06-29 11:34:36 -07:00
Danila Kutenin
e855b78be6 Include what you use in zstd_ldm_geartab 2021-06-29 17:57:53 +01:00
Danila Kutenin
2c2c9e7dfd Add possible improvements for gcc-11 2021-06-29 09:06:47 +01:00
Yann Collet
88f107b5b9
Merge pull request #2716 from facebook/fix2701 
remove invalid test
 2021-06-24 10:57:18 -07:00
sen
45d707e908
Merge pull request #2715 from senhuang42/sequence_api_3 
[RFC] Add internal API for converting ZSTD_Sequence into seqStore
 2021-06-24 13:02:11 -04:00
Yann Collet
139a2f1e0c remove invalid test 
`--mt` is no longer supported by `zstreamtest`
(relevant API entry point has been removed from `libzstd`).

fix #2701
 2021-06-24 09:07:55 -07:00
senhuang42
76466dfadf Add simple API for converting ZSTD_Sequence into seqStore 2021-06-23 12:10:48 -04:00
Érico Nogueira
4d09952701 [lib] Fix libzstd.pc for lib-mt builds 
Add the libzstd.pc target to the lib target in lib/Makefile, which makes
it inherit LDFLAGS_DYNLIB from the lib-mt target. This allows us to add
a Libs.private field to libzstd.pc which gets conditionally populated
with '-pthread'.

The 1.5.0 release notes mention that the static library isn't
multi-threaded by default, due to concern for people building static
binaries with libzstd:

   Now the dynamic library supports multi-threaded compression by
   default.  Note that this property is not extended to the static
   library because doing so would have impacted the build script of
   existing client applications (requiring them to add -pthread to their
   recipe), thus potentially breaking their build.

To get closer to being able to enable multi-threading for all library
builds by default, this commit makes it so that any libzstd consumer
using pkg-config gets the correct flags.

We also fix the indentation of the rule for libzstd.pc and move it
outside the if/endif block for install rules (which uses a list of OSs
where the rules were validated), so the rule is available for all users
of the 'lib*' targets.
 2021-06-19 17:42:24 -03:00
Felix Handte
628f65b79c
Merge pull request #2714 from luisdallos/build-macros-typos 
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
 2021-06-18 16:16:44 -04:00
Usuario
8bdce1ff97 lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros 2021-06-18 10:07:39 -04:00
binhdvo
b99d9f5ebd
Merge pull request #2710 from binhdvo/bootcamp 
Add option to use logical cores for default threads
 2021-06-17 09:06:18 -04:00
Binh Vo
6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
binhdvo
0152435ab0
Merge pull request #2708 from binhdvo/skippable 
Add API for fetching skippable frame content
 2021-06-14 19:00:31 -04:00
Binh Vo
9d9f7680f8 Add API for fetching skippable frame content 2021-06-14 16:01:28 -04:00
Nick Terrell
6917c4ecdf
Merge pull request #2709 from terrelln/bounds-check-fix 
[fix] Add missing bounds checks during compression
 2021-06-14 12:33:50 -07:00
Nick Terrell
05b6773fbc [fix] Add missing bounds checks during compression 
* The block splitter missed a bounds check, so when the buffer is too small it
  passes an erroneously large size to `ZSTD_entropyCompressSeqStore()`, which
  can then write the compressed data past the end of the buffer. This is a new
  regression in v1.5.0 when the block splitter is enabled. It is either enabled
  explicitly, or implicitly when using the optimal parser and `ZSTD_compress2()`
  or `ZSTD_compressStream*()`.
* `HUF_writeCTable_wksp()` omits a bounds check when calling
  `HUF_compressWeights()`. If it is called with `dstCapacity == 0` it will pass
  an erroneously large size to `HUF_compressWeights()`, which can then write
  past the end of the buffer. This bug has been present for ages. However, I
  believe that zstd cannot trigger the bug, because it never calls
  `HUF_compress*()` with `dstCapacity == 0` because of [this check][1].

Credit to: Oss-Fuzz

[1]: 89127e5ee2/lib/compress/zstd_compress_literals.c (L100)
 2021-06-14 11:35:33 -07:00
binhdvo
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp 
Add support for negative values in advanced flags
 2021-06-11 14:08:23 -04:00
Binh Vo
6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
koala
e00412f6b5
Z_PREFIX zError function 
When a project use zError function,linker can not find z_zError function
 2021-06-11 19:29:27 +08:00
Felix Handte
67a25968f0
Merge pull request #2702 from felixhandte/human_size_output 
Format File Sizes Human-Readable in the CLI
 2021-06-10 16:53:43 -04:00
W. Felix Handte
8c00807bbc Whitespace Fixes to Improve Cross-Line Alignment 2021-06-10 13:28:38 -04:00
W. Felix Handte
94cf57bb13 Update Tests to Reflect New Formatting 2021-06-10 13:14:18 -04:00
W. Felix Handte
87e94e3e39 Convert Other Size Displays to Use Human-Readable Formatting 2021-06-10 12:57:46 -04:00
W. Felix Handte
2af3687c50 Switch to Binary Size Prefixes (e.g., "MB" -> "MiB") 
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
 2021-06-10 12:53:07 -04:00
W. Felix Handte
9c340ce014 Require -vv to Enable Full Precision 2021-06-10 12:53:07 -04:00
W. Felix Handte
bc46b6efe4 Apply to Other Print Statement as Well 2021-06-10 12:53:07 -04:00
W. Felix Handte
7e0058848c Fix Whitespace 2021-06-10 12:53:07 -04:00
W. Felix Handte
93bb368d74 Change Suffix (e.g., "G" -> " GB") 2021-06-10 12:53:07 -04:00
W. Felix Handte
464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
W. Felix Handte
9b67219b1e Fix Integer Constants; Fix Comparison 2021-06-10 12:53:07 -04:00
W. Felix Handte
bbb81c8801 Avoid snprintf() in Preparing Human-Readable Sizes; Improve Formatting 
This produces the following formatting:

   Size    | `zstd` | `ls -lh`
---------- | ------ | --------
1          | 1      | 1
12         | 12     | 12
123        | 123    | 123
1234       | 1.21K  | 1.3K
12345      | 12.1K  | 13K
123456     | 121K   | 121K
1234567    | 1.18M  | 1.2M
12345678   | 11.8M  | 12M
123456789  | 118M   | 118M
1234567890 | 1.15G  | 1.2G
999        | 999    | 999
1000       | 1000   | 1000
1001       | 1001   | 1001
1023       | 1023   | 1023
1024       | 1.000K | 1.0K
1025       | 1.00K  | 1.1K
999999     | 977K   | 977K
1000000    | 977K   | 977K
1000001    | 977K   | 977K
1023999    | 1000K  | 1000K
1024000    | 1000K  | 1000K
1024001    | 1000K  | 1001K
1048575    | 1024K  | 1.0M
1048576    | 1.000M | 1.0M
1048577    | 1.00M  | 1.1M

This was produced with the following invocation:

```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
  head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
 2021-06-10 12:53:07 -04:00
Scott Baker
8e0a9695d7 Attempt to fix a failing test with help from @aqrit 2021-06-10 12:53:07 -04:00
Scott Baker
1eb852854b Some fixes to address things @felixhandte found 2021-06-10 12:53:07 -04:00
Scott Baker
376a2730a8 Try enabling the BIG strings now the unsigned long long is in effect 2021-06-10 12:53:07 -04:00
Scott Baker
20b9b00b41 Try unsigned long long 2021-06-10 12:53:07 -04:00
Scott Baker
64385ef7cb Update humanSize() to skip the big numbers (it requires 64 bit) 2021-06-10 12:53:07 -04:00
Scott Baker
1ef6f3d079 Use unsigned long instead to help with some tests 2021-06-10 12:53:07 -04:00
Scott Baker
e5fc830795 human_size() should use size_t 2021-06-10 12:53:07 -04:00
Scott Baker
35576e63ce Convert tabs to spaces 2021-06-10 12:53:07 -04:00
Scott Baker
77001f00fb Use human_size() on the "multiple files compressed" output also 2021-06-10 12:53:07 -04:00
Scott Baker
894698d3b6 Use human_size() in the benchmark output also 2021-06-10 12:53:07 -04:00
Scott Baker
4e0d9f1cc8 Move the variable declarations to the top 2021-06-10 12:53:07 -04:00
Scott Baker
eefdbcd93a Make the variable types match 2021-06-10 12:53:07 -04:00