senhuang42
aa1957477b
Improve Huffman sorting algorithm
2021-08-04 12:43:34 -04:00
Nick Terrell
6ee70bae46
Merge pull request #2733 from terrelln/huf-cspeed
...
[HUF] Improve Huffman encoding speed
2021-08-03 12:59:54 -04:00
Nick Terrell
d8a0797268
[fuzz] Add Huffman round trip fuzzer
...
* Add a Huffman round trip fuzzer
* Fix two minor bugs in Huffman that aren't exposed in zstd
- Incorrect weight comparison (weights are allowed to be equal to
table log).
- HUF_compress1X_usingCTable_internal() can return compressed
size >= source size, so the assert that `cSize <= 65535` isn't
correct, and it needs to be checked instead.
2021-08-03 08:10:06 -07:00
sen
5c46f62006
Merge pull request #2677 from senhuang42/ci_overhaul_2
...
[CI][2/2] Migrate CI tests which (currently) fail
2021-08-02 09:55:49 -04:00
Sen Huang
5ec7897a26
Fix static analyzer warnings
2021-07-29 09:11:12 -07:00
Sen Huang
43aad811db
Add VS2019 test, migrate minimal decompressor macros to GHActions
2021-07-29 08:49:36 -07:00
Nick Terrell
32cfe15edc
Merge pull request #2724 from jonringer/fix-pzstd-static-build
...
pzstd: fix linking for static builds
2021-07-28 14:53:01 -07:00
Nick Terrell
d69e007cca
Merge pull request #2707 from koalabearguo/dev
...
Z_PREFIX zError function
2021-07-28 14:52:15 -07:00
Felix Handte
495597699c
Merge pull request #2729 from felixhandte/fix-ddss-load
...
Fix DDSS Load
2021-07-28 13:12:50 -04:00
Nick Terrell
46f2710562
[HUF] Improve Huffman encoding speed
...
Improve Huffman encoding speed by 20% for gcc and 10% for clang.
| Compiler | Benchmark | Config | Dataset | Ratio | Speed MB/s (dev) | Speed MB/s (huf-cspeed) | Speed MB/s (huf-cspeed - dev) |
|----------|-------------------|---------|-------------|-------|------------------|-------------------------|-------------------------------|
| gcc | compress | level_1 | enwik7 | 2.43 | 253.70 | 258.72 | 2.0% |
| gcc | compress | level_1 | silesia | 2.88 | 341.90 | 348.15 | 1.8% |
| gcc | compress_literals | level_1 | enwik7 | 1.49 | 761.83 | 912.76 | 19.8% |
| gcc | compress_literals | level_1 | silesia | 1.28 | 754.83 | 902.37 | 19.5% |
| gcc | compress_literals | level_7 | enwik7 | 1.29 | 502.81 | 552.79 | 9.9% |
| gcc | compress_literals | level_7 | silesia | 1.11 | 675.97 | 776.44 | 14.9% |
| clang | compress | level_1 | enwik7 | 2.43 | 277.54 | 280.98 | 1.2% |
| clang | compress | level_1 | silesia | 2.88 | 369.98 | 375.46 | 1.5% |
| clang | compress_literals | level_1 | enwik7 | 1.49 | 828.83 | 918.41 | 10.8% |
| clang | compress_literals | level_1 | silesia | 1.28 | 815.81 | 905.41 | 11.0% |
| clang | compress_literals | level_7 | enwik7 | 1.29 | 533.13 | 553.30 | 3.8% |
| clang | compress_literals | level_7 | silesia | 1.11 | 714.52 | 775.38 | 8.5% |
2021-07-27 15:10:35 -07:00
W. Felix Handte
da58821ff2
Fix DDSS Load
...
This PR fixes an incorrect comparison in figuring out `minChain` in
`ZSTD_dedicatedDictSearch_lazy_loadDictionary()`. This incorrect comparison
had been masked by the fact that `idx` was always 1, until @terrelln changed
that in #2726 .
Credit-to: OSS-Fuzz
2021-07-27 11:49:44 -04:00
Nick Terrell
d2b5e5496d
Merge pull request #2726 from terrelln/oss-fuzz-fix
...
[bug-fix] Fix a determinism bug with the DUBT
2021-07-21 13:46:21 -07:00
Nick Terrell
ba044bd6f1
[bug-fix] Fix a determinism bug with the DUBT
...
The DUBT can be non-deterministic if an index is equal to
`ZSTD_DUBT_UNSORTED_MARK`. Ensure that never happens by starting the
indices at 2.
This bug was found by the OSS-Fuzz determinism fuzzer. With this change
the fuzzer test passes. And I've confirmed that this is the root cause,
not just hiding the problem.
Aside: This took me a long time to figure out, because I thought I had
tried this first thing. But, apparantly I messed it up, because when I
was going through it again with @felixhandte, I was pointing out that it
wasn't the case, but it turns out it was.
Credit to: OSS-Fuzz
2021-07-15 13:02:49 -07:00
Yann Collet
b18febe52c
Merge pull request #2725 from makise-homura/mcst-lcc-support
...
Add support for MCST LCC compiler
2021-07-15 00:09:12 +02:00
makise-homura
3cd085cec3
Clarify no-tree-vectorize usage for ICC and LCC
2021-07-14 20:00:44 +03:00
makise-homura
a5f518ae27
Change zstdcli's main() declaration due to -Wmain on some compilers
2021-07-14 19:55:47 +03:00
makise-homura
d4ad02c721
Add support for MCST LCC compiler
2021-07-10 03:57:06 +03:00
Jonathan Ringer
e1f85dbca3
pzstd: fix linking for static builds
2021-07-09 13:21:15 -07:00
binhdvo
b3e372c171
Merge pull request #2717 from binhdvo/bootcamp
...
Proactively skip huffman compression based on sampling where non-comp…
2021-07-01 10:39:58 -04:00
sen
d90bc0e0b6
Merge pull request #2720 from senhuang42/remove_folder
...
Remove folder when done with test
2021-06-30 14:37:21 -04:00
senhuang42
f5f6cc2e48
Remove folder when done with test
2021-06-30 13:09:00 -04:00
Binh Vo
dc5b693f1e
Proactively skip huffman compression based on sampling where non-compressibility is suspected
2021-06-30 11:02:47 -04:00
Nick Terrell
7b43e69b45
Merge pull request #2655 from heitbaum/patch-1
...
[trace] remove zstd_trace.c reference from freestanding
2021-06-29 18:33:40 -07:00
Nick Terrell
609be382ac
Merge pull request #2719 from danlark1/danlark_iwyu
...
Include what you use in zstd_ldm_geartab
2021-06-29 16:53:10 -07:00
Nick Terrell
094b26081f
Merge pull request #2689 from danlark1/dev
...
Optimize zstd decompression by another x%
2021-06-29 11:34:36 -07:00
Danila Kutenin
e855b78be6
Include what you use in zstd_ldm_geartab
2021-06-29 17:57:53 +01:00
Danila Kutenin
2c2c9e7dfd
Add possible improvements for gcc-11
2021-06-29 09:06:47 +01:00
Yann Collet
88f107b5b9
Merge pull request #2716 from facebook/fix2701
...
remove invalid test
2021-06-24 10:57:18 -07:00
sen
45d707e908
Merge pull request #2715 from senhuang42/sequence_api_3
...
[RFC] Add internal API for converting ZSTD_Sequence into seqStore
2021-06-24 13:02:11 -04:00
Yann Collet
139a2f1e0c
remove invalid test
...
`--mt` is no longer supported by `zstreamtest`
(relevant API entry point has been removed from `libzstd`).
fix #2701
2021-06-24 09:07:55 -07:00
senhuang42
76466dfadf
Add simple API for converting ZSTD_Sequence into seqStore
2021-06-23 12:10:48 -04:00
Felix Handte
628f65b79c
Merge pull request #2714 from luisdallos/build-macros-typos
...
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
2021-06-18 16:16:44 -04:00
Usuario
8bdce1ff97
lib/Makefile: Fix small typo in ZSTD_FORCE_DECOMPRESS_* build macros
2021-06-18 10:07:39 -04:00
binhdvo
b99d9f5ebd
Merge pull request #2710 from binhdvo/bootcamp
...
Add option to use logical cores for default threads
2021-06-17 09:06:18 -04:00
Binh Vo
6a46e38deb
Add option to use logical cores for default threads
2021-06-16 15:46:17 -04:00
binhdvo
0152435ab0
Merge pull request #2708 from binhdvo/skippable
...
Add API for fetching skippable frame content
2021-06-14 19:00:31 -04:00
Binh Vo
9d9f7680f8
Add API for fetching skippable frame content
2021-06-14 16:01:28 -04:00
Nick Terrell
6917c4ecdf
Merge pull request #2709 from terrelln/bounds-check-fix
...
[fix] Add missing bounds checks during compression
2021-06-14 12:33:50 -07:00
Nick Terrell
05b6773fbc
[fix] Add missing bounds checks during compression
...
* The block splitter missed a bounds check, so when the buffer is too small it
passes an erroneously large size to `ZSTD_entropyCompressSeqStore()`, which
can then write the compressed data past the end of the buffer. This is a new
regression in v1.5.0 when the block splitter is enabled. It is either enabled
explicitly, or implicitly when using the optimal parser and `ZSTD_compress2()`
or `ZSTD_compressStream*()`.
* `HUF_writeCTable_wksp()` omits a bounds check when calling
`HUF_compressWeights()`. If it is called with `dstCapacity == 0` it will pass
an erroneously large size to `HUF_compressWeights()`, which can then write
past the end of the buffer. This bug has been present for ages. However, I
believe that zstd cannot trigger the bug, because it never calls
`HUF_compress*()` with `dstCapacity == 0` because of [this check][1].
Credit to: Oss-Fuzz
[1]: 89127e5ee2/lib/compress/zstd_compress_literals.c (L100)
2021-06-14 11:35:33 -07:00
binhdvo
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp
...
Add support for negative values in advanced flags
2021-06-11 14:08:23 -04:00
Binh Vo
6fad35c6a1
Add support for negative levels in --adapt=min and --adapt=max"
2021-06-11 12:13:09 -04:00
koala
e00412f6b5
Z_PREFIX zError function
...
When a project use zError function,linker can not find z_zError function
2021-06-11 19:29:27 +08:00
Felix Handte
67a25968f0
Merge pull request #2702 from felixhandte/human_size_output
...
Format File Sizes Human-Readable in the CLI
2021-06-10 16:53:43 -04:00
W. Felix Handte
8c00807bbc
Whitespace Fixes to Improve Cross-Line Alignment
2021-06-10 13:28:38 -04:00
W. Felix Handte
94cf57bb13
Update Tests to Reflect New Formatting
2021-06-10 13:14:18 -04:00
W. Felix Handte
87e94e3e39
Convert Other Size Displays to Use Human-Readable Formatting
2021-06-10 12:57:46 -04:00
W. Felix Handte
2af3687c50
Switch to Binary Size Prefixes (e.g., "MB" -> "MiB")
...
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
2021-06-10 12:53:07 -04:00
W. Felix Handte
9c340ce014
Require -vv to Enable Full Precision
2021-06-10 12:53:07 -04:00
W. Felix Handte
bc46b6efe4
Apply to Other Print Statement as Well
2021-06-10 12:53:07 -04:00
W. Felix Handte
7e0058848c
Fix Whitespace
2021-06-10 12:53:07 -04:00
