1598e6c634
Async write for decompression ( #2975 )
...
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
2022-01-21 13:55:41 -08:00
ef1eafded3
Merge pull request #3009 from terrelln/asm-flags
...
[build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
2022-01-18 15:48:11 -08:00
8ea3d57de4
[build][asm] Pass ASFLAGS to the assembler instead of CFLAGS
...
* Add `-Wa,--noexecstack` to both `ASFLAGS` and `CFLAGS`
* Pass `ASFLAGS` to `.S` compilation instead of `CFLAGS`
Fixes #3006 .
2022-01-18 15:11:29 -08:00
57a86d9ec6
Avoid updating timestamps when the destination is stdout
...
Fixes: 9cd6c1ff4dhttps://github.com/facebook/zstd/issues/2997
2022-01-13 16:47:18 -05:00
f4a552a3fa
Merge pull request #2987 from felixhandte/prepare-v1.5.2
...
Prepare v1.5.2
2022-01-11 17:47:39 -05:00
78633272a0
Merge pull request #2982 from terrelln/decompress-progress-0
...
Fix stderr progress logging for decompression
2022-01-11 10:15:15 -08:00
df5013b463
ZSTD CLI: Use buffered output for improved performance
2022-01-07 17:10:13 -08:00
308a11b8e8
Fix stderr progress logging for decompression
...
When decompressing with `-q` and an output file, the progress bar was mistakenly printed. This is a minimal fix, with a larger refactor to be stacked on top of it.
Fixes #2967 .
2022-01-07 15:09:56 -08:00
e1323744b6
Update Docs
2022-01-07 14:14:26 -05:00
4bd96a61f1
Avoid xxHash Dependency by Inlining
...
xxHash symbols are present in `libzstd.so`, but they are local and therefore
unavailable outside the lib. There are two possible solutions to the problem.
We could make those symbols global, or we could remove the dependency.
This commit chooses the latter approach. I suppose this comes at the cost of
code size / build time. I'm open to comments on whether this is a good thing
to do, especially since this will apply even when we are statically linking
everything.
2022-01-06 00:25:26 -05:00
fc946d131b
Remove Unused Include
2022-01-06 00:25:26 -05:00
4620ce6a9a
Makefiles: Add `noexecstack` Options to Compilation and Linking
...
Hopefully this marks the binary artifacts `noexecstack` even on platforms
where binaries default to true.
2022-01-05 15:12:31 -05:00
abc694f53e
update man pages
2021-12-20 14:24:06 -08:00
c5f1e826ca
Merge pull request #2925 from embg/dict_training_sample_limit_size
...
Allow user to specify memory limit for dictionary training
2021-12-15 15:58:17 -05:00
71c0c07c19
Allow user to specify memory limit for dictionary training
2021-12-14 14:29:01 -05:00
0c26d98c0d
Merge pull request #2910 from felixhandte/reject-irregular-dicts
...
Reject Irregular Dictionary Files
2021-12-09 11:44:37 -05:00
9985e10fda
Reject Irregular Dictionary Files
...
I hadn't seen #2890 , so I wrote my own version. I like this approach a little
better, since it does an explicit check for a regular file, rather than
passing a magic value.
Addresses #2874 .
2021-12-08 16:17:04 -05:00
38dfc4699e
Imply -q when stderr is not a tty ( #2884 )
...
* Imply -q when stderr is not a tty
2021-12-07 16:56:19 -05:00
e7b0ae385e
Merge pull request #2890 from 15596858998/dec_1201
...
fixbug CLI's -D fails when the argument is not a regular file
2021-12-06 13:18:15 -05:00
a134737ad4
Merge pull request #2687 from sapiippo/dev
...
Makefile: fix build for mingw
2021-12-02 10:53:02 -08:00
1e514feec6
Makefile: sort all wildcard file list expansions
...
Otherwise the order is non-deterministic and breaks
reproducible builds.
2021-12-02 12:04:11 +01:00
3133d1e86e
Merge pull request #2876 from 15596858998/dev
...
Solve the bug of extra output newline character
2021-12-01 15:10:08 -08:00
a8adfa7f67
fixbug CLI's -D fails when the argument is not a regular file
2021-12-01 21:47:35 +08:00
d7e1736375
Fix build for cygwin/bsd ( #2882 )
2021-11-29 14:11:39 -05:00
7abebc847b
Clarify documentation for -c ( #2883 )
2021-11-29 14:10:43 -05:00
e315a047c7
更新 fileio.c
...
Solve the logic problem of wrong output of newline characters.
2021-11-23 22:08:23 +08:00
ebbd675998
Fix typos
2021-11-13 10:04:04 +02:00
7fbd126e08
Suppress spurious unused parameter warning
2021-11-12 10:44:26 -08:00
375e3aad6c
Ensure formatting directives for displaying size_t are portable
2021-11-11 13:17:32 -08:00
365c91194c
Ensure print*CParams functions are only defined when used
2021-11-11 12:14:56 -08:00
df9b7755cb
Fix const-ness of FIO_displayCompressionParameters
2021-11-10 17:25:27 -08:00
63fe6198ed
Display --zstd= subparameters in command-line ready form in verbose mode
2021-11-10 17:20:43 -08:00
0665d4c1c2
Display command line parameters with concrete values in verbose mode
2021-11-05 12:01:20 -07:00
b3888193d9
Report memory required to decompress while compressing in verbose mode
2021-11-04 16:25:40 -07:00
23dd28df67
minor improvements to benchmark display
2021-10-26 23:23:30 -07:00
52598d54e9
Limit train samples ( #2809 )
...
* Limit training samples size to 2GB
* simplified DISPLAYLEVEL() macro to use global vqriable instead of local.
* refactored training samples loading
* fixed compiler warning
* addressed comments from the pull request
* addressed @terrelln comments
* missed some fixes
* fixed type mismatch
* Fixed bug passing estimated number of samples rather insted of the loaded number of samples.
Changed unit conversion not to use bit-shifts.
* fixed a declaration after code
* fixed type conversion compile errors
* fixed more type castting
* fixed more type mismatching
* changed sizes type to size_t
* move type casting
* more type cast fixes
2021-10-04 17:47:52 -07:00
7868f38019
Merge pull request #2747 from Helflym/dev
...
Add AIX support in Makefile
2021-10-01 08:13:39 -07:00
8150891939
regenerated zstdless.1
2021-09-22 14:48:51 -07:00
3addf2f277
updated zstdgrep man page
2021-09-22 14:30:59 -07:00
70b36c2308
update zstdgrep doc
...
to mention ripgrep alternative which transparently supports zstd-compressed files.
2021-09-22 14:30:06 -07:00
999f8778af
updated man pages
...
using ronn-ng
2021-09-22 14:18:24 -07:00
2832bbbbdc
emphasize usage of -r in documentation
...
notably as a way to overcome shell expansion limitations,
notably in a scenario of dictionary training.
2021-09-22 14:00:20 -07:00
b5c35d7ea3
Use new paramSwitch enum for LCM, row matchfinder, and block splitter
2021-09-21 14:22:02 -04:00
8bf699aa59
[build] Add support for ASM files in Make + CMake
...
* Extract out common portion of `lib/Makefile` into `lib/libzstd.mk`.
Most relevantly, the way we find library files.
* Use `lib/libzstd.mk` in the other Makefiles instead of repeating the
same code.
* Add a test `tests/test-variants.sh` that checks that the builds of
`make -C programs allVariants` are correct, and run it in Actions.
* Adds support for ASM files in the CMake build.
The Meson build is not updated because it lists every file in zstd,
and supports ASM off the bat, so the Huffman ASM commit will just add
the ASM file to the list.
The Visual Studios build is not updated because I'm not adding ASM
support to Visual Studios yet.
2021-09-17 14:13:53 -07:00
f58e63bee7
Merge branch 'dev' into opt_investigation
2021-09-12 01:42:49 -07:00
640c5b1f77
fix automated_benchmarking
...
make it able to process text output sent into either stdout or stderr
2021-09-12 01:36:18 -07:00
d68aa19a2f
Merge pull request #2749 from felixhandte/zstd-fast-pipelined
...
Pipelined Implementation of ZSTD_fast (~+5% Speed)
2021-09-09 17:05:30 -04:00
30fe49af4e
Fix patch-from help msg typo
2021-09-07 10:08:35 -04:00
08ceda3dfc
new statistics update policy
...
small general compression ratio improvement for btopt+ strategies/
2021-09-04 00:52:44 -07:00
eab692211e
removed pretty-print of sizes in benchmark
...
This is less appropriate for this mode :
benchmark is about accuracy,
it's important to read the exact values.
2021-09-03 12:51:02 -07:00
ab8aa49b8d
Fix Benchmark Corruption Display
2021-09-01 14:15:03 -04:00
f21977c5e6
fix playTests.sh when EXE_PREFIX not null
2021-08-29 17:20:12 -07:00
dce48f53df
Fix benchzstd error message
2021-08-23 19:10:16 -04:00
399849e236
Makefile: add AIX support
...
For lib, AIX linker doesn't allow --soname.
2021-08-13 10:25:14 +02:00
9cd6c1ff4d
Update mtime and atime for Written Files
2021-08-04 14:49:56 -04:00
a719edbbc3
Pull `utime()` Call into Helper
2021-08-04 14:49:00 -04:00
a5f518ae27
Change zstdcli's main() declaration due to -Wmain on some compilers
2021-07-14 19:55:47 +03:00
d4ad02c721
Add support for MCST LCC compiler
2021-07-10 03:57:06 +03:00
6a46e38deb
Add option to use logical cores for default threads
2021-06-16 15:46:17 -04:00
89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp
...
Add support for negative values in advanced flags
2021-06-11 14:08:23 -04:00
6fad35c6a1
Add support for negative levels in --adapt=min and --adapt=max"
2021-06-11 12:13:09 -04:00
8c00807bbc
Whitespace Fixes to Improve Cross-Line Alignment
2021-06-10 13:28:38 -04:00
87e94e3e39
Convert Other Size Displays to Use Human-Readable Formatting
2021-06-10 12:57:46 -04:00
2af3687c50
Switch to Binary Size Prefixes (e.g., "MB" -> "MiB")
...
Suggested by @aqrit, a little more verbose, but hopefully addresses a real
ambiguity.
2021-06-10 12:53:07 -04:00
9c340ce014
Require `-vv` to Enable Full Precision
2021-06-10 12:53:07 -04:00
bc46b6efe4
Apply to Other Print Statement as Well
2021-06-10 12:53:07 -04:00
7e0058848c
Fix Whitespace
2021-06-10 12:53:07 -04:00
93bb368d74
Change Suffix (e.g., "G" -> " GB")
2021-06-10 12:53:07 -04:00
464bfb022e
In Verbose Mode, Preserve Full Precision Where Possible
2021-06-10 12:53:07 -04:00
9b67219b1e
Fix Integer Constants; Fix Comparison
2021-06-10 12:53:07 -04:00
bbb81c8801
Avoid `snprintf()` in Preparing Human-Readable Sizes; Improve Formatting
...
This produces the following formatting:
Size | `zstd` | `ls -lh`
---------- | ------ | --------
1 | 1 | 1
12 | 12 | 12
123 | 123 | 123
1234 | 1.21K | 1.3K
12345 | 12.1K | 13K
123456 | 121K | 121K
1234567 | 1.18M | 1.2M
12345678 | 11.8M | 12M
123456789 | 118M | 118M
1234567890 | 1.15G | 1.2G
999 | 999 | 999
1000 | 1000 | 1000
1001 | 1001 | 1001
1023 | 1023 | 1023
1024 | 1.000K | 1.0K
1025 | 1.00K | 1.1K
999999 | 977K | 977K
1000000 | 977K | 977K
1000001 | 977K | 977K
1023999 | 1000K | 1000K
1024000 | 1000K | 1000K
1024001 | 1000K | 1001K
1048575 | 1024K | 1.0M
1048576 | 1.000M | 1.0M
1048577 | 1.00M | 1.1M
This was produced with the following invocation:
```
for N in 1 12 123 1234 12345 123456 1234567 12345678 123456789 1234567890 999 1000 1001 1023 1024 1025 999999 1000000 1000001 1023999 1024000 1024001 1048575 1048576 1048577; do
head -c $N /dev/urandom > r$N
done
./zstd -i1 -b1 -S r1 r12 r123 r1234 r12345 r123456 r1234567 r12345678 r123456789 r1234567890 r999 r1000 r1001 r1023 r1024 r1025 r999999 r1000000 r1000001 r1023999 r1024000 r1024001 r1048575 r1048576 r1048577
```
2021-06-10 12:53:07 -04:00
8e0a9695d7
Attempt to fix a failing test with help from @aqrit
2021-06-10 12:53:07 -04:00
1eb852854b
Some fixes to address things @felixhandte found
2021-06-10 12:53:07 -04:00
376a2730a8
Try enabling the BIG strings now the unsigned long long is in effect
2021-06-10 12:53:07 -04:00
20b9b00b41
Try unsigned long long
2021-06-10 12:53:07 -04:00
64385ef7cb
Update humanSize() to skip the big numbers (it requires 64 bit)
2021-06-10 12:53:07 -04:00
1ef6f3d079
Use unsigned long instead to help with some tests
2021-06-10 12:53:07 -04:00
e5fc830795
human_size() should use size_t
2021-06-10 12:53:07 -04:00
35576e63ce
Convert tabs to spaces
2021-06-10 12:53:07 -04:00
77001f00fb
Use human_size() on the "multiple files compressed" output also
2021-06-10 12:53:07 -04:00
894698d3b6
Use human_size() in the benchmark output also
2021-06-10 12:53:07 -04:00
4e0d9f1cc8
Move the variable declarations to the top
2021-06-10 12:53:07 -04:00
eefdbcd93a
Make the variable types match
2021-06-10 12:53:07 -04:00
b6b23dfe64
Convert names to CamelCase
2021-06-10 12:53:07 -04:00
b70175e5ec
Put the human_size() function in util.c
2021-06-10 12:53:07 -04:00
26fab1d963
Make the CLI output the file sizes in human readable format
2021-06-10 12:53:07 -04:00
325952f878
Revert "Add support for --long-param flag, fix #2104"
2021-06-09 15:35:43 -04:00
6583fa3f0a
Add support for --long-param flag
2021-06-09 14:07:52 -04:00
d2f31b6627
Fix --progress flag to properly control progress display and default progress display on when using -v
2021-06-08 17:24:38 -04:00
1e17184ad0
Add documentation for --patch-from
2021-06-03 11:12:27 -04:00
fa2a6d4746
Makefile: fix build for mingw
...
Add ${EXT} to required places to make install succeed for mingw build.
2021-05-27 16:09:37 +03:00
8fae35591e
Merge branch 'dev' of github.com:facebook/zstd into dev
2021-05-12 13:12:30 -07:00
d4548c96cb
fileio: clamp value of windowLog in patch-mode ( #2637 )
...
With small enough input files, the inferred value of fileWindowLog could
be smaller than ZSTD_WINDOWLOG_MIN.
This can be reproduced like so:
$ echo abc > small
$ echo abcdef > small2
$ zstd --patch-from small small2 -o patch
previously, this would fail with the error "zstd: error 11 : Parameter is out of bound"
2021-05-12 16:11:15 -04:00
cb0cad9b79
reduce Max nb Workers to 64 in 32-bit mode
...
and restored limit to 256 when in 64-bit mode
(it was reduced to 200 to give more room for 32-bit).
This should fix test instability issues
using lot of threads in 32-bit environments.
2021-05-12 13:10:25 -07:00
8a53a882f2
updated generated man pages for v1.5.0 ( #2635 )
2021-05-11 18:17:31 -04:00
13449d7ce1
Add PHONY targets to makefiles ( #2629 )
2021-05-07 14:03:19 -04:00
6030cdfede
Add --progress flag ( #2595 )
2021-05-06 14:50:28 -04:00
2f7bbd6539
Merge pull request #2620 from facebook/winFilelist
...
fix --filelist compatibility with Windows cr+lf line ending
2021-05-06 11:35:16 -07:00
df05b2ba7c
fix --filelist compatibility with Windows cr+lf line ending
2021-05-05 18:01:55 -07:00
9750f3c87b
improved benchmark experience on Windows
...
benchmark results are not progressively displayed on Windows terminal.
For long benchmark sessions, nothing is displayed,
until the end, where everything is flushed.
Force display to be flushed after each update.
Updates happen roughtly every second, or even less,
so it's not a substantial workload.
2021-05-05 16:52:21 -07:00
Yonatan Komornik
Nick Terrell
Mike Gilbert
Felix Handte
Yann Collet
Yonatan Komornik
Yann Collet
Elliot Gorokhovsky
Elliot Gorokhovsky
binhdvo
Alexander Kanavin
15596858998
Dimitris Apostolou
Kevin Svetlitski
Yann Collet
stanjo74
senhuang42
Clément Chigot
makise-homura
Binh Vo
Scott Baker
Samuli Piippo
Olivier Perret