Commit Graph

431 Commits (2f7bbd65399669d9ee39a0cc7cb19d13f3001a3a)

Author SHA1 Message Date
W. Felix Handte 33f3e293e8 Allow Reading from Block Devices with `--force` 2021-05-04 16:25:26 -04:00
Nick Terrell 4694423c4f Add and integrate lazy row hash strategy 2021-04-07 09:53:34 -07:00
Nick Terrell a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
senhuang42 444c4650a0 Add newline to end of cli help message 2021-02-17 12:30:42 -05:00
Nick Terrell 54a4998a80 Add basic tracing functionality 2021-02-05 16:28:52 -08:00
W. Felix Handte 8b6a4b5b7c Allow Input From Console When `--force` is Passed
Also update option flag documentation.
2021-01-11 17:53:20 -05:00
Yann Collet 3324e87cff Added library version check 2021-01-07 10:37:27 -08:00
Nick Terrell 66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
senhuang42 7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
Yann Collet f7d4943788
Merge pull request #2330 from senhuang42/fix_stdinout_error_messages
Improve error messages on console input/output
2020-10-06 15:58:37 -07:00
senhuang42 ce56810a32 Modify error messages on console input/output 2020-09-28 12:15:18 -04:00
senhuang42 432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
Yann Collet 05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor
Make multifile (de)compression print out a summary
2020-09-14 14:06:26 -07:00
Yann Collet e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads
Allow environment variable to specify number of threads for compression
2020-09-14 14:04:19 -07:00
senhuang42 a71963c7b8 nbThreads instead of numThreads 2020-09-09 12:40:00 -04:00
senhuang42 243c8dfb41 Add include guards for init_numThreads() 2020-09-08 09:26:16 -04:00
senhuang42 136a0673f6 Add quick documentation for ZSTD_NUMTHREADS in the code 2020-09-07 18:19:31 -04:00
senhuang42 888c385a49 Change param name to ZSTDCLI_NUMTHREADS_DEFAULT 2020-09-07 18:11:42 -04:00
senhuang42 972e063219 Change default num_threads value 2020-09-07 18:09:48 -04:00
senhuang42 5123496104 Adjust function signatures 2020-09-07 13:13:05 -04:00
senhuang42 48bca10792 Address memory leak in CLI from fCtx 2020-09-03 10:14:04 -04:00
senhuang42 3a7d625d6b Cleanup comments, add function to set FIO_ctx_t.nbFilesProcessed 2020-09-01 12:54:21 -04:00
senhuang42 a6414f1247 Integrate refactor into status print for multifiles, adjust logic for printing as needed 2020-09-01 12:34:43 -04:00
senhuang42 565f116a56 Change name to ZSTD_NUMTHREADS 2020-09-01 09:02:23 -04:00
senhuang42 d54566f334 First pass at refactoring and creating new FIO_ctx_t */ 2020-08-28 11:01:04 -04:00
Yann Collet a8c66881e5
Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files
Refreshing progress bar for processing multiple files
2020-08-26 11:54:50 -07:00
senhuang42 ead2387a7e Remove extraneous FIO_setNbFiles() 2020-08-26 12:05:04 -04:00
senhuang42 b6abbc3ce5 Fix single file decompression summary, remove escape code dependency, add currFileIdx field to FIO_prefs, general cleanups/bugfixes 2020-08-26 11:35:07 -04:00
senhuang42 70fa7b37ca Added env var function for threads 2020-08-25 18:49:52 -04:00
senhuang42 da38891a87 Add initial live update with displayLevel = 2, add new field to FIO_prefs_t 2020-08-25 16:46:47 -04:00
senhuang42 e1e41addb3 Adjust #ifndef logic 2020-08-24 17:36:36 -04:00
senhuang42 44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
senhuang42 ffaa0df76d Document change in CLI for --no-check during decompression in --help menu 2020-08-24 09:49:12 -04:00
Yann Collet 15c5e20023 updated documentation 2020-07-23 12:10:57 -07:00
Yann Collet 610c41f90f improved 2-args macro to support both syntaxes
both `--long-commmand=arg` and `--long-command arg`
are supported by macro `NEXT_FIELD()`
so that the long-command only needs to be listed once.

This extends the syntax to support new syntaxes like
`-o=FILE` or `-D=dict`,
though there is no need to advertise this capability for the time being.

Also : added `NEXT_UINT32()`,
which is wrapper around `NEXT_FIELD()`
to read integer parameters.
Use the wrapper to new field, such as `--memlimit`
which can now support both syntaxes too.
2020-07-20 17:41:32 -07:00
Yann Collet 23941eec04 added tests for newly enabled syntax
for --patch-from origin
and --filelist list

Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.

For example :
    zstd file -of dest
used to be disallowed.

It's now allowed, and understood as:
    zstd file -o dest -f
2020-07-17 13:31:15 -07:00
Yann Collet 7280c1dafa extended 2-fields macro usage to number arguments
leading to further simplications
2020-07-17 13:09:23 -07:00
Yann Collet a7e5652672 CLI: simplified handling of separated-fields arguments
this patch reduces complexity associated with
commands requiring a separated arguments
such as :
-o filename
-D dictionary
--output-dir-flat dir
--output-dir-mirror dir

It used to be a multi-stage process with explicit context,
it's now simplified as a single macro.

Thanks to this simplification,
separated arguments logic has also been extended to
--patch-from XXX
--filelist XXX
which extends existing capability using =XXX
--patch-from=XXX
--filelist=XXX

Separated argument is useful for filenames and directories,
as it benefits from shell expansion
such as ~/dir/file
where the ~ is automatically translated by the shell.
In contrast --long-command=FILE does not interpret FILE,
so ~/ is transmitted as is to the main() function,
generally resulting in incorrect file name.
2020-07-17 12:46:36 -07:00
Xin Xie 9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
Tobias Hieta b1cce9a6ad Short version string output
This adds support for just showing the version string
without the full welcome message if the log level is
less than the default. That means that you pass -q once
and you will just see "1.4.5".

This makes it easier to parse in scripts.
2020-06-09 08:40:41 +02:00
Yann Collet f13eb2ec01
Merge pull request #2123 from facebook/zstd_help
updated `zstd -h`
2020-05-11 18:54:07 -07:00
Yann Collet 37b2dafbed minor clarification on `-o file` 2020-05-08 14:20:47 -07:00
Yann Collet e5366bee16 updated `zstd -h`
to help readability.
Group arguments into categories,
try to give better visibility to more common arguments.
2020-05-08 14:14:46 -07:00
W. Felix Handte 376c26bf56 Use Unused Variable 2020-05-08 16:43:39 -04:00
W. Felix Handte 7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
Bimba Shrestha 5b0a452cac
Adding --long support for --patch-from (#1959)
* adding long support for patch-from

* adding refPrefix to dictionary_decompress

* adding refPrefix to dictionary_loader

* conversion nit

* triggering log mode on chainLog < fileLog and removing old threshold

* adding refPrefix to dictionary_round_trip

* adding docs

* adding enableldm + forceWindow test for dict

* separate patch-from logic into FIO_adjustParamsForPatchFromMode

* moving memLimit adjustment to outside ifdefs (need for decomp)

* removing refPrefix gate on dictionary_round_trip

* rebase on top of dev refPrefix change

* making sure refPrefx + ldm is < 1% of srcSize

* combining notes for patch-from

* moving memlimit logic inside fileio.c

* adding display for optimal parser and long mode trigger

* conversion nit

* fuzzer found heap-overflow fix

* another conversion nit

* moving FIO_adjustMemLimitForPatchFromMode outside ifndef

* making params immutable

* moving memLimit update before createDictBuffer call

* making maxSrcSize unsigned long long

* making dictSize and maxSrcSize params unsigned long long

* error on files larger than 4gb

* extend refPrefix test to include round trip

* conversion to size_t

* making sure ldm is at least 10x better

* removing break

* including zstd_compress_internal and removing redundant macros

* exposing ZSTD_cycleLog()

* using cycleLog instead of chainLog

* add some more docs about user optimizations

* formatting
2020-04-17 15:58:53 -05:00
Nick Terrell ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Bimba Shrestha 43874730f4 Adding --content-size 2020-03-09 14:19:05 -05:00
Bimba Shrestha 6d8e761ba3 Inverting content size prefs bool 2020-03-09 14:16:10 -05:00
Bimba Shrestha 03ee18d9c5 Adding (void)noContentSize 2020-03-09 14:00:52 -05:00