Commit Graph

443 Commits (23c1a2d260e6b50279fbb25d5e13e52516c6d19b)

Author SHA1 Message Date
stanjo74 52598d54e9
Limit train samples (#2809)
* Limit training samples size to 2GB

* simplified DISPLAYLEVEL() macro to use global vqriable instead of local.

* refactored training samples loading

* fixed compiler warning

* addressed comments from the pull request

* addressed @terrelln comments

* missed some fixes

* fixed type mismatch

* Fixed bug passing estimated number of samples rather insted of the loaded number of samples.
Changed unit conversion not to use bit-shifts.

* fixed a declaration after code

* fixed type conversion compile errors

* fixed more type castting

* fixed more type mismatching

* changed sizes type to size_t

* move type casting

* more type cast fixes
2021-10-04 17:47:52 -07:00
senhuang42 b5c35d7ea3 Use new paramSwitch enum for LCM, row matchfinder, and block splitter 2021-09-21 14:22:02 -04:00
makise-homura a5f518ae27 Change zstdcli's main() declaration due to -Wmain on some compilers 2021-07-14 19:55:47 +03:00
makise-homura d4ad02c721 Add support for MCST LCC compiler 2021-07-10 03:57:06 +03:00
Binh Vo 6a46e38deb Add option to use logical cores for default threads 2021-06-16 15:46:17 -04:00
binhdvo 89127e5ee2
Merge pull request #2705 from binhdvo/bootcamp
Add support for negative values in advanced flags
2021-06-11 14:08:23 -04:00
Binh Vo 6fad35c6a1 Add support for negative levels in --adapt=min and --adapt=max" 2021-06-11 12:13:09 -04:00
W. Felix Handte 464bfb022e In Verbose Mode, Preserve Full Precision Where Possible 2021-06-10 12:53:07 -04:00
binhdvo 325952f878
Revert "Add support for --long-param flag, fix #2104" 2021-06-09 15:35:43 -04:00
Binh Vo 6583fa3f0a Add support for --long-param flag 2021-06-09 14:07:52 -04:00
Binh Vo 1e17184ad0 Add documentation for --patch-from 2021-06-03 11:12:27 -04:00
sen 6030cdfede
Add --progress flag (#2595) 2021-05-06 14:50:28 -04:00
W. Felix Handte 33f3e293e8 Allow Reading from Block Devices with `--force` 2021-05-04 16:25:26 -04:00
Nick Terrell 4694423c4f Add and integrate lazy row hash strategy 2021-04-07 09:53:34 -07:00
Nick Terrell a494308ae9 [copyright][license] Switch to yearless copyright and some cleanup in the linux-kernel files
* Switch to yearless copyright per FB policy
* Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources
* Add zstd copyright/license header to the `contrib/linux-kernel` sources
* Update the `tests/test-license.py` to check for yearless copyright
* Improvements to `tests/test-license.py`
* Check `contrib/linux-kernel` in `tests/test-license.py`
2021-03-30 10:30:43 -07:00
senhuang42 444c4650a0 Add newline to end of cli help message 2021-02-17 12:30:42 -05:00
Nick Terrell 54a4998a80 Add basic tracing functionality 2021-02-05 16:28:52 -08:00
W. Felix Handte 8b6a4b5b7c Allow Input From Console When `--force` is Passed
Also update option flag documentation.
2021-01-11 17:53:20 -05:00
Yann Collet 3324e87cff Added library version check 2021-01-07 10:37:27 -08:00
Nick Terrell 66e811d782 [license] Update year to 2021 2021-01-04 17:53:52 -05:00
senhuang42 7259b258d1 Add callsites to zstdcli.c and tests to playTests.sh 2020-10-07 13:47:38 -04:00
Yann Collet f7d4943788
Merge pull request #2330 from senhuang42/fix_stdinout_error_messages
Improve error messages on console input/output
2020-10-06 15:58:37 -07:00
senhuang42 ce56810a32 Modify error messages on console input/output 2020-09-28 12:15:18 -04:00
senhuang42 432186cbea Add FIO_determineHasStdinInput() function and member to fCtx 2020-09-24 15:55:30 -04:00
Yann Collet 05622992d5
Merge pull request #2292 from senhuang42/multifile_status_update_fio_refactor
Make multifile (de)compression print out a summary
2020-09-14 14:06:26 -07:00
Yann Collet e583e0be8c
Merge pull request #2299 from senhuang42/env_var_num_threads
Allow environment variable to specify number of threads for compression
2020-09-14 14:04:19 -07:00
senhuang42 a71963c7b8 nbThreads instead of numThreads 2020-09-09 12:40:00 -04:00
senhuang42 243c8dfb41 Add include guards for init_numThreads() 2020-09-08 09:26:16 -04:00
senhuang42 136a0673f6 Add quick documentation for ZSTD_NUMTHREADS in the code 2020-09-07 18:19:31 -04:00
senhuang42 888c385a49 Change param name to ZSTDCLI_NUMTHREADS_DEFAULT 2020-09-07 18:11:42 -04:00
senhuang42 972e063219 Change default num_threads value 2020-09-07 18:09:48 -04:00
senhuang42 5123496104 Adjust function signatures 2020-09-07 13:13:05 -04:00
senhuang42 48bca10792 Address memory leak in CLI from fCtx 2020-09-03 10:14:04 -04:00
senhuang42 3a7d625d6b Cleanup comments, add function to set FIO_ctx_t.nbFilesProcessed 2020-09-01 12:54:21 -04:00
senhuang42 a6414f1247 Integrate refactor into status print for multifiles, adjust logic for printing as needed 2020-09-01 12:34:43 -04:00
senhuang42 565f116a56 Change name to ZSTD_NUMTHREADS 2020-09-01 09:02:23 -04:00
senhuang42 d54566f334 First pass at refactoring and creating new FIO_ctx_t */ 2020-08-28 11:01:04 -04:00
Yann Collet a8c66881e5
Merge pull request #2283 from senhuang42/progress_bars_for_multiple_files
Refreshing progress bar for processing multiple files
2020-08-26 11:54:50 -07:00
senhuang42 ead2387a7e Remove extraneous FIO_setNbFiles() 2020-08-26 12:05:04 -04:00
senhuang42 b6abbc3ce5 Fix single file decompression summary, remove escape code dependency, add currFileIdx field to FIO_prefs, general cleanups/bugfixes 2020-08-26 11:35:07 -04:00
senhuang42 70fa7b37ca Added env var function for threads 2020-08-25 18:49:52 -04:00
senhuang42 da38891a87 Add initial live update with displayLevel = 2, add new field to FIO_prefs_t 2020-08-25 16:46:47 -04:00
senhuang42 e1e41addb3 Adjust #ifndef logic 2020-08-24 17:36:36 -04:00
senhuang42 44c54a3e31 Addressing comments: more comments, cleanup, remove extra function, checksum logic 2020-08-24 16:14:19 -04:00
senhuang42 ffaa0df76d Document change in CLI for --no-check during decompression in --help menu 2020-08-24 09:49:12 -04:00
Yann Collet 15c5e20023 updated documentation 2020-07-23 12:10:57 -07:00
Yann Collet 610c41f90f improved 2-args macro to support both syntaxes
both `--long-commmand=arg` and `--long-command arg`
are supported by macro `NEXT_FIELD()`
so that the long-command only needs to be listed once.

This extends the syntax to support new syntaxes like
`-o=FILE` or `-D=dict`,
though there is no need to advertise this capability for the time being.

Also : added `NEXT_UINT32()`,
which is wrapper around `NEXT_FIELD()`
to read integer parameters.
Use the wrapper to new field, such as `--memlimit`
which can now support both syntaxes too.
2020-07-20 17:41:32 -07:00
Yann Collet 23941eec04 added tests for newly enabled syntax
for --patch-from origin
and --filelist list

Also : removed some constrained syntax tests,
as the new argument parsing syntax is more permissive.

For example :
    zstd file -of dest
used to be disallowed.

It's now allowed, and understood as:
    zstd file -o dest -f
2020-07-17 13:31:15 -07:00
Yann Collet 7280c1dafa extended 2-fields macro usage to number arguments
leading to further simplications
2020-07-17 13:09:23 -07:00
Yann Collet a7e5652672 CLI: simplified handling of separated-fields arguments
this patch reduces complexity associated with
commands requiring a separated arguments
such as :
-o filename
-D dictionary
--output-dir-flat dir
--output-dir-mirror dir

It used to be a multi-stage process with explicit context,
it's now simplified as a single macro.

Thanks to this simplification,
separated arguments logic has also been extended to
--patch-from XXX
--filelist XXX
which extends existing capability using =XXX
--patch-from=XXX
--filelist=XXX

Separated argument is useful for filenames and directories,
as it benefits from shell expansion
such as ~/dir/file
where the ~ is automatically translated by the shell.
In contrast --long-command=FILE does not interpret FILE,
so ~/ is transmitted as is to the main() function,
generally resulting in incorrect file name.
2020-07-17 12:46:36 -07:00