Commit Graph

564 Commits (7238cca1a1f00b27570b721b852317dfcfbffa72)

Author SHA1 Message Date
W. Felix Handte 7238cca1a1 Deduplicate Some Low-Hanging Fruit of Redundant Stat Calls 2020-08-05 01:08:34 -04:00
W. Felix Handte b6e24bc4dc Rename UTIL_getFileStat() -> UTIL_statFile() and UTIL_getDirectoryStat() -> UTIL_statDir()
I want to introduce versions of many of these functions that take pre-
populated `stat_t` objects and use those rather than doing their own redundant
`stat()` internally. These functions will have `...Stat()` suffixes. So this
commit renames these existing functions into the active voice, to avoid
confusion.
2020-08-05 00:40:16 -04:00
Xin Xie 9a8ccd4ba3 Add output-dir-mirror option 2020-06-24 22:12:11 -07:00
Bimba Shrestha b06710889f error when not using --stream-size on pipe 2020-06-18 09:31:06 -07:00
Bimba Shrestha 6653321ba3 passing streamSrcSize when set 2020-06-18 09:28:18 -07:00
Bimba Shrestha e2838d9eb9 Spelling mistakes 2020-06-05 05:11:26 -05:00
Bimba Shrestha f847909cf2 Using ZSTD_MAX_WINDOWLOG instead of INT_MAX 2020-05-26 09:23:26 -07:00
Bimba Shrestha 01b38ef162 Change --patch-from limit from 4gb to 2gb 2020-05-26 06:52:30 -07:00
W. Felix Handte 7dcca6bc64 Also Move programs/ Directory to Relative Includes 2020-05-04 15:20:26 -04:00
Bimba Shrestha 659ff85cf5 changing to cParams.hashLog 2020-04-21 21:12:50 -05:00
Bimba Shrestha b640802ff8 updating chainlog bound 2020-04-20 11:34:24 -07:00
Bimba Shrestha 587a20a12a adding newline 2020-04-20 10:25:58 -07:00
Bimba Shrestha 5b0a452cac
Adding --long support for --patch-from (#1959)
* adding long support for patch-from

* adding refPrefix to dictionary_decompress

* adding refPrefix to dictionary_loader

* conversion nit

* triggering log mode on chainLog < fileLog and removing old threshold

* adding refPrefix to dictionary_round_trip

* adding docs

* adding enableldm + forceWindow test for dict

* separate patch-from logic into FIO_adjustParamsForPatchFromMode

* moving memLimit adjustment to outside ifdefs (need for decomp)

* removing refPrefix gate on dictionary_round_trip

* rebase on top of dev refPrefix change

* making sure refPrefx + ldm is < 1% of srcSize

* combining notes for patch-from

* moving memlimit logic inside fileio.c

* adding display for optimal parser and long mode trigger

* conversion nit

* fuzzer found heap-overflow fix

* another conversion nit

* moving FIO_adjustMemLimitForPatchFromMode outside ifndef

* making params immutable

* moving memLimit update before createDictBuffer call

* making maxSrcSize unsigned long long

* making dictSize and maxSrcSize params unsigned long long

* error on files larger than 4gb

* extend refPrefix test to include round trip

* conversion to size_t

* making sure ldm is at least 10x better

* removing break

* including zstd_compress_internal and removing redundant macros

* exposing ZSTD_cycleLog()

* using cycleLog instead of chainLog

* add some more docs about user optimizations

* formatting
2020-04-17 15:58:53 -05:00
Yann Collet 458a1a1723 minor refactor
- fix a few comments
- reorder some parameters, to enforce "mutable references first"
- simplified fwriteSparse()
2020-04-13 14:09:57 -07:00
Bimba Shrestha d0412f3aba no percentage on readsize == 0 2020-04-03 12:10:02 -07:00
Bimba Shrestha bf1856c26f removing max(1, ..) 2020-04-03 12:07:17 -07:00
Nick Terrell ac58c8d720 Fix copyright and license lines
* All copyright lines now have -2020 instead of -present
* All copyright lines include "Facebook, Inc"
* All licenses are now standardized

The copyright in `threading.{h,c}` is not changed because it comes from
zstdmt.

The copyright and license of `divsufsort.{h,c}` is not changed.
2020-03-26 17:02:06 -07:00
Bimba Shrestha 9388dac2c0 Setting ZSTD_c_contentSizeFlag instead of UNKNOWN filesize 2020-03-09 15:40:18 -05:00
Bimba Shrestha 6d8e761ba3 Inverting content size prefs bool 2020-03-09 14:16:10 -05:00
Bimba Shrestha 167244a64f --no-content-size cli 2020-03-09 13:07:29 -05:00
Bimba Shrestha f25a6e9f8f Adding new cli endpoint --patch-from= (#1940)
* Adding new cli endpoint --diff-from=

* Appveyor conversion nit

* Using bool set trick instead of direct set

* Removing --diff-from and only leaving --diff-from=#

* Throwing error when both dictFileName vars are set

* Clean up syntax

* Renaming diff-from to patch-from

* Revering comma separated syntax clean up

* Updating playtests with patch-from

* Uncommenting accidentally commented

* Updating remaining docs and var names to be patch-from instead of diff-from

* Constifying

* Using existing log2 function and removing newly created one

* Argument order (moving prefs to end)

* Using comma separated syntax

* Moving to outside #ifndef
2020-01-10 14:25:24 -08:00
Peter Lesslie 8eb499d354 Check for fread failure
On failure fread may return either a short read or 0. Need to use
ferror to detect error versus eof.
2019-11-26 20:47:58 -06:00
Yann Collet 9a22140ef4 created UTIL_chmod()
protecting "/dev/null" from having its permissions changed.

also : minor : improved consistency of util.h API
2019-11-25 13:45:22 -08:00
Yann Collet 7aaac3f69c fix #1904
/dev/null permissions were modified when using sudo rights.
This fixes this bug during decompression.

More importantly, this patch  adds a test, triggered in TravisCI,
ensuring unaltered /dev/null permissions.
2019-11-25 10:35:36 -08:00
Yann Collet 47034cd6c3
Merge pull request #1850 from bimbashrestha/named_pipes
Gating named pipe support on windows
2019-10-30 12:18:58 -07:00
Shashank Tavildar 9ab6a747d4 Created utility function to extract extension from filename, fixed tests 2019-10-29 12:27:54 -07:00
Bimba Shrestha da73b11b14
Merge branch 'dev' into named_pipes 2019-10-29 01:17:13 -04:00
Shashank Tavildar 0f2bff2faf Addressing comments, removing cyclic dependency with header file, updating tests 2019-10-28 18:21:47 -07:00
Shashank Tavildar 02433e0b15 Addressing comments: -Created a list of extensions defined in fileio.h, -Updated test 2019-10-28 14:54:54 -07:00
Shashank Tavildar 48f856640e Added --exclude-compressed flag feature that skips compression of precompressed files 2019-10-28 13:57:40 -07:00
Bimba Shrestha 5e59d51991 Merge branch 'master' of https://github.com/bimbashrestha/zstd into named_pipes 2019-10-25 15:43:41 -07:00
Bimba Shrestha 0b52d878b2 Cleaning up gate and adding comment to flag 2019-10-25 14:06:50 -07:00
Bimba Shrestha 5b45ff4f1c Gating named file support on windows 2019-10-25 11:32:38 -07:00
W. Felix Handte 4eccc82875 Minor Fixes 2019-10-25 13:59:19 -04:00
W. Felix Handte 91c3f545cc Fix Build; Refactor 2019-10-24 20:18:57 -04:00
Felix Handte 506e1a1176
Merge branch 'tzst-ext-tmp' into tzst-ext 2019-10-24 17:49:34 -04:00
Yann Collet 37f90930da
Merge pull request #1839 from bimbashrestha/named_pipes
Allowing named pipes to go through in zstdcli
2019-10-24 10:53:12 -07:00
Bimba Shrestha 8a3974807e Allowing named pipes to go through zstdcli 2019-10-22 15:23:22 -07:00
Nick Terrell b1ec94e63c Fix ZSTD_f_zstd1_magicless for small data
* Fix `ZSTD_FRAMEHEADERSIZE_PREFIX` and `ZSTD_FRAMEHEADERSIZE_MIN` to
  take a `format` parameter, so it is impossible to get the wrong size.
* Fix the places that called `ZSTD_FRAMEHEADERSIZE_PREFIX` without
  taking the format into account, which is now impossible by design.
* Call `ZSTD_frameHeaderSize_internal()` with `dctx->format`.
* The added tests catch both bugs in `ZSTD_decompressFrame()`.

Fixes #1813.
2019-10-21 21:16:17 -07:00
Yann Collet 632e07747c minor refactor of FIO_compressGzFrame()
mostly narrowing variables lifetime and mutability.
2019-10-21 12:14:59 -07:00
Yann Collet ad86a5d0bc rewrite FIO_createFilename_fromOutDir() 2019-10-18 11:15:10 -07:00
Yann Collet caf40d0ae4 fix : no output file opened in test mode
also : redistributed error code within fileio.c
for more precise diagnosis.
2019-10-17 16:58:49 -07:00
Yann Collet 0a24d4ef18 removed regular file test on Windows
since it does not work well on this platform
(tested with MinGW).

Note : could be an issue within UTIL_isRegularFile()
2019-10-17 16:39:47 -07:00
Yann Collet 0ee360982d improved test mode `-t`
The test mode do no longer open a file `/dev/null` nor write anything to output.

This is supposed to be more efficient than writing to `/dev/null`,
and more universal too : the previous method was failing on Windows.
2019-10-17 16:09:53 -07:00
Yann Collet 1795133c45 refactored FIO_compressMultipleFilenames() prototype
for consistency
2019-10-17 15:32:03 -07:00
Yann Collet 8b6d96827c
Merge pull request #1731 from chungy/stdin_nochmod
[programs] don’t do chmod when coming from stdin or multiple files
2019-10-14 09:34:56 -07:00
Yann Collet b3e92ad346
Merge pull request #1812 from senhuang42/output-dir-flat
Support for --output-dir-flat: output into a directory
2019-10-14 09:25:05 -07:00
Eric van Gyzen 46ee10dfb5 Include errno in all fwrite error messages in fileio.c 2019-10-10 09:21:29 -05:00
Sen Huang c5ebb37051 Only check for filename collisions when using --output-dir-flat 2019-10-09 09:39:52 -04:00
Sen Huang 6e406b55ee Casting to void* to avoid C4090 warnings 2019-10-08 09:54:59 -04:00