facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
6,546 Commits 2 Branches 0 Tags
Commit Graph

6546 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Felix Handte
344f942573
Merge pull request #1620 from michaelforney/test-no-threads 
Skip --adapt and --rsyncable tests when built without thread support
 2019-06-11 15:05:25 -04:00
Nick Terrell
81276c4ee4
Merge pull request #1646 from terrelln/oss-fuzz 
[fuzz] Remove max_len from the options
 2019-06-10 22:13:16 -07:00
Yann Collet
c601622aa7
Merge pull request #1619 from j301scott/dev 
CMake: Check for existing custom target 'uninstall'
 2019-06-10 11:56:49 -07:00
Nick Terrell
6810dd6191 [fuzz] Remove max_len from the options 2019-06-10 11:05:45 -07:00
Felix Handte
e79dbb8074
Merge pull request #1642 from Absotively/dev 
VS2010 project settings improvements
 2019-06-10 10:39:25 -04:00
Jen Pollock
176a543dde In VS2010+, turn off assembler output for libzstd & libzstd-dll, and don't export functions from libzstd 2019-06-07 17:21:51 -06:00
Nick Terrell
d06c15cc89
Merge pull request #1635 from terrelln/opt-opt 
[libzstd] Optimize ZSTD_insertBt1() for repetitive data
 2019-06-06 13:15:23 -07:00
Nick Terrell
f3800ba073 [regression] Update results.csv 2019-06-05 20:40:52 -07:00
Nick Terrell
cdb9481e38 [libzstd] Optimize ZSTD_insertBt1() for repetitive data 
We would only skip at most 192 bytes at a time before this diff.
This was added to optimize long matches and skip the middle of the
match. However, it doesn't handle the case of repetitive data.

This patch keeps the optimization, but also handles repetitive data
by taking the max of the two return values.

```
> for n in $(seq 9); do echo strategy=$n; dd status=none if=/dev/zero bs=1024k count=1000 | command time -f %U ./zstd --zstd=strategy=$n >/dev/null; done
strategy=1
0.27
strategy=2
0.23
strategy=3
0.27
strategy=4
0.43
strategy=5
0.56
strategy=6
0.43
strategy=7
0.34
strategy=8
0.34
strategy=9
0.35
```

At level 19 with multithreading the compressed size of `silesia.tar` regresses 300 bytes, and `enwik8` regresses 100 bytes.
In single threaded mode `enwik8` is also within 100 bytes, and I didn't test `silesia.tar`.

Fixes Issue #1634.
 2019-06-05 20:34:00 -07:00
Yann Collet
c0eb6c9c81
Merge pull request #1632 from ephiepark/dev 
CircleCI : use custom Docker Image with pre-installed dependencies
 2019-06-05 12:30:21 -07:00
Felix Handte
015985c1f9
Merge pull request #1633 from yk-tanigawa/patch-1 
Update README.md
 2019-06-05 14:55:19 -04:00
Yosuke Tanigawa
f46a3603c9
Update README.md 
Fix minor typo in the documentation.
 2019-06-05 09:11:21 -07:00
Ephraim Park
286394aa9b Merge branch 'dev' of https://github.com/ephiepark/zstd into dev 2019-06-04 21:14:20 -07:00
Ephraim Park
c96535f67e circleci : remove install dependencies command from config.yml 2019-06-04 21:13:40 -07:00
Ephraim Park
3e9572bfa9 circleci : use custom docker image with installations 2019-06-04 21:13:40 -07:00
Ephraim Park
f1bf281715
Merge pull request #2 from ephiepark/circleci 
CircleCI : use custom Docker Image with pre-installed dependencies
 2019-06-04 20:53:29 -07:00
Yann Collet
c6ca65ab78
Merge pull request #1631 from ephiepark/dev 
zstdcli : expose cpu load indicator for each file on -vv mode
 2019-06-04 15:14:35 -07:00
Yann Collet
61d912f3ff
Merge pull request #1629 from facebook/streamSize 
Added comments on I/O buffer sizes for streaming
 2019-06-04 12:38:43 -07:00
Ephraim Park
a6a7f08eaf circleci : remove install dependencies command from config.yml 2019-06-04 12:18:33 -07:00
Ephraim Park
a371df534d circleci : use custom docker image with installations 2019-06-04 11:05:42 -07:00
Yann Collet
b3af1873a0 better title formatting for html documentation 
must pay attention to /** and /*! patterns.
 2019-06-04 10:35:40 -07:00
Yann Collet
b5c98fbfd0 Added comments on I/O buffer sizes for streaming 
It seems this is still a confusing topic,
as in https://github.com/klauspost/compress/issues/109 .
 2019-06-04 10:26:16 -07:00
Ephraim Park
e498bb64b6 zstdcli : align output message with previous message 2019-06-04 09:42:18 -07:00
Ephraim Park
a38601f05a zstdcli : remove extra semicolon 2019-06-04 09:25:16 -07:00
Ephraim Park
ef2c836955 zstdcli : trying to fix declaration after statement 2019-06-04 09:23:49 -07:00
Ephraim Park
5fe974261b zstdcli : moving cpu load calculation from FIO_compressFilename_dstFile to FIO_compressFilename_internal 2019-06-04 09:04:35 -07:00
Ephraim Park
2841c79e87 zstdcli : fixing mixed declarations and code error 2019-06-03 17:36:09 -07:00
Ephraim Park
43d463eccb zstdcli : moving start time and cpu initialization after potential prompt 2019-06-03 17:31:14 -07:00
Ephraim Park
5029ebce2f zstdcli : exposing cpu load indicator for each file on -vv mode 2019-06-03 14:38:40 -07:00
Yann Collet
b13a9207f9
Merge pull request #1623 from facebook/fullbench 
fullbench minor improvements
 2019-05-31 14:40:19 -07:00
Yann Collet
78056a3e9c consistency : changed constant name to kSampleSizeDefault 
as suggested by @terrelln.

added <assert.h> include
 2019-05-30 10:13:56 -07:00
Yann Collet
ab3625cffd define CONTROL, to ensure check cannot be disabled 
assert() can be disabled with NDEBUG
 2019-05-29 16:18:22 -07:00
Yann Collet
904d4da239 fullbench : minor refactoring, for readability 2019-05-29 16:08:49 -07:00
Yann Collet
ed38b645db fullbench: pass proper parameters in scenario 43 2019-05-29 15:26:06 -07:00
Yann Collet
b84274da0f
Merge pull request #1622 from facebook/nextToUpdate3 
nextToUpdate3 does not need to be maintained outside of zstd_opt.c
 2019-05-28 18:26:21 -07:00
Yann Collet
c63081623f fullbench 43: ensure context is freed after each usage 2019-05-28 17:27:52 -07:00
Yann Collet
eb6b199038 fullbench: added streaming with fresh CCtx scenario 
worst case situation, where context must be recreated every time
but without knowledge of the input size
thus sizing the context for some large input.
 2019-05-28 16:50:49 -07:00
Yann Collet
4c4149452c fullbench can now select sample size with command -B# 2019-05-28 16:37:03 -07:00
Yann Collet
9719fd616c removed nextToUpdate3 from ZSTD_window 
it's now a local variable of ZSTD_compressBlock_opt()
 2019-05-28 16:18:12 -07:00
Yann Collet
33dabc8c80 get bt matches : made it a bit clearer which parameters are input and output 2019-05-28 16:11:32 -07:00
Yann Collet
327cf6fac1 nextToUpdate3 does not need to be maintained outside of zstd_opt.c 
It's re-synchronized with nextToUpdate at beginning of each block.
It only needs to be tracked from within zstd_opt block parser.

Made the logic clear, so that no code tried to maintain this variable.

An even better solution would be to make nextToUpdate3
an internal variable of ZSTD_compressBlock_opt_generic().
That would make it possible to remove it from ZSTD_matchState_t,
thus restricting its visibility to only where it's actually useful.

This would require deeper changes though,
since the matchState is the natural structure to transport parameters into and inside the parser.
 2019-05-28 15:26:52 -07:00
Yann Collet
6453f8158f complementary code comments 
on variables used / impacted during maxDist check
 2019-05-28 14:12:16 -07:00
Yann Collet
4baecdf72a added comments to better understand enforceMaxDist() 2019-05-28 13:15:48 -07:00
Michael Forney
1bc77ee174 Skip --adapt and --rsyncable tests when built without thread support 
These options require ZSTD_MULTITHREAD.
 2019-05-26 03:20:37 -07:00
Jonathan Scott
3f5737afd7 CMake: Check for existing custom target 'uninstall' 2019-05-25 17:30:04 -07:00
Felix Handte
21bb78e908
Merge pull request #1618 from felixhandte/zstgrep-f-flag 
Handle `-f` Flag in `zstgrep`
 2019-05-24 20:59:11 -04:00
W. Felix Handte
6a0638048a Add Test 2019-05-24 17:21:44 -04:00
W. Felix Handte
61025d5b7d zstdgrep: Handle -f Flag 2019-05-24 16:55:59 -04:00
W. Felix Handte
ff0be17cf7 Build Manual 2019-05-24 16:55:43 -04:00
Tyler-Tran
cb47871a0a [dictBuilder] Be more specific than ERROR(generic) (#1616) 
* Specify errors at a finer granularity than `ERROR(generic)`.
* Add tests for bad parameters in the dictionary builder.
 2019-05-22 18:57:50 -07:00