1438 Commits

Author SHA1 Message Date
Yann Collet
09c9cf3f51 simplified rateLimiter
resists better to changing in/out conditions
limits risks of "catching up"
2018-08-13 12:13:47 -07:00
Yann Collet
e11f91b039 remove error message for Ctrl+C 2018-08-13 11:48:25 -07:00
Yann Collet
f3aa510738 rateLimiter does not "catch up" when input speed is slow 2018-08-13 11:38:55 -07:00
Yann Collet
a996b1fd2d fixed rate limited for high speed 2018-08-10 17:39:00 -07:00
Yann Collet
681a382eea added rateLimiter.py, by @felixhandte
this rate limiter avoid the problem of `pv`
which "catch up" after a blocked period
instead of preserving a constant speed cap.
2018-08-10 12:25:52 -07:00
George Lu
3ac2c22485 Reorder declaration 2018-08-09 16:38:32 -07:00
George Lu
bfe8392e23 Remove ctx from benchMem 2018-08-09 12:07:57 -07:00
George Lu
0ece2e5cdc Add consts
+ fix gcc-8 warnings
2018-08-09 11:38:09 -07:00
George Lu
6f480927af argument parsing cleanup
+ clarifying comment
2018-08-09 10:42:58 -07:00
George Lu
ad16a69408 Readability improvements, renaming 2018-08-09 10:42:58 -07:00
George Lu
8278a49cb6 const srcPtrs 2018-08-09 10:42:58 -07:00
George Lu
3d230db853 Change speed representation from floating point to integral 2018-08-09 10:42:58 -07:00
George Lu
8faeb41679 Update Documentation
Change comment // to  /* */
Add more description of what functions do
Remove outdated comments
2018-08-09 10:42:58 -07:00
George Lu
dd270b2f75 Renaming / Style fixes 2018-08-09 10:42:58 -07:00
George Lu
e148db366e Separate capacity vs size
Also:
Make suggested fixes
-varInds_t
-reorder some arguments
-remove code duplication
-update README / -h
-Fix memory leaks
2018-08-09 10:42:58 -07:00
George Lu
df026e159f Fix windows implicit casting bugs 2018-08-09 10:42:58 -07:00
George Lu
0f91b039ff Add Levels 2018-08-09 10:42:58 -07:00
George Lu
7b5b3d7ae3 BenchMem with block compressed sizes passed back up 2018-08-09 10:42:58 -07:00
George Lu
3adc217ea4 Total Changes:
Add different constraint types (decompression speed, compression memory, parameter constraints)
Separate search space by strategy + strategy selection
Memoize results
Real random restarts
Support multiple files
Support Dictionary inputs
Debug Macro for extra printing
2018-08-09 10:42:58 -07:00
George Lu
fab4438801 Dictionary + Multiple file Loading 2018-08-09 10:42:58 -07:00
George Lu
eb21b7f482 Not crashing 2018-08-09 10:42:58 -07:00
George Lu
5f49034520 Working V1 2018-08-09 10:42:58 -07:00
George Lu
cffb6da339 Parses additional parameters
Additional constraint checking

Minor fixes

more param parsing

Add Memory

Change paramVariation

work on feasibility

reformat bench

Changed Paramgrill to use bench.c benchmarking

customlevel macro

Printing Flag

Minor changes

Explicit casting

Makefile fix

casting, type fix

Printing Flag

Minor Changes

comments, helper fn's
2018-08-09 10:42:58 -07:00
Yann Collet
5808027abf Merge branch 'dev' into fix1241 2018-08-03 16:08:33 -07:00
Nick Terrell
b9faaa1dc3 [FSE] Add division by zero test 2018-07-30 13:24:09 -07:00
cyan4973
3f535007e4 fix %zu support under minGW
and relevant test on Appveyor
2018-07-30 16:56:18 +02:00
cyan4973
aade1e5904 Merge branch 'dev' into fix1241 2018-07-30 16:30:35 +02:00
cyan4973
3d4b09a5aa support %zu under mingw 2018-07-30 16:29:20 +02:00
Yann Collet
effa84c8d1
Merge pull request #1230 from terrelln/train-out
zstdcli: Allow -o before --train
2018-07-18 16:34:10 +02:00
Yann Collet
6e490a2f09
Merge pull request #1237 from terrelln/init-cstream-adv
Set requestedParams in ZSTD_initCStream*()
2018-07-18 16:33:30 +02:00
cyan4973
7d1bc9cc8c fix minor conversion warning 2018-07-18 16:10:23 +02:00
cyan4973
9597b438e9 fix #1241
Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).
2018-07-17 18:52:57 +02:00
Nick Terrell
58b8219475 zstdcli: Allow -o before --train
Only set the default value if `outFileName` is unset.

Fixes #1227.
2018-07-16 12:45:34 -07:00
Nick Terrell
45821fac0c
Merge pull request #1225 from jennifermliu/dev
Split samples when building dictionary for COVER
2018-07-13 13:26:15 -07:00
Nick Terrell
6d222c437c Set requestedParams in ZSTD_initCStream*()
The correct parameters are used once, but once `ZSTD_resetCStream()` is
called the default parameters (level 3) are used. Fix this by setting
`requestedParams` in the `ZSTD_initCStream*()` functions.

The added tests both fail before this patch and pass after.
2018-07-12 18:35:55 -07:00
Dmitry V. Levin
7ca12a1455 tests: use /dev/zero instead of /dev/random on all systems except GNU/Hurd
https://github.com/facebook/zstd/pull/1124 broke the test suite
on Linux.  Looks like GNU/Hurd is the only operating system where
/dev/random is available for writing by unprivileged processes.

https://github.com/facebook/zstd/pull/1222 reverted the change
introduced by https://github.com/facebook/zstd/pull/1124 for OpenBSD
only, other operating systems need that change to be reverted, too.

Fixes: 2dde9d5abaad ("Write to /dev/random for test")
Complements: 276988f7948d ("OpenBSD is unable to write to /dev/zero")
2018-07-11 12:41:50 +00:00
Jennifer Liu
a085d1aae1 Allow splitPoint==1.0 (using all samples for both training and testing) 2018-07-05 10:38:45 -07:00
Yann Collet
1bf0784975
Merge pull request #1206 from GeorgeLu97/FullBenchParam
Fullbench Param
2018-07-02 13:33:53 -07:00
Yann Collet
689bfecd48
Merge pull request #1188 from GeorgeLu97/BenchModule
Bench module
2018-07-02 13:33:27 -07:00
Jennifer Liu
1a14f8639c Update COVER dictionary builder tests 2018-07-02 11:37:04 -07:00
Björn Ketelaars
527dbf89ec xz/lzma warning causes test to fail
OpenBSD's port building infrastructure is able to build in a privilege
separated mode. It uses a privilege drop model. Regression tests fail in
this mode as xz/lzma is unable to set file group and errors out with:

xz: tmp.xz: Cannot set the file group: Operation not permitted
gmake[1]: *** [Makefile:307: zstd-playTests] Error 2

Actually it is not a xz/lzma error but a warning causing zstd's
regression test to fail. Proposed fix is to have xz/lzma not set the
exit status to 2 even if a condition worth a warning was detected (-Q
flag).
2018-06-30 13:54:34 +02:00
Björn Ketelaars
276988f794 OpenBSD is unable to write to /dev/zero
https://github.com/facebook/zstd/pull/1124 fixes an issue with GNU/Hurd
being unable to write to /dev/zero. Implemented fix is writing to
/dev/random instead.

On OpenBSD a regular user is unable to write to /dev/random because of
permissions set on this device. Result is failing a regression test.

Proposed solution should work for all platforms.
2018-06-30 13:54:34 +02:00
Yann Collet
328ec7e34c fix test-zstd can be run with parallel compilation
fix #1221
2018-06-29 16:52:21 -07:00
W. Felix Handte
8e7bdc18d6 Fix Tests of --list Behavior with stdin 2018-06-29 16:31:22 -04:00
Yann Collet
a2c3a4cd0e
Merge pull request #1214 from jennifermliu/dev
Make --fast=0 fail
2018-06-27 18:53:39 -07:00
Jennifer Liu
1ab57a7ce1 Redirect failed test result to INTOVOID and update comment about parsing fast command 2018-06-27 16:27:45 -07:00
W. Felix Handte
c10561c752 poolTests.c: Fix Interval Var Type 2018-06-27 19:15:38 -04:00
Jennifer Liu
aef8486fee Make fast=0 fail 2018-06-27 14:27:27 -07:00
George Lu
ceb4b9e670 New fullbench args
-l# gives some level
--zstd= style parameters also supported
2018-06-25 18:43:55 -07:00
Nick Terrell
b426bcc097
[zstdmt] Fix jobsize bugs (#1205)
[zstdmt] Fix jobsize bugs

* `ZSTDMT_serialState_reset()` should use `targetSectionSize`, not `jobSize` when sizing the seqstore.
  Add an assert that checks that we sized the seqstore using the right job size.
* `ZSTDMT_compressionJob()` should check if `rawSeqStore.seq == NULL`.
* `ZSTDMT_initCStream_internal()` should not adjust `mtctx->params.jobSize` (clamping to MIN/MAX is okay).
2018-06-25 15:21:08 -07:00