facebook/zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,413 Commits 2 Branches 0 Tags
Commit Graph

123 Commits

Author SHA1 Message Date
Yann Collet
e0065cf660 make test : removed zstreamtest unit tests for variants 
slightly reduced time to create dictionary
at beginning of unit tests
 2017-09-28 18:34:38 -07:00
Yann Collet
47c6a95d07 zstreamtest : run unit tests only during "normal" session 
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
 2017-09-28 18:27:22 -07:00
Yann Collet
e4ec427720 Merge branch 'dev' into shorterTests 
fixed conflicts
 2017-09-28 12:19:28 -07:00
Yann Collet
bbef058ae6 zstreamtest --newapi : reduced maximum allocated memory 2017-09-28 11:48:45 -07:00
Yann Collet
d6770f80af minor : rewrite unit tests using CHECK_Z macro 2017-09-28 02:14:48 -07:00
Yann Collet
9fe50ed623 fixed maximum windowLog for zstreamtest --newapi 
for compatibility with low memory VM
 2017-09-28 01:42:06 -07:00
Yann Collet
9b5b47ac93 ensure adjustCParams adjust hLog and cLog even without srcSize 
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.

Reduces maximum memory allocated during zstreamtest --newapi
 2017-09-28 01:25:40 -07:00
Yann Collet
aa800c4793 reduced memory usage of zstreamtest --newapi 
to run on memory-constrained VM
 2017-09-27 18:00:15 -07:00
Yann Collet
bc32b40b98 reduced zstreamtest --mt memory load 
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
 2017-09-27 17:27:38 -07:00
Yann Collet
54a827fff0 Merge branch 'dev' into newFormats 
Fixed conflicts in zstdmt_compress.c
 2017-09-27 16:39:40 -07:00
Yann Collet
bfabd1d4dc fixed zstreamtest decoding error 
same error (wrong output buffer size) was present
on --mt and --new_api tests.
 2017-09-27 01:01:11 -07:00
Yann Collet
ca306c1c84 fixed a bug in zstreamtest 
decoder output buffer would receive a wrong size.

In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.

This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
 2017-09-27 00:39:41 -07:00
Nick Terrell
c233bdbaee Increase maximum window size 
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
  on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
  power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
  `--zstd=wlog=#`. These options also set the window size during
  decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
  decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
  which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
  window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
  and decompression.
 2017-09-26 14:00:01 -07:00
Yann Collet
96f0cde31a minor function rename 
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
 2017-09-24 16:47:02 -07:00
Yann Collet
f1571dad8f Merge pull request #838 from stellamplau/ldm-mergeDev 
Add long distance matcher
 2017-09-13 13:24:08 -07:00
Stella Lau
3d8e313f64 Reduce ldm hash table size in test 2017-09-11 17:21:28 -07:00
Stella Lau
eb3327c10a Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev 2017-09-11 15:00:01 -07:00
Yann Collet
b3f33ccfb3 use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream() 
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
 2017-09-09 14:37:28 -07:00
Yann Collet
3128e03be6 updated license header 
to clarify dual-license meaning as "or"
 2017-09-08 00:09:23 -07:00
Stella Lau
9e4060200b Add tests and fix pointer alignment 2017-09-06 09:14:05 -07:00
Stella Lau
67d4a6161c Add ldmBucketSizeLog param 2017-09-02 21:55:29 -07:00
Stella Lau
a1f04d518d Move hashEveryLog to cctxParams and update cli 2017-09-01 15:05:47 -07:00
Stella Lau
767a0b3be1 Move ldm hashLog, bucketLog, and mml to cctxParams 2017-09-01 12:24:59 -07:00
Stella Lau
17d8e0bdcc Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate 2017-09-01 10:19:38 -07:00
Stella Lau
8081becadc Add long distance matching as a CCtxParam 2017-09-01 09:18:58 -07:00
Yann Collet
d7ad99b2ab Merge branch 'longRangeMatcher' into dev 2017-08-31 18:08:37 -07:00
Stella Lau
6a546efb8c Add long distance matcher 
Move last literals section to ZSTD_block_internal
 2017-08-31 12:53:19 -07:00
Stella Lau
90a31bfa16 Pass dictMode to ZSTDMT_initCStream; fix nits 
- Return error code in estimate{CCtx,CStream}Size functions
 2017-08-30 16:19:07 -07:00
Stella Lau
82d636b76a Rename applyCCtxParams() 2017-08-29 18:03:06 -07:00
Stella Lau
c88fb9267f Replace 'byReference' with enum 2017-08-29 11:55:02 -07:00
Stella Lau
b5b9275e67 Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced() 2017-08-29 10:49:29 -07:00
Stella Lau
eb7bbab36a Remove ZSTD_p_refDictContent and dictContentByRef 2017-08-25 11:11:45 -07:00
Stella Lau
6f1a21c7e9 Remove formatting-only changes 2017-08-23 10:24:19 -07:00
Stella Lau
73c73bf16a Reduce code duplication in zstreamtest 2017-08-21 12:41:19 -07:00
Stella Lau
91b30dbe84 Remove test parameter 2017-08-21 10:09:06 -07:00
Stella Lau
f181f33bdf Disable tests and refactor 2017-08-21 01:59:08 -07:00
Stella Lau
023b24e6d4 Add cctx param tests 2017-08-20 22:55:07 -07:00
Yann Collet
32fb407c9d updated a bunch of headers 
for the new license
 2017-08-18 16:52:05 -07:00
Yann Collet
052a95f77c fix : ZSTDMT_compress_advanced() correctly generates checksum 
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
 2017-07-11 17:18:26 -07:00
Yann Collet
4616fad18b improved ZSTDMT_compress() memory usage 
does not need the input buffer for streaming operations

also : reduced a few tests time length
 2017-07-10 17:16:41 -07:00
Yann Collet
afb0aca739 zstreamtest : big tests are only enabled in 64-bits mode 
to avoid requesting too much memory in 32-bits mode during MT tests
 2017-06-29 18:19:09 -07:00
Yann Collet
62f7efc04a added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job 2017-06-28 16:25:13 -07:00
Yann Collet
33a6639039 fixed ZSTD_refPrefix with Multithread-enabled CCtx 2017-06-28 11:09:43 -07:00
Yann Collet
1fd2df9c2c fixed zstreamtest 
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
 2017-06-28 09:57:59 -07:00
Yann Collet
2e4274262d controlled dictMode 2017-06-27 17:09:12 -07:00
Yann Collet
b7372933b8 implemented ZSTD_refPrefix() 2017-06-27 15:49:12 -07:00
Yann Collet
fecc721fd9 added parameter ZSTD_p_refDictContent 2017-06-27 11:46:39 -07:00
Yann Collet
dde10b23fe refactored ZSTD_estimateDStreamSize() 
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
 2017-06-26 17:44:26 -07:00
Yann Collet
09ae03a570 ZSTD_estimateCDictSize_advanced() 
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
 2017-06-26 16:47:32 -07:00
Yann Collet
0c9a915a28 ZSTD_estimateCStreamSize_advanced() 2017-06-26 16:02:25 -07:00