150354c5fe
minor refactor
...
added some traces and assert
related to hunting a potential ubsan error in 32-bits more
(it ends up being a compiler-side issue : https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82802 ).
Modified one pointer arithmetic expression for a more conformant way.
2017-11-01 16:57:48 -07:00
e19b0822bc
Test large skippable frames
2017-11-01 13:10:03 -07:00
1ff8a8c109
Merge pull request #891 from facebook/contentSize
...
Content size
2017-10-17 17:24:51 -07:00
e600b5d0da
[test] Exercise all codes in dictionary tables
2017-10-16 18:05:36 -07:00
beb9b4b398
fixed ZSTDMT_initCStream() when contentSizeFlag==1 by default
...
and a wrong test in zstreamtest --mt
2017-10-13 19:09:30 -07:00
213ef3b510
fixed ZSTD_initCStream_advanced() behavior, which depends on contentSizeFlag,
...
and a stream fuzzer test, which was incorrect
(relied on 0 being unconditionnally transformed into `ZSTD_CONTENTSIZE_UNKNOWN`)
2017-10-13 19:01:58 -07:00
3c1e3f8ec9
contentSizeFlag enabled by default would also fail for streaming and MT operations
...
fixed
2017-10-13 18:32:06 -07:00
e0065cf660
make test : removed zstreamtest unit tests for variants
...
slightly reduced time to create dictionary
at beginning of unit tests
2017-09-28 18:34:38 -07:00
47c6a95d07
zstreamtest : run unit tests only during "normal" session
...
not during --mt, --newapi and --opaque
this avoids running them 4x during `make test`
2017-09-28 18:27:22 -07:00
e4ec427720
Merge branch 'dev' into shorterTests
...
fixed conflicts
2017-09-28 12:19:28 -07:00
bbef058ae6
zstreamtest --newapi : reduced maximum allocated memory
2017-09-28 11:48:45 -07:00
d6770f80af
minor : rewrite unit tests using CHECK_Z macro
2017-09-28 02:14:48 -07:00
9fe50ed623
fixed maximum windowLog for zstreamtest --newapi
...
for compatibility with low memory VM
2017-09-28 01:42:06 -07:00
9b5b47ac93
ensure adjustCParams adjust hLog and cLog even without srcSize
...
It would previously exit when srcSize is unknown.
But in the case of custom parameters,
hLog and cLog can still be too large in comparison with windowLog.
Reduces maximum memory allocated during zstreamtest --newapi
2017-09-28 01:25:40 -07:00
aa800c4793
reduced memory usage of zstreamtest --newapi
...
to run on memory-constrained VM
2017-09-27 18:00:15 -07:00
bc32b40b98
reduced zstreamtest --mt memory load
...
adjust compression level, hence memory usage, depending on nb threads
in order to run correctly on memory-starved VM.
2017-09-27 17:27:38 -07:00
54a827fff0
Merge branch 'dev' into newFormats
...
Fixed conflicts in zstdmt_compress.c
2017-09-27 16:39:40 -07:00
bfabd1d4dc
fixed zstreamtest decoding error
...
same error (wrong output buffer size) was present
on --mt and --new_api tests.
2017-09-27 01:01:11 -07:00
ca306c1c84
fixed a bug in zstreamtest
...
decoder output buffer would receive a wrong size.
In previous version, ZSTD_decompressStream() would blindly trust the caller that pos <= size.
In this version, this condition is actively checked,
and the function returns an error code if this condition is not respected.
This check could also be done with an assert(),
but since this is a user-facing interface, it seems better to keep this check at runtime.
2017-09-27 00:39:41 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
3d8e313f64
Reduce ldm hash table size in test
2017-09-11 17:21:28 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
b3f33ccfb3
use ZSTD_decodingBufferSize_min() inside ZSTD_decompressStream()
...
Use same definition as public one
minor : reduce allocated buffer size in some cases
(when frameContentSize is known and == windowSize)
2017-09-09 14:37:28 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
b5b9275e67
Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()
2017-08-29 10:49:29 -07:00
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
afb0aca739
zstreamtest : big tests are only enabled in 64-bits mode
...
to avoid requesting too much memory in 32-bits mode during MT tests
2017-06-29 18:19:09 -07:00
62f7efc04a
added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job
2017-06-28 16:25:13 -07:00
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
1fd2df9c2c
fixed zstreamtest
...
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
2017-06-28 09:57:59 -07:00
2e4274262d
controlled dictMode
2017-06-27 17:09:12 -07:00
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
cca1ed14db
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-06-23 00:13:03 -07:00
32f546c763
zstreamtest : disabled multi-threading tests with --newapi --no-big-tests
...
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.
In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.
This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
2017-06-23 00:09:02 -07:00
f99c2c1a22
reduced --no-big-tests even more
...
to pass tests on qemu-aarch64
2017-06-21 23:35:58 -07:00
49f8459d11
fixed minor cast warning
2017-06-21 18:43:39 -07:00
4d3bdcf130
reduced CLevelMax for --no-big-tests
2017-06-21 18:20:02 -07:00
bfc2f00080
--no-big-tests for zstreamtest
...
Apply --no-big-tests for tsan tests
2017-06-21 17:57:14 -07:00
dce789281b
fixed : decompression of skippable frames in streaming mode
2017-06-21 15:53:42 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
688952062b
minor declaration statement warning fix
2017-06-19 11:53:42 -07:00
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
ed1d039127
newapi fuzzer tests : random flush orders in main loop
2017-06-19 11:07:33 -07:00
f5deae8a67
new api fuzzer tests : frame parameters are randonly set
2017-06-18 23:41:38 -07:00
ea3630a889
new api fuzzer set : dictionary is randomly set
2017-06-18 23:31:55 -07:00
8dee0ec99e
new api fuzzer tests : compression parameters are randomly set
2017-06-18 23:25:15 -07:00
01743a36e7
fuzzer tests for new API
2017-06-16 17:56:41 -07:00
8c910d2097
updated ZSTDMT streaming API
...
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
009d604e00
ZSTD_compress_generic() supports multiple successive frames
...
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
a5ffe3d370
pushed enum values for strategy by one (ZSTD_fast==1)
...
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
30ab64e21d
added test for ZSTD_estimateCStreamSize()
2017-05-10 11:30:19 -07:00
f16f4497ca
added ZSTD_estimateDStreamSize()
2017-05-09 16:18:17 -07:00
0be6fd3429
merged CCtx and CStream as a single same object
...
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
a92cbb7004
Added a secondary test, checking dictID presence after setting noDictIdFLag=1
2017-04-27 15:08:56 -07:00
7d283cdfa2
added test for ZSTD_initCStream_usingCDict_advanced()
2017-04-27 14:48:34 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
2c5514c759
fixed ZSTDMT_initCStream_advanced()
...
Must use the new ZSTD_compressBegin_usingCDict_advanced()
to enforce correct frame parameters
2017-04-18 22:52:41 -07:00
0bb381dad8
added test for ZSTD_initCStream_advanced()
...
when params.fParams.contentSizeFlags = 1
and pledgedSrcSize = 0
then srcSize should be considered 0 (empty), and not "unknown"
2017-04-18 15:08:52 -07:00
4ee6b15dac
force contentSizeFlag=0 when using ZSTD_initCStream_usingCDict()
...
because by definition srcSize is not known when using this prototype.
added relevant test
Note : this use was already working, because at a later stage
(both ZSTD_compressBegin_usingCDict() and ZSTD_copyCCtx())
pledgedSrcSize=0 is translated into "unknown", no matter the frame parameter.
This is not correct, but of little importance,
as the medium term plan is to no longer set fParams within CDict
2017-04-11 11:59:44 -07:00
4b987ad8ce
Introduce ZSTD_initCStream_internal()
...
This is now the regroup point for ZSTD_initCStream*() functions
ZSTD_initCStream_advanced() now properly checks for parameters validity.
Also : added <assert.h> usage inside zstd_compress.c
Needs ZSTD_DEBUG=1 macro to be triggered.
Will be triggered by default from `tests` directory
2017-04-10 17:50:44 -07:00
ce80098f14
improved zstreamtest --mt to trap bug #644
2017-04-05 16:34:09 -07:00
62ecad3819
Fix ZSTD_initCStream_usingCDict() to use dictionary
2017-04-03 21:05:59 -07:00
30c7698970
optimize ZSTDMT_compress() memory usage
...
does no longer allocate temporary buffers
when there is enough room in dstBuffer to decompress directly there.
(previous method would skip that for 1st chunk only).
Also : fix ZSTD_compressBound() for small srcSize
2017-03-31 18:27:03 -07:00
042ba122ae
Change g_displayLevel to int and fix DISPLAYUPDATE flush
2017-03-23 11:21:59 -07:00
7ebf2de02d
Add ability to strongly limit fuzzer test size with flag
2017-03-20 11:25:00 -07:00
f5e50519e0
Prevent fuzz testers from combining large dicts with high clevel
2017-03-15 15:04:54 -07:00
a33ae64204
fixed decoding skippable frames
2017-02-28 01:15:28 -08:00
684858e7b7
fix memory leaks
2017-02-21 18:17:24 +01:00
0ed3901b05
Update overlength match test case
2017-02-16 13:36:57 -08:00
887eaa9e21
Fix wildcopy overwriting data still in window
2017-02-15 16:43:45 -08:00
2db7249265
Make pledgedSrcSize meaning clear for other functions
...
- Added tests
- Moved new size functions to static link only
2017-02-09 11:49:58 -08:00
4e709712e1
Decompressed size functions now handle multiframes and distinguish cases
...
- Add ZSTD_findDecompressedSize
- Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
- Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
48bed91606
Merge pull request #527 from facebook/zstdmt
...
zstdmt refinements
2017-01-31 16:36:46 -08:00
92c98a5b21
zstreamtest uses random section sizes for fuzzing
2017-01-30 12:50:31 -08:00
cd23dd24af
zstreamtest uses random overlapLog for fuzzing
2017-01-30 12:46:35 -08:00
64bf8ffce6
report @terrelln patch to ST fuzzer tests
2017-01-27 17:25:07 -08:00
d98bf49224
Fix segfault in zstreamtest MT
...
It was reading beyond the end of the input buffer because no errors were
detected. Once that was fixed, it wasn't making forward progress because
no errors were detected and it was waiting for input.
2017-01-27 15:42:36 -08:00
8dafb1acf5
CLI : automatically set overlap size to max (windowSize) for max compression level
2017-01-25 17:01:13 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
d7e3cb58c5
Resolved merge conflict dev+zstdmt
2017-01-20 16:44:50 -08:00
19d670ba9d
Added ZSTDMT_initCStream_advanced() variant
...
Correctly compress with custom params and dictionary
Added relevant fuzzer test in zstreamtest
Also :
new macro ZSTDMT_SECTION_LOGSIZE_MIN, which sets a minimum size for a full job
(note : a flush() command can still generate a partial job anytime)
2017-01-19 15:32:07 -08:00
f22adae984
fixed minor warning (unused variable) in fuzzer
2017-01-19 13:46:30 -08:00
736788f8e8
added streaming fuzzer tests for MT API
...
Also : fixed corner case, where nb of jobs completed becomes > jobQueueSize
which is possible when many flushes are issued
while there is not enough dst buffer to flush completed ones.
2017-01-19 12:15:29 -08:00
33fce03045
added test checking dictID when using ZSTD_initCStream_usingCDict()
...
It shows that dictID is not properly added into frame header
2017-01-16 19:46:22 -08:00
d564faa3c6
fix : ZSTD_initCStream_srcSize() correctly set srcSize in frame header
2016-12-18 21:39:15 +01:00
e795c8a5f6
Added ZSTD_initCStream_srcSize().
...
Added relevant test cases in zstreamtest
2016-12-13 17:00:14 +01:00
9ffbeea875
API : changed : streaming decompression : implicit reset on starting new frames
2016-12-02 18:37:38 -08:00
4c0b44f8ce
minor display improvement in fuzzer when nb of tests is very large
2016-11-01 11:13:22 -07:00
335ad5d4d4
added ZSTD_initDStream_usingDDict() .
...
slightly optimized ZSTD_initDStream() when no dictionary .
fixed ZSTD_sizeof_CStream() .
2016-10-25 17:47:02 -07:00
9516234e67
first sketch for ZSTD_initCStream_usingCDict()
2016-10-25 16:19:52 -07:00
cf409a7e2a
fixed : init*_advanced() followed by reset() with different pledgedSrcSiz
2016-09-26 16:41:05 +02:00
2f2639438a
zstreamtest can fuzztest pledgedSrcSize
2016-09-26 14:06:08 +02:00
58d5dfea54
zstreamtest uses ZSTD_reset?Stream
2016-09-25 01:34:03 +02:00
3ecbe6a37c
fileio uses ZSTD_resetDStream()
2016-09-14 17:26:59 +02:00
b3060f7a9e
changed streaming decoder behavior : now, when all compressed frame is consumed, it means decompression is completed, with regenerated data fully flushed.
2016-09-09 16:44:16 +02:00
12083a45d4
more context-reuse tests
2016-09-06 15:01:51 +02:00
ef9999f0b9
zstreamtest depends only on standard C time.h
2016-09-01 16:44:48 -07:00
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
9a021c1aae
fixed some minor clang warnings
2016-08-26 09:05:06 +02:00
bb8558164e
fixed welcome message
2016-08-25 19:11:11 +02:00
17e482efdd
added ZSTD_setDStreamParameter()
2016-08-23 16:58:10 +02:00
70e3b31306
fixed playtests on os-x
2016-08-23 01:18:06 +02:00
cb3276329a
added sizeof CStream and DStream
2016-08-23 00:31:59 +02:00
2868d565e4
test-related files moved to tests/
2016-08-18 13:18:11 +02:00
Yann Collet
Nick Terrell
Yann Collet
Stella Lau
Yann Collet
Sean Purcell
Przemyslaw Skibinski