Shawn Landden
914d983879
fix unbounded range
...
I think you meant 8 MiB or smaller, instead of an unbounded (and illogical) range
2017-12-21 16:15:12 -08:00
Yann Collet
78de28239f
minor readme formatting update
2017-12-15 13:26:39 -08:00
Yann Collet
2cff66b62f
version bump to v1.3.3
2017-12-14 16:11:20 -08:00
Yann Collet
bd88f633ac
zstreamtest : in `-T#s`, s considered a suffix meaning "seconds"
...
avoid unintentionnally triggering `seedset`,
so that seed gets automatically determined when not set.
2017-11-27 12:15:23 -08:00
Yann Collet
4154aec679
fixed comment, as suggested by @terrelln
2017-11-21 10:26:17 -08:00
Yann Collet
fccb46fbe0
minor spelling fixes
2017-11-18 11:28:00 -08:00
Yann Collet
ca1a9ebac5
fixed zlib wrapper
...
it was invoking ZSTD_initCStream_advanced() with pledgedSrcSize==0 and contentSizeFlag=1
which means "empty"
while the intention was to mean "unknown".
The contentSizeFlag==1 is new, it is a consequence of setting this value to 1 by default.
The solution selected here is to pass ZSTD_CONTENTSIZE_UNKNOWN to mean "unknown".
So contentSizeFlag remains set (it wasn't in previous versions).
2017-10-18 11:22:23 -07:00
Yann Collet
5eed8e7a55
changed API comments to invite using macro ZSTD_CONTENTSIZE_UNKNOWN
...
to mean "pledgedSrcSize is not known at init time" instead of `0`.
Note that, a few prototypes created and documented with `0` to mean "unknown" still interpret "0" as unknown,
to avoid breaking 3rd party applications which depend on this behavior.
But this value is no longer recommended to mean "unknown".
In some future version, it might be possible to switch "0" to mean "empty",
as is already the case for several prototypes.
The advantage is that pledgedSrcSize field would have same behavior accross entire API,
making it easier to reason about.
Note that all concerned prototypes belong to the "experimental" API section.
srcSize is controlled at end of compression,
so if someone uses "0" to mean "unknown" while it effectively means "empty",
this is immediately caught by the compression function, which generates an error code : ZSTD_ERROR_srcSize_wrong
2017-10-14 00:32:06 -07:00
Nick Terrell
6d8778d8e7
[doc] Add images for release notes
2017-10-09 16:44:31 -07:00
Yann Collet
7f6a783862
fixed a small error in decodeCorpus
...
a compressed block must be strictly smaller than its decompressed size.
2017-10-07 15:19:52 -07:00
Yann Collet
ecf1778e23
updated ZSTD_format_e value validation
...
also updated manual
2017-09-27 11:19:21 -07:00
Yann Collet
52a1d1c6dc
added ZSTD_DCtx_reset()
2017-09-25 16:56:48 -07:00
Yann Collet
f2a913862c
added ZSTD_decompress_generic_simpleArgs()
2017-09-25 15:46:34 -07:00
Yann Collet
6ee05a02b8
added ZSTD_decompress_generic()
...
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
2017-09-25 15:41:48 -07:00
Yann Collet
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
Yann Collet
7c3dea42ce
added prototypes for advanced parameters for decompression API
...
required to decode custom formats
2017-09-24 15:57:29 -07:00
Yann Collet
086b9597d9
added ability to split input files for dictionary training
...
using command -B#
This is the same behavior as benchmark module,
which can also split input into arbitrary size blocks, using -B#.
2017-09-14 16:45:10 -07:00
Yann Collet
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
Yann Collet
ce31004f20
fix following suggestions by @terrelln
2017-09-11 13:12:52 -07:00
Yann Collet
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
Stella Lau
eeff55dfa8
Merge remote-tracking branch 'upstream/dev' into ldm-mergeDev
2017-09-06 15:56:32 -07:00
Stella Lau
8c33cfe0bc
Add ldm documentation in README
2017-09-06 15:21:01 -07:00
Yann Collet
0c314cde4b
updated zstd API manual
...
for new CCtxParams object
2017-08-31 18:28:19 -07:00
Yann Collet
b0cb081dc8
last batch of header files changed to reflect new license ( #825 )
...
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
Yann Collet
e21384fffb
fixed more file headers after license change ( #825 )
2017-08-31 12:11:57 -07:00
Yann Collet
e8d35cc5e9
minor formulation change, recommended by @ulikunitz
2017-08-20 10:39:20 -07:00
Yann Collet
d0d06e421f
added alternative representation for huffman bistream
2017-08-19 12:26:09 -07:00
Yann Collet
8b12812147
fix #803 : wrong example in huffman bitstream section, reported by @ulikunitz
2017-08-19 12:17:57 -07:00
Paul Cruz
bef5eda8d9
const vars, change copy_literals() to only take size_t literal_length
2017-08-16 11:11:52 -07:00
Paul Cruz
8d3f18af2c
renamed IO functions for clarity
2017-08-14 17:51:51 -07:00
Paul Cruz
6aebcfa0bc
added comments for new functions
2017-08-14 15:11:01 -07:00
Paul Cruz
9d56c21279
added separate function for executing match copy command
2017-08-14 15:06:03 -07:00
Paul Cruz
d3e57db0bd
created separate function for offset computation
2017-08-14 14:20:12 -07:00
Paul Cruz
b6d6be58c9
created separate function for copying literals during sequence execution
2017-08-14 14:05:16 -07:00
Paul Cruz
93c1309fd4
added free to free_dictionary()
2017-08-14 13:08:30 -07:00
Paul Cruz
38f4e43381
added error checking for dictionary initialized with null src
2017-08-14 09:41:04 -07:00
Paul Cruz
7ef9c6f4b2
made separate API for dictionary management
2017-08-11 18:40:19 -07:00
Paul Cruz
bfc6db8d6a
exposed dictionary functions/types
2017-08-11 17:53:37 -07:00
Paul Cruz
bd308d806b
remove debug symbols when cleaning, added a simple test
2017-08-11 14:42:15 -07:00
Paul Cruz
9f67e8652e
fixed warnings shown by compiler
2017-08-11 14:41:44 -07:00
Paul Cruz
d0dc675596
add makefile
2017-08-11 14:35:13 -07:00
Paul Cruz
736a28d835
reduce educational decoder to single frame decompression
2017-08-11 14:34:49 -07:00
Yann Collet
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
Yann Collet
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
Yann Collet
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
Yann Collet
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
Yann Collet
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
Yann Collet
7758ed8458
fixed fullbench, part 2
2017-07-06 02:48:00 -07:00
Yann Collet
037466245f
refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget()
...
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.
Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
2017-06-28 20:24:08 -07:00
Yann Collet
2bf428df45
Merge branch 'advancedAPI2' into refPrefix
2017-06-28 16:35:49 -07:00
Yann Collet
813535105b
added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0]
...
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
2017-06-28 15:34:56 -07:00
Yann Collet
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
Yann Collet
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
Yann Collet
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
Yann Collet
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
Yann Collet
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
Yann Collet
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
Yann Collet
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
Yann Collet
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
Yann Collet
c08e649e95
first implementation of bench.c with new API ZSTD_compress_generic()
...
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.
Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
Yann Collet
bf99150be3
update new api presentation in zstd.h and manual
2017-06-19 12:56:25 -07:00
Yann Collet
c7dcf0f379
updated manual
2017-06-19 12:03:25 -07:00
Yann Collet
01743a36e7
fuzzer tests for new API
2017-06-16 17:56:41 -07:00
Yann Collet
f35e2de61c
linked newAPI to ZSTDMT
2017-06-05 18:32:48 -07:00
Yann Collet
8c910d2097
updated ZSTDMT streaming API
...
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
Yann Collet
58e8d793e1
made debug definitions common within zstd_internal.h
2017-06-02 18:20:48 -07:00
Yann Collet
deee6e523f
expose ZSTD_compress_generic_simpleArgs()
...
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
2017-05-30 17:42:00 -07:00
Yann Collet
44e45e8423
added ZSTDMT_createCCtx_advanced()
...
make it possible to use custom allocators
2017-05-30 16:12:06 -07:00
Yann Collet
f45ca527a1
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-05-30 10:02:03 -07:00
Yann Collet
b8136f019a
static dctx is incompatible with legacy support
...
documented, and runtime tested
2017-05-27 00:03:08 -07:00
Yann Collet
7028cbd7fd
fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader
2017-05-25 18:29:08 -07:00
Yann Collet
cdf7e82222
Added ZSTD_initStaticCDict()
2017-05-25 18:05:49 -07:00
Yann Collet
57827f906f
added ZSTD_initStaticDDict()
2017-05-25 15:44:06 -07:00
Yann Collet
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
Yann Collet
0fdc71c3dc
added ZSTD_initStaticDCtx()
2017-05-24 17:41:41 -07:00
Yann Collet
c7fe262dc9
added ZSTD_initStaticCCtx()
...
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
Yann Collet
b0739bcf8f
simplified reset by removing full-reset policy
...
this was meant to be applied prior to dictionary loading.
But effectively, it seems redundant with later loading stage,
so it can be skipped safely.
2017-05-22 17:45:15 -07:00
Yann Collet
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
Yann Collet
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
Yann Collet
a5ffe3d370
pushed enum values for strategy by one (ZSTD_fast==1)
...
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
Yann Collet
add66f816d
changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams()
...
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
2017-05-12 16:01:15 -07:00
Yann Collet
b0edb7fb0e
added ZSTD_CCtx_setParameter()
2017-05-12 15:31:53 -07:00
Yann Collet
db8e21d5a0
made ZSTD_compress_generic() definition accessible
...
note that the implementation is not done yet.
2017-05-12 13:46:49 -07:00
Yann Collet
33eb7ac6b6
updated Advanced API proposal
...
only declarations in zstd.h
2017-05-12 12:36:11 -07:00
Nick Terrell
5f2c7213c7
Merge remote-tracking branch 'upstream/dev' into btopt
...
* upstream/dev: (305 commits)
added test for ZSTD_estimateCStreamSize()
changed variable name, for clarity
fixed ZSTD_estimateCStreamSize()
shortened ZSTD_createCStream_Advanced()
fixed symbols test
added ZSTD_estimateDStreamSize()
changed name frameParams into frameHeader
regroup memory usage function declarations
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
bumped version number
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
Updated ZSTD_freeCCtx()
updated ZSTD_estimateCCtxSize()
Updated ZSTD_sizeof_CCtx()
merged CCtx and CStream as a single same object
cli : -d and -t do not stop after a failed decompression
added dev branch CircleCI badge
added dev branch Appveyor badge
keep dev branch status only
creates a binary archive without the `programs` directory
...
2017-05-10 16:49:58 -07:00
Yann Collet
cef02d9317
changed variable name, for clarity
...
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
Yann Collet
f16f4497ca
added ZSTD_estimateDStreamSize()
2017-05-09 16:18:17 -07:00
Yann Collet
542c9dfcf8
changed name frameParams into frameHeader
...
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()
The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.
ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
Yann Collet
5a36c069e7
regroup memory usage function declarations
...
in a single paragraph in zstd.h, for clarity
2017-05-09 15:11:30 -07:00
Yann Collet
fa8dadb294
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
...
for clarity
2017-05-08 18:24:16 -07:00
Yann Collet
a1d6704d7f
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
...
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
Yann Collet
fc5145955a
updated ZSTD_estimateCCtxSize()
...
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).
Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
Yann Collet
0be6fd3429
merged CCtx and CStream as a single same object
...
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
Yann Collet
c08e56861e
updated dict graphs to 2D mode
2017-05-03 14:33:28 -07:00
Yann Collet
01a71739b0
updated DSpeed chart to remove 3D effect ( #589 )
2017-05-02 14:39:03 -07:00
Yann Collet
202082f285
sync bitstream from FSE project
...
add assert into unsafe *_fast() variants
2017-04-28 17:00:31 -07:00
Yann Collet
a51cab6e68
Merge pull request #678 from facebook/apiChange
...
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
Yann Collet
8b669535f8
bumped version number to v1.2.0
2017-04-27 12:50:20 -07:00
Yann Collet
77bf59ef50
added ZSTD_initCStream_usingCDict_advanced()
2017-04-27 11:43:04 -07:00
Yann Collet
f4bd857d81
created ZSTD_compress_usingCDict_advanced()
2017-04-27 11:31:55 -07:00