73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
2300c58a6f
Move dictContentByRef to cctx params
2017-08-18 12:03:16 -07:00
b6cb2ed8cb
Move dictMode to cctxParams
2017-08-18 11:43:31 -07:00
97e27affcb
Move compression level to cctx params
2017-08-18 11:20:08 -07:00
c0221124d5
Add function to set opaque parameters
2017-08-17 19:30:22 -07:00
4169f49171
Add initialization/allocation functions for opaque params
2017-08-17 18:45:04 -07:00
ade95b8bed
Add opaque interfaces for static initialization
2017-08-17 18:13:08 -07:00
699f11b4f7
Create opaque parameter structure
2017-08-17 17:33:46 -07:00
e1222544be
Merge pull request #753 from paulcruz74/adapt-approach-3
...
adaptive compression v1
2017-07-27 10:00:10 -07:00
6945b3c43d
removed previous version of completion for compression
2017-07-19 11:51:50 -07:00
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
4e77f7761d
clarified comment on ZSTD_p_contentSizeFlag
2017-07-13 02:09:07 -07:00
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
97f2bf66da
minor : fix typo
2017-06-29 11:31:40 -07:00
acbef3decd
ZSTD_getFrameContentSize() is promoted to "stable" status
2017-06-29 05:19:51 -07:00
2bf428df45
Merge branch 'advancedAPI2' into refPrefix
2017-06-28 16:35:49 -07:00
813535105b
added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0]
...
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
2017-06-28 15:34:56 -07:00
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
78b8234554
fixed comments, following suggestion by @terrelln
2017-06-20 14:26:48 -07:00
fe234bf48b
fix attempts : fullbench for VS2008
2017-06-19 15:23:19 -07:00
bf99150be3
update new api presentation in zstd.h and manual
2017-06-19 12:56:25 -07:00
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
381e66cfbd
added ZSTD_clampCParams()
...
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
2017-06-16 17:34:54 -07:00
559ee82e90
fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps
2017-06-16 11:58:21 -07:00
bd18c885a3
added ZSTD_CCtx_reset
2017-06-16 10:17:50 -07:00
05ae4b2190
added protection : MT incompatible with Static allocation
2017-06-15 18:03:34 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
f35e2de61c
linked newAPI to ZSTDMT
2017-06-05 18:32:48 -07:00
8c910d2097
updated ZSTDMT streaming API
...
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
c35e535002
added support for multithreading parameters
2017-06-01 18:44:06 -07:00
deee6e523f
expose ZSTD_compress_generic_simpleArgs()
...
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
2017-05-30 17:42:00 -07:00
44e45e8423
added ZSTDMT_createCCtx_advanced()
...
make it possible to use custom allocators
2017-05-30 16:12:06 -07:00
f45ca527a1
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-05-30 10:02:03 -07:00
b8136f019a
static dctx is incompatible with legacy support
...
documented, and runtime tested
2017-05-27 00:03:08 -07:00
7028cbd7fd
fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader
2017-05-25 18:29:08 -07:00
cdf7e82222
Added ZSTD_initStaticCDict()
2017-05-25 18:05:49 -07:00
57827f906f
added ZSTD_initStaticDDict()
2017-05-25 15:44:06 -07:00
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
0fdc71c3dc
added ZSTD_initStaticDCtx()
2017-05-24 17:41:41 -07:00
11ea2f7fda
Merged ZSTD_DCtx and ZSTD_DStream objects
...
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
2017-05-23 16:19:43 -07:00
c7fe262dc9
added ZSTD_initStaticCCtx()
...
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
24de7b0346
Implemented ZSTD_CCtx_refCDict()
2017-05-22 13:05:45 -07:00
ee970398b2
Merge branch 'dev' into advancedAPI2
2017-05-22 12:33:56 -07:00
a1280406b0
[libzstd] Allow users to define custom visibility
2017-05-19 18:01:59 -07:00
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
009d604e00
ZSTD_compress_generic() supports multiple successive frames
...
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
23c256e44b
removed useless variable from CCtx
...
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
9f95e445ab
minor comment clarifications
2017-05-16 17:26:43 -07:00
0bdb575c31
Merge branch 'dev' into advancedAPI2
2017-05-16 16:32:29 -07:00
bfff8999c5
added prototype ZSTD_versionString()
2017-05-16 16:12:23 -07:00
4eff8136aa
added prototype ZSTD_decompressBegin_usingDDict ( #700 )
2017-05-16 16:05:27 -07:00
a5ffe3d370
pushed enum values for strategy by one (ZSTD_fast==1)
...
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
add66f816d
changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams()
...
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
2017-05-12 16:01:15 -07:00
b0edb7fb0e
added ZSTD_CCtx_setParameter()
2017-05-12 15:31:53 -07:00
db8e21d5a0
made ZSTD_compress_generic() definition accessible
...
note that the implementation is not done yet.
2017-05-12 13:46:49 -07:00
33eb7ac6b6
updated Advanced API proposal
...
only declarations in zstd.h
2017-05-12 12:36:11 -07:00
5f2c7213c7
Merge remote-tracking branch 'upstream/dev' into btopt
...
* upstream/dev: (305 commits)
added test for ZSTD_estimateCStreamSize()
changed variable name, for clarity
fixed ZSTD_estimateCStreamSize()
shortened ZSTD_createCStream_Advanced()
fixed symbols test
added ZSTD_estimateDStreamSize()
changed name frameParams into frameHeader
regroup memory usage function declarations
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
bumped version number
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
Updated ZSTD_freeCCtx()
updated ZSTD_estimateCCtxSize()
Updated ZSTD_sizeof_CCtx()
merged CCtx and CStream as a single same object
cli : -d and -t do not stop after a failed decompression
added dev branch CircleCI badge
added dev branch Appveyor badge
keep dev branch status only
creates a binary archive without the `programs` directory
...
2017-05-10 16:49:58 -07:00
cef02d9317
changed variable name, for clarity
...
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
f16f4497ca
added ZSTD_estimateDStreamSize()
2017-05-09 16:18:17 -07:00
542c9dfcf8
changed name frameParams into frameHeader
...
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()
The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.
ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
5a36c069e7
regroup memory usage function declarations
...
in a single paragraph in zstd.h, for clarity
2017-05-09 15:11:30 -07:00
fa8dadb294
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
...
for clarity
2017-05-08 18:24:16 -07:00
51652522a2
bumped version number
2017-05-08 17:52:46 -07:00
a1d6704d7f
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
...
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
fc5145955a
updated ZSTD_estimateCCtxSize()
...
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).
Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
791d744279
Updated ZSTD_sizeof_CCtx()
...
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
2017-05-08 16:17:30 -07:00
0be6fd3429
merged CCtx and CStream as a single same object
...
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
a51cab6e68
Merge pull request #678 from facebook/apiChange
...
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
a92cbb7004
Added a secondary test, checking dictID presence after setting noDictIdFLag=1
2017-04-27 15:08:56 -07:00
8b669535f8
bumped version number to v1.2.0
2017-04-27 12:50:20 -07:00
77bf59ef50
added ZSTD_initCStream_usingCDict_advanced()
2017-04-27 11:43:04 -07:00
f4bd857d81
created ZSTD_compress_usingCDict_advanced()
2017-04-27 11:31:55 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
768df129d2
changed ZSTD_compressBegin_usingCDict()
...
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
2017-04-26 15:42:10 -07:00
71ddeb67b1
made room in workspace for FSE tables
...
still need to be transferred from CCtx into workspace
2017-04-20 22:54:54 -07:00
715b9aa113
created ZSTD_compressBegin_usingCDict_advanced()
2017-04-18 13:55:53 -07:00
af4f45b682
Improved code comments for block functions
2017-04-18 03:17:44 -07:00
4f818182b8
clarified frame parameters for ZSTD_compress*_usingCDict()
...
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
2017-04-17 18:29:06 -07:00
36c2a03757
updated comments for ZSTD_resetCStream()
2017-04-05 22:06:21 -07:00
7cf78f1be7
Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference
...
Will issue an error (GENERIC) is cdict==NULL
2017-04-04 12:38:14 -07:00
f3dfcdccd1
bump version number
2017-03-21 12:18:28 -07:00
dec2b96536
Add functions missing from manual, and fix parameter alignment
2017-03-14 11:24:09 -07:00
eeb31eed39
s/ZSTD_btopt2/ZSTD_btultra/g
2017-03-09 11:44:25 -08:00
e65aab8e0f
Remove 'mem.h' dependency from ZSTD_WINDOWLOG_MAX
2017-03-08 15:40:13 -08:00
d44703d145
Offsets >= 32MB in 32-bits mode
2017-03-01 16:27:56 -08:00
14312d833e
zstdmt : fix : loading prefix from previous segments
...
There used to be a (very small) chance that
loading prefix from previous segment
would be confused with a real zstd dictionary.
For that to happen, the prefix needs to start
with the same value as dictionary magic.
That's 1 chance in 4 billions if all values have equal probability.
But in fact, since some values are more common (0x00000000 for example)
others are less common, and dictionary magic was selected to be one of them,
so probabilities are likely even lower.
Anyway, this risk is no down to zero
by adding a new CCtx parameter : ZSTD_p_forceRawDict
Current parameter policy : the parameter "stick" to its CCtx,
so any dictionary loading after ZSTD_p_forceRawDict is set
will be loaded in "raw" ("content only") mode,
even if CCtx is re-used multiple times with multiple different dictionary.
It's up to the user to reset this value differently if it needs so.
2017-02-23 23:42:12 -08:00
831b4890ce
minor tests/Makefile refactoring
...
and update of zstd_manual,html
2017-02-23 23:09:10 -08:00
64417cd2ff
Describe ambiguity around skippable frames
2017-02-22 13:29:01 -08:00
9757cc811b
Update comment
2017-02-22 12:28:21 -08:00
9050e1925e
Change name to to findFrameCompressedSize and add skippable support
2017-02-22 12:12:34 -08:00
517577bf53
spelling fixes in comments
...
i.e. occurred labeled Huffman
2017-02-20 12:08:59 -08:00
58af614ef2
push version and NEWS to v1.1.4
2017-02-13 18:32:44 -08:00
d7bfcac18a
Expose frameSrcSize to experimental API
2017-02-10 11:55:44 -08:00
269b2cd3d8
Documentation updates
2017-02-09 13:25:30 -08:00
2db7249265
Make pledgedSrcSize meaning clear for other functions
...
- Added tests
- Moved new size functions to static link only
2017-02-09 11:49:58 -08:00
0f5c95af44
Disambiguate pledgedSrcSize == 0
...
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
2017-02-08 15:12:46 -08:00
4e709712e1
Decompressed size functions now handle multiframes and distinguish cases
...
- Add ZSTD_findDecompressedSize
- Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
- Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
2017-02-08 14:50:10 -08:00
ef33d00532
fixed : ZSTD_setCCtxParameter() properly exposed in DLL
2017-01-26 12:24:21 -08:00
4a62f79ec9
fixed clang documentation warning
2017-01-26 09:16:56 -08:00
06e7697f96
added test of new parameter ZSTD_p_forceWindow
2017-01-25 16:39:03 -08:00
bb0027405a
fixed zstdmt corruption issue when enabling overlapped sections
...
see Asana board for detailed explanation on why and how to fix it
2017-01-25 16:25:38 -08:00
c593348722
ZSTDMT_initCStream_usingDict() can outlive dict
...
Like ZSTD_initCStream_usingDict(),
ZSTDMT_initCStream_usingDict() now keep a copy of dict internally.
This way, dict can be released :
it does not longer have to outlive all future compression sessions.
2017-01-22 16:44:15 -08:00
d7e3cb58c5
Resolved merge conflict dev+zstdmt
2017-01-20 16:44:50 -08:00
dab5ea93f2
Merge pull request #515 from iburinoc/emptydict
...
Don't create dict in streaming apis if dictSize == 0
2017-01-19 09:02:42 -08:00
0b5370ae38
Prefix notes with /**<
2017-01-18 13:45:02 -08:00
57d423c5df
Don't create dict in streaming apis if dictSize == 0
2017-01-17 14:31:35 -08:00
d72f4b6b7a
added "Makefile is validated"
2017-01-17 12:40:06 +01:00
5eb749e734
ZSTDMT_compress() creates a single frame
...
The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()
As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().
This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.
2017-01-11 18:21:25 +01:00
0ec6a95ba1
minor fixes
2017-01-02 00:49:42 +01:00
ba75e9d8c3
fix : zlib wrapper compile in gnu90 mode
2016-12-21 19:57:18 +01:00
4e5eea61a8
added ZSTD_createDDict_byReference()
2016-12-21 16:44:35 +01:00
1f57c2ed32
added : ZSTD_createCDict_byReference()
2016-12-21 16:20:11 +01:00
d46ecb58a5
added dll compilation tests
2016-12-17 16:28:12 +01:00
8de46ab51a
Export all API functions
2016-12-16 13:27:30 -08:00
2b36b238d3
changed variable name to estimatedSrcSize, to emphasize it does not need to be exact
2016-12-13 17:59:55 +01:00
e795c8a5f6
Added ZSTD_initCStream_srcSize().
...
Added relevant test cases in zstreamtest
2016-12-13 17:00:14 +01:00
7687913178
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
2016-12-08 10:42:42 +01:00
426a9d4b71
changed : dll : only approved ZSTD symbols are now exposed. All other symbols remain internal.
2016-12-07 16:39:34 -08:00
4da53219a0
zstd Manual updated to 1.1.2
2016-12-07 11:18:40 +01:00
379908be3d
fixed zstd.h for manual
2016-12-06 10:36:15 -08:00
825dffbc43
moved zbuff source files into lib/deprecated
2016-12-05 19:28:19 -08:00
e7a41a5955
added : dictID retrieval functions.
...
added : unit tests for dictID retrieval functions
2016-12-05 16:21:06 -08:00
9ffbeea875
API : changed : streaming decompression : implicit reset on starting new frames
2016-12-02 18:37:38 -08:00
05c00f2ff7
Fix ZSTD_STATIC_LINKING_ONLY with double include
2016-11-29 11:54:39 -08:00
179555c1d1
working fullbench-dll
2016-11-15 18:05:46 +01:00
fd3be6bc97
bump version number to 1.1.2
2016-11-07 14:35:41 -08:00
11812260d1
Merge pull request #439 from terrelln/dev
...
ZSTD_compress_usingDict() specify dict == NULL
2016-11-03 14:15:36 -07:00
d82efd8a70
ZSTD_compress_usingDict() when dict gets loaded
...
Specify that when `dict == NULL || dictSize < 8` no dictionary
gets loaded.
Also add some periods.
2016-11-02 18:07:16 -07:00
179b19776f
fileio.c does no longer need ZSTD_LEGACY_SUPPORT, and does no longer depend on zstd_legacy.h
...
Added : ZSTD_isFrame() in experimental section
2016-11-02 17:30:49 -07:00
ee5b725823
ZSTD_initCStream() optimization : do not allocate a CDict when no dictionary used
2016-10-27 14:20:55 -07:00
9516234e67
first sketch for ZSTD_initCStream_usingCDict()
2016-10-25 16:19:52 -07:00
62d9a7ddfd
Merge pull request #429 from inikep/btopt2
...
Btopt2
2016-10-25 14:48:43 -07:00
5c5f01f3da
added ZSTD_btopt2 strategy
2016-10-25 12:25:07 +02:00
37d130031d
updated comments on context re-use
2016-10-24 17:22:12 -07:00
Stella Lau
Yann Collet
Yann Collet
Paul Cruz
Yann Collet
Nick Terrell
Sean Purcell
Anders Oleson
Przemyslaw Skibinski