df4e9bba25
fixed constant errors for gcc in c99 mode
...
C standard does not consider a `static const int` as a constant.
This is a problem for initializer, and ZSTD_STATIC_ASSERT().
Replaced by macro values
2017-09-26 14:31:06 -07:00
c233bdbaee
Increase maximum window size
...
* Maximum window size in 32-bit mode is 1GB, since allocations for 2GB fail
on my Mac.
* Maximum window size in 64-bit mode is 2GB, since that is the largest
power of 2 that works with the overflow prevention.
* Allow `--long=windowLog` to set the window log, along with
`--zstd=wlog=#`. These options also set the window size during
decompression, but don't override `--memory=#` if it is set.
* Present a helpful error message when the window size is too large during
decompression.
* The long range matcher defaults to a hash log 7 less than the window log,
which keeps it at 20 for window log 27.
* Keep the default long range matcher window size and the default maximum
window size at 27 for the API and CLI.
* Add tests that use the maximum window size and hash size for compression
and decompression.
2017-09-26 14:00:01 -07:00
52a1d1c6dc
added ZSTD_DCtx_reset()
2017-09-25 16:56:48 -07:00
f2a913862c
added ZSTD_decompress_generic_simpleArgs()
2017-09-25 15:46:34 -07:00
6ee05a02b8
added ZSTD_decompress_generic()
...
same as ZSTD_decompressStream(),
just for a similar feeling as the compression side, which uses ZSTD_compress_generic()
2017-09-25 15:41:48 -07:00
62568c9a42
added capability to generate magic-less frames
...
decoder not implemented yet
2017-09-25 14:26:26 -07:00
96f0cde31a
minor function rename
...
ZSTD_estimateCStreamSize_advanced_usingCParams -> ZSTD_estimateCStreamSize_usingCParams
_usingX is clear.
_advanced feels redundant
2017-09-24 16:47:02 -07:00
7c3dea42ce
added prototypes for advanced parameters for decompression API
...
required to decode custom formats
2017-09-24 15:57:29 -07:00
e60f48c549
Merge branch 'dev' into newFormats
2017-09-24 14:33:37 -07:00
d6abb28951
Prepare for ZSTD_WINDOWLOG_MAX == 31
2017-09-21 17:18:41 -07:00
f97c2dbd39
created ZSTD_format declaration
2017-09-21 16:07:29 -07:00
f1571dad8f
Merge pull request #838 from stellamplau/ldm-mergeDev
...
Add long distance matcher
2017-09-13 13:24:08 -07:00
eb3327c10a
Merge branch 'dev' of https://github.com/facebook/zstd into ldm-mergeDev
2017-09-11 15:00:01 -07:00
ce31004f20
fix following suggestions by @terrelln
2017-09-11 13:12:52 -07:00
058ed2ad33
ZSTD_decodingBufferSize_min()
...
supporting function for bufferless streaming API (ZSTD_decompressContinue())
makes it possible to correctly size a round buffer for decoding using this API.
also : added field blockSizeMax within ZSTD_frameHeader,
as it's a necessary information to know when to restart at beginning of decoding buffer.
2017-09-09 01:03:29 -07:00
3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
9e4060200b
Add tests and fix pointer alignment
2017-09-06 09:14:05 -07:00
c706de5395
Rename and add short ldm parameters in cli
2017-09-05 21:11:18 -07:00
643d28c701
Add ldm options to 'man zstd'
2017-09-05 11:27:15 -07:00
67d4a6161c
Add ldmBucketSizeLog param
2017-09-02 21:55:29 -07:00
a1f04d518d
Move hashEveryLog to cctxParams and update cli
2017-09-01 15:05:47 -07:00
767a0b3be1
Move ldm hashLog, bucketLog, and mml to cctxParams
2017-09-01 12:24:59 -07:00
17d8e0bdcc
Merge remote-tracking branch 'upstream/longRangeMatcher' into ldm-integrate
2017-09-01 10:19:38 -07:00
8081becadc
Add long distance matching as a CCtxParam
2017-09-01 09:18:58 -07:00
d7ad99b2ab
Merge branch 'longRangeMatcher' into dev
2017-08-31 18:08:37 -07:00
6a546efb8c
Add long distance matcher
...
Move last literals section to ZSTD_block_internal
2017-08-31 12:53:19 -07:00
90a31bfa16
Pass dictMode to ZSTDMT_initCStream; fix nits
...
- Return error code in estimate{CCtx,CStream}Size functions
2017-08-30 16:19:07 -07:00
82d636b76a
Rename applyCCtxParams()
2017-08-29 18:03:06 -07:00
c7a18b7c21
Localize 'dictMode' from cctx to function param
2017-08-29 15:52:24 -07:00
c88fb9267f
Replace 'byReference' with enum
2017-08-29 11:55:02 -07:00
b5b9275e67
Rename estimateCCtxSize_advanced() and estimateCStreamSize_advanced()
2017-08-29 10:49:29 -07:00
18224608ff
Remove ZSTD_setCCtxParameter()
2017-08-25 13:58:41 -07:00
0744592d38
Add function initializing cctxParams from clevel
2017-08-25 13:36:47 -07:00
eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
5bc2c1e982
Add prototype support for customMem with cctxParams
2017-08-23 12:03:30 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
1c0dbe81b1
Add documentation for CCtx_params
2017-08-21 13:18:00 -07:00
939f954285
Pass ZSTD_CCtx_params as const ptr when possible
2017-08-21 12:57:18 -07:00
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
2300c58a6f
Move dictContentByRef to cctx params
2017-08-18 12:03:16 -07:00
b6cb2ed8cb
Move dictMode to cctxParams
2017-08-18 11:43:31 -07:00
97e27affcb
Move compression level to cctx params
2017-08-18 11:20:08 -07:00
c0221124d5
Add function to set opaque parameters
2017-08-17 19:30:22 -07:00
4169f49171
Add initialization/allocation functions for opaque params
2017-08-17 18:45:04 -07:00
ade95b8bed
Add opaque interfaces for static initialization
2017-08-17 18:13:08 -07:00
699f11b4f7
Create opaque parameter structure
2017-08-17 17:33:46 -07:00
e1222544be
Merge pull request #753 from paulcruz74/adapt-approach-3
...
adaptive compression v1
2017-07-27 10:00:10 -07:00
6945b3c43d
removed previous version of completion for compression
2017-07-19 11:51:50 -07:00
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
4e77f7761d
clarified comment on ZSTD_p_contentSizeFlag
2017-07-13 02:09:07 -07:00
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
97f2bf66da
minor : fix typo
2017-06-29 11:31:40 -07:00
acbef3decd
ZSTD_getFrameContentSize() is promoted to "stable" status
2017-06-29 05:19:51 -07:00
2bf428df45
Merge branch 'advancedAPI2' into refPrefix
2017-06-28 16:35:49 -07:00
813535105b
added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0]
...
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
2017-06-28 15:34:56 -07:00
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
78b8234554
fixed comments, following suggestion by @terrelln
2017-06-20 14:26:48 -07:00
fe234bf48b
fix attempts : fullbench for VS2008
2017-06-19 15:23:19 -07:00
bf99150be3
update new api presentation in zstd.h and manual
2017-06-19 12:56:25 -07:00
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
381e66cfbd
added ZSTD_clampCParams()
...
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
2017-06-16 17:34:54 -07:00
559ee82e90
fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps
2017-06-16 11:58:21 -07:00
bd18c885a3
added ZSTD_CCtx_reset
2017-06-16 10:17:50 -07:00
05ae4b2190
added protection : MT incompatible with Static allocation
2017-06-15 18:03:34 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
f35e2de61c
linked newAPI to ZSTDMT
2017-06-05 18:32:48 -07:00
8c910d2097
updated ZSTDMT streaming API
...
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
c35e535002
added support for multithreading parameters
2017-06-01 18:44:06 -07:00
deee6e523f
expose ZSTD_compress_generic_simpleArgs()
...
which is a binding towards ZSTD_compress_generic()
using only integral types for arguments.
2017-05-30 17:42:00 -07:00
44e45e8423
added ZSTDMT_createCCtx_advanced()
...
make it possible to use custom allocators
2017-05-30 16:12:06 -07:00
f45ca527a1
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-05-30 10:02:03 -07:00
b8136f019a
static dctx is incompatible with legacy support
...
documented, and runtime tested
2017-05-27 00:03:08 -07:00
7028cbd7fd
fixed a few code comments : ZSTD_getFrameParams => ZSTD_getFrameHeader
2017-05-25 18:29:08 -07:00
cdf7e82222
Added ZSTD_initStaticCDict()
2017-05-25 18:05:49 -07:00
57827f906f
added ZSTD_initStaticDDict()
2017-05-25 15:44:06 -07:00
25989e361c
updated ZSTD_estimate?DictSize() to pass parameter byReference
...
resulting ?Dict object is smaller when created byReference.
Seems better than a documentation note.
2017-05-25 15:07:37 -07:00
0fdc71c3dc
added ZSTD_initStaticDCtx()
2017-05-24 17:41:41 -07:00
11ea2f7fda
Merged ZSTD_DCtx and ZSTD_DStream objects
...
They are now the same object.
It's recommended to keep both types in source code
as previous versions of library (<v1.3.0)
still need this differentiation.
2017-05-23 16:19:43 -07:00
c7fe262dc9
added ZSTD_initStaticCCtx()
...
makes it possible to statically or externally allocate CCtx.
static CCtx will only use provided memory area,
it will never resize nor malloc.
2017-05-23 13:20:41 -07:00
24de7b0346
Implemented ZSTD_CCtx_refCDict()
2017-05-22 13:05:45 -07:00
ee970398b2
Merge branch 'dev' into advancedAPI2
2017-05-22 12:33:56 -07:00
a1280406b0
[libzstd] Allow users to define custom visibility
2017-05-19 18:01:59 -07:00
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
009d604e00
ZSTD_compress_generic() supports multiple successive frames
...
also : clarified streaming API implementation
2017-05-19 10:17:59 -07:00
6d4fef36de
Added ZSTD_compress_generic()
...
Used in fileio.c (zstd cli).
Need to set macro ZSTD_NEWAPI to trigger it.
2017-05-17 18:36:15 -07:00
23c256e44b
removed useless variable from CCtx
...
CStream's pledgedSrcSize is no longer necessary
srcSize control is realized within bufferless interface.
2017-05-16 18:10:11 -07:00
9f95e445ab
minor comment clarifications
2017-05-16 17:26:43 -07:00
0bdb575c31
Merge branch 'dev' into advancedAPI2
2017-05-16 16:32:29 -07:00
bfff8999c5
added prototype ZSTD_versionString()
2017-05-16 16:12:23 -07:00
4eff8136aa
added prototype ZSTD_decompressBegin_usingDDict ( #700 )
2017-05-16 16:05:27 -07:00
a5ffe3d370
pushed enum values for strategy by one (ZSTD_fast==1)
...
this makes it possible to use `0` to mean:
"do not change strategy"
2017-05-12 16:29:19 -07:00
add66f816d
changed macro LOADCPARAMS by static function ZSTD_cLevelToCParams()
...
for improved compiler checks.
Also : ensure most parameters can receive value "0"
to mean "do not change".
2017-05-12 16:01:15 -07:00
b0edb7fb0e
added ZSTD_CCtx_setParameter()
2017-05-12 15:31:53 -07:00
db8e21d5a0
made ZSTD_compress_generic() definition accessible
...
note that the implementation is not done yet.
2017-05-12 13:46:49 -07:00
33eb7ac6b6
updated Advanced API proposal
...
only declarations in zstd.h
2017-05-12 12:36:11 -07:00
5f2c7213c7
Merge remote-tracking branch 'upstream/dev' into btopt
...
* upstream/dev: (305 commits)
added test for ZSTD_estimateCStreamSize()
changed variable name, for clarity
fixed ZSTD_estimateCStreamSize()
shortened ZSTD_createCStream_Advanced()
fixed symbols test
added ZSTD_estimateDStreamSize()
changed name frameParams into frameHeader
regroup memory usage function declarations
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
bumped version number
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
Updated ZSTD_freeCCtx()
updated ZSTD_estimateCCtxSize()
Updated ZSTD_sizeof_CCtx()
merged CCtx and CStream as a single same object
cli : -d and -t do not stop after a failed decompression
added dev branch CircleCI badge
added dev branch Appveyor badge
keep dev branch status only
creates a binary archive without the `programs` directory
...
2017-05-10 16:49:58 -07:00
cef02d9317
changed variable name, for clarity
...
fhiPtr -> zfhPtr
https://github.com/facebook/zstd/pull/689#discussion_r115638676
2017-05-10 11:14:08 -07:00
f16f4497ca
added ZSTD_estimateDStreamSize()
2017-05-09 16:18:17 -07:00
542c9dfcf8
changed name frameParams into frameHeader
...
ZSTD_frameParams => ZSTD_frameHeader
ZSTD_getFrameParams() -> ZSTD_getFrameHeader()
The new naming is more distinctive from ZSTD_frameParameters,
which is used during compression.
ZSTD_frameHeader is clearer in its intention to described frame header content.
It also implies we are decoding a ZSTD frame, hence we are at decoding stage.
2017-05-09 15:46:07 -07:00
5a36c069e7
regroup memory usage function declarations
...
in a single paragraph in zstd.h, for clarity
2017-05-09 15:11:30 -07:00
fa8dadb294
separated ZSTD_estimateCStreamSize() from ZSTD_estimateCCtxSize()
...
for clarity
2017-05-08 18:24:16 -07:00
51652522a2
bumped version number
2017-05-08 17:52:46 -07:00
a1d6704d7f
added ZSTD_estimateCDictSize() and ZSTD_estimateDDictSize()
...
it complements ZSTD_estimateCCtxSize()
for the special case of ZSTD_initCStream_usingDict()
2017-05-08 17:51:49 -07:00
fc5145955a
updated ZSTD_estimateCCtxSize()
...
added a parameter streaming,
to estimate memory allocation size
when the CCtx is used for streaming (CStream).
Note : this function is not able to estimate
memory cost of a potential internal CDict
which can only happen when starting with ZSTD_initCStream_usingDict()
2017-05-08 17:07:59 -07:00
791d744279
Updated ZSTD_sizeof_CCtx()
...
can now contain buffers if object used as CStream.
ZSTD_sizeof_CStream() is now just a thin wrapper of ZSTD_sizeof_CCtx().
2017-05-08 16:17:30 -07:00
0be6fd3429
merged CCtx and CStream as a single same object
...
To be changed : ZSTD_sizeof_CCtx(), ZSTD_estimateCCtxSize()
2017-05-08 16:08:01 -07:00
a51cab6e68
Merge pull request #678 from facebook/apiChange
...
Breaking API Change around CDict
2017-04-28 10:02:45 -07:00
a92cbb7004
Added a secondary test, checking dictID presence after setting noDictIdFLag=1
2017-04-27 15:08:56 -07:00
8b669535f8
bumped version number to v1.2.0
2017-04-27 12:50:20 -07:00
77bf59ef50
added ZSTD_initCStream_usingCDict_advanced()
2017-04-27 11:43:04 -07:00
f4bd857d81
created ZSTD_compress_usingCDict_advanced()
2017-04-27 11:31:55 -07:00
31533bacce
Changed ZSTD_createCDict_advanced()
...
It now only uses compressionParameters as argument.
It produces many changes throughout user code,
though hopefully they tend to be simple :
just provide the cParams part from existing ZSTD_parameters.
Some programs might depend on ZSTD_createCDict_advanced() to pass frame parameters.
This change will force them to revisit this strategy and fix it,
since frame parameters are effectively silently ignored in current version.
2017-04-27 00:29:04 -07:00
768df129d2
changed ZSTD_compressBegin_usingCDict()
...
No longer takes `pledgedSrcSize` as argument
this is in line with similar functions ZSTD_compress_usingCDict()
and ZSTD_initCStream_usingCDict().
2017-04-26 15:42:10 -07:00
71ddeb67b1
made room in workspace for FSE tables
...
still need to be transferred from CCtx into workspace
2017-04-20 22:54:54 -07:00
715b9aa113
created ZSTD_compressBegin_usingCDict_advanced()
2017-04-18 13:55:53 -07:00
af4f45b682
Improved code comments for block functions
2017-04-18 03:17:44 -07:00
4f818182b8
clarified frame parameters for ZSTD_compress*_usingCDict()
...
created ZSTD_compressBegin_usingCDict_internal(),
which gives direct control to frame Parameters.
ZSTD_resetCStream_internal() now points into it.
2017-04-17 18:29:06 -07:00
36c2a03757
updated comments for ZSTD_resetCStream()
2017-04-05 22:06:21 -07:00
7cf78f1be7
Protects ZSTD_compressBegin_usingCDict() vs NULL cdict dereference
...
Will issue an error (GENERIC) is cdict==NULL
2017-04-04 12:38:14 -07:00
f3dfcdccd1
bump version number
2017-03-21 12:18:28 -07:00
dec2b96536
Add functions missing from manual, and fix parameter alignment
2017-03-14 11:24:09 -07:00
eeb31eed39
s/ZSTD_btopt2/ZSTD_btultra/g
2017-03-09 11:44:25 -08:00
e65aab8e0f
Remove 'mem.h' dependency from ZSTD_WINDOWLOG_MAX
2017-03-08 15:40:13 -08:00
d44703d145
Offsets >= 32MB in 32-bits mode
2017-03-01 16:27:56 -08:00
14312d833e
zstdmt : fix : loading prefix from previous segments
...
There used to be a (very small) chance that
loading prefix from previous segment
would be confused with a real zstd dictionary.
For that to happen, the prefix needs to start
with the same value as dictionary magic.
That's 1 chance in 4 billions if all values have equal probability.
But in fact, since some values are more common (0x00000000 for example)
others are less common, and dictionary magic was selected to be one of them,
so probabilities are likely even lower.
Anyway, this risk is no down to zero
by adding a new CCtx parameter : ZSTD_p_forceRawDict
Current parameter policy : the parameter "stick" to its CCtx,
so any dictionary loading after ZSTD_p_forceRawDict is set
will be loaded in "raw" ("content only") mode,
even if CCtx is re-used multiple times with multiple different dictionary.
It's up to the user to reset this value differently if it needs so.
2017-02-23 23:42:12 -08:00
831b4890ce
minor tests/Makefile refactoring
...
and update of zstd_manual,html
2017-02-23 23:09:10 -08:00
64417cd2ff
Describe ambiguity around skippable frames
2017-02-22 13:29:01 -08:00
9757cc811b
Update comment
2017-02-22 12:28:21 -08:00
9050e1925e
Change name to to findFrameCompressedSize and add skippable support
2017-02-22 12:12:34 -08:00
517577bf53
spelling fixes in comments
...
i.e. occurred labeled Huffman
2017-02-20 12:08:59 -08:00
58af614ef2
push version and NEWS to v1.1.4
2017-02-13 18:32:44 -08:00
d7bfcac18a
Expose frameSrcSize to experimental API
2017-02-10 11:55:44 -08:00
269b2cd3d8
Documentation updates
2017-02-09 13:25:30 -08:00
Yann Collet
Nick Terrell
Yann Collet
Stella Lau
Paul Cruz
Yann Collet
Sean Purcell
Anders Oleson