89dc856cae
[pool] Fix formatting
2017-08-24 16:48:32 -07:00
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
376f435914
[dictBuilder] Set default compression level to 3
2017-08-24 16:21:05 -07:00
2fbf0285b2
Fix interaction with ZSTD_setCCtxParameter() and cleanup
2017-08-24 11:25:41 -07:00
fd9bf42516
Fix forceWindow and dictMode setting for zstdmt jobs
2017-08-23 19:16:57 -07:00
bf3108fb50
Ensure zstdmt uses 'job version' of cctx parameters
2017-08-23 17:03:31 -07:00
1c81f725ff
Remove duplicated testing code
2017-08-23 15:47:15 -07:00
64ce49426b
Fix cstream compression level
2017-08-23 12:30:47 -07:00
5bc2c1e982
Add prototype support for customMem with cctxParams
2017-08-23 12:03:30 -07:00
e9ce1208a1
Merge pull request #812 from facebook/longRangeFix
...
fixed extraordinary scenario where all fields use maximum nbBits
2017-08-23 11:35:28 -07:00
74cde5a4d8
Merge pull request #813 from stellamplau/highbit32fix
...
Fix undefined behavior when srcSize==1
2017-08-23 11:31:06 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
20f715d709
Fix displayLevel overflow
2017-08-23 15:56:15 +05:00
11303778d0
Add function to make cctxParams from ZSTD_parameters
2017-08-22 14:53:13 -07:00
bd9c8ca146
Merge pull request #811 from terrelln/segmentSize
...
[cover] Fix end condition for small dictionary
2017-08-22 14:36:30 -07:00
23fc0e41fa
Remove 'opaque' naming from internal functions
2017-08-22 14:24:47 -07:00
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
6b2b6a9bd5
fixed extraordinary scenario where all fields use maximum possible nb of bits simultaneously
...
can only happen if windowLog>=27 (level 22 --ultra)
2017-08-22 12:09:21 -07:00
e50ed1fa3a
Fix undefined behavior when srcSize==1
2017-08-22 11:55:42 -07:00
60e1bc617c
Explicitly create a job cctxParam for multithreading
2017-08-21 15:39:37 -07:00
5b956f4753
Comment out CCtx_param versions of CDict functions
2017-08-21 14:49:16 -07:00
29c2d9a4d0
[cover] Turn down notification for ZDICT subroutines
2017-08-21 14:28:31 -07:00
98de3f6847
[cover] Add dictionary size to compressed size
2017-08-21 14:23:17 -07:00
78c3d16bf4
Merge pull request #809 from terrelln/dev
...
[cover] Fix divide by zero
2017-08-21 13:33:19 -07:00
9a54a315aa
[cover] Convert score to U32 and check for zero
2017-08-21 13:30:07 -07:00
fd8a25786e
Check parameters are valid in initCCtxParams
2017-08-21 13:23:35 -07:00
1c0dbe81b1
Add documentation for CCtx_params
2017-08-21 13:18:00 -07:00
d49eb40c03
[cover] Stop when segmentSize is less than d
2017-08-21 13:10:03 -07:00
939f954285
Pass ZSTD_CCtx_params as const ptr when possible
2017-08-21 12:57:18 -07:00
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
560b34f6d2
Return error code when initializing NULL cctxParams
2017-08-21 11:52:26 -07:00
25be09c6b4
Set some parameters to zero before initializing cdict
2017-08-21 11:35:46 -07:00
232d62b637
fixed a few headers that were too hastily copy/pasted during last license change
2017-08-21 11:24:32 -07:00
f306d400c0
[cover] Fix divide by zero
2017-08-21 11:12:11 -07:00
502031ca10
Use cctxParam version of createCDict internally
2017-08-21 11:00:44 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
7db552676e
reduced pool queue to 0 to save memory
...
fixed : pool performance when jobs are fires fast and queueSize==0
2017-08-19 15:07:54 -07:00
6cee6e07e5
Add internal createCDict function
2017-08-18 22:48:31 -07:00
d775519296
Add cctxParam versions of internal functions
2017-08-18 17:37:58 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
63b8c98531
Pass cctx parameters to MTCtx
2017-08-18 16:17:24 -07:00
399ae013d4
Add function to apply cctx params
2017-08-18 13:01:55 -07:00
81d89d82a6
Move nbThreads to cctx params
2017-08-18 12:08:57 -07:00
2300c58a6f
Move dictContentByRef to cctx params
2017-08-18 12:03:16 -07:00
b6cb2ed8cb
Move dictMode to cctxParams
2017-08-18 11:43:31 -07:00
97e27affcb
Move compression level to cctx params
2017-08-18 11:20:08 -07:00
c0221124d5
Add function to set opaque parameters
2017-08-17 19:30:22 -07:00
4169f49171
Add initialization/allocation functions for opaque params
2017-08-17 18:45:04 -07:00
ade95b8bed
Add opaque interfaces for static initialization
2017-08-17 18:13:08 -07:00
699f11b4f7
Create opaque parameter structure
2017-08-17 17:33:46 -07:00
f9e6590715
Merge pull request #796 from terrelln/is-error
...
[FSE][HUF] Inline error checks
2017-08-15 12:37:28 -07:00
2dbcfc6994
Merge pull request #794 from terrelln/force-inline
...
[libzstd] Fix FORCE_INLINE macro
2017-08-15 12:03:44 -07:00
07c6ff588e
[FSE][HUF] Inline error checks
...
Caught by Clang's optimization remarks.
2017-08-15 11:23:28 -07:00
565e925eb7
[libzstd] Fix FORCE_INLINE macro
2017-08-14 21:12:05 -07:00
b9d4f4fb74
Fix ZSTD_estimateDStreamSize function after ZSTD_DStream and ZSTD_DCtx were merged
2017-08-13 13:29:42 +03:00
9ba97182d1
[CI] Add gcc7build test
2017-08-08 13:28:56 -07:00
d9f2893eb9
Merge pull request #782 from terrelln/dstSizeTooSmall
...
Fix compression failure on incompressible data
2017-08-07 14:52:02 -07:00
8049556928
Merge pull request #778 from terrelln/bad-huff
...
[libzstd] Fix bug in Huffman decompresser
2017-08-07 14:05:58 -07:00
abe12b3399
[libzstd] Fix bug in Huffman decompresser
...
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.
* Make the decompresser accept Huffman compressed literals whose size
increases.
* Add a test case that exposes the bug. The compressed file has to be
statically generated, since the compressor won't normally produce files
that expose the bug.
2017-08-07 12:37:48 -07:00
308047eb5d
Fix compression failure on incompressible data
...
If the destination buffer is the minimum allowed size in
`ZSTD_compressSequences()` (2^17), then if the block isn't compressible
compression might fail with `dstSize_tooSmall`, when it should instead emit
a raw uncompressed block.
Additionally, `ZSTD_compressLiterals()` implicitly called
`ZSTD_noCompressLiterals()` if Huffman compression failed. Make that
explicit.
2017-08-07 11:45:24 -07:00
73ba58955f
Signal after finishing job when queueSize=0
2017-08-01 20:12:06 -07:00
1d76da1d87
Replace marker with queueEmpty variable and update pool.h comment
2017-08-01 12:30:16 -07:00
5adceeed01
Allow queueSize=0 in pool.c and update poolTests
2017-07-31 10:10:16 -07:00
e1222544be
Merge pull request #753 from paulcruz74/adapt-approach-3
...
adaptive compression v1
2017-07-27 10:00:10 -07:00
ae20d413da
[libzstd] Fix CHECK_V_F macros
2017-07-25 12:52:01 -07:00
a90b16e150
Visual blind fix 2
2017-07-20 15:57:55 -07:00
b4d460f32c
pool.c : blindfix for Visual warnings
2017-07-20 01:13:14 -07:00
3974d2b38a
blind fix for Windows Multithreading module
...
adds a fake 0 return value for mutex/cond init
2017-07-19 13:33:21 -07:00
6945b3c43d
removed previous version of completion for compression
2017-07-19 11:51:50 -07:00
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
77d67fb167
Merge pull request #766 from terrelln/real-block-split
...
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-18 08:26:24 -07:00
14c83b05c7
Merge pull request #765 from terrelln/real-block-split
...
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 19:25:55 -07:00
7a28b9e4a3
[libzstd] Pull optimal parser state out of seqStore_t
2017-07-17 15:29:11 -07:00
3381bf4b84
Merge pull request #764 from terrelln/real-block-split
...
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 14:46:01 -07:00
e198230645
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
2017-07-17 12:27:24 -07:00
634f012420
[libzstd] Refactor ZSTD_compressSequences()
2017-07-17 11:36:11 -07:00
50ce4eaeb6
added error detection for pthread initialization, added compression completion measurement, fixed const values
2017-07-17 10:12:44 -07:00
3b0cff3c33
fixed clang's -Wdocumentation
2017-07-13 18:58:30 -07:00
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
830ef4152a
[libzstd] Increase granularity of FSECTable repeat mode
2017-07-13 12:45:39 -07:00
d985319337
Merge pull request #759 from terrelln/real-block-split
...
[libzstd] Pull CTables into sub-structure
2017-07-13 10:24:19 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
132e6efd76
switched ZSTDMT_compress_advanced() last argument to overlapLog
...
overlapRLog (== 9 - overlapLog) was a bit "strange"
as all other public entry points use overlapLog
2017-07-13 02:22:58 -07:00
4e77f7761d
clarified comment on ZSTD_p_contentSizeFlag
2017-07-13 02:09:07 -07:00
de0414b736
[libzstd] Pull CTables into sub-structure
2017-07-12 19:49:19 -07:00
8ef666c325
slightly increased buffer pool, to cover normal "full load" scenarios
...
2 buffers per active worker
+ 1 buffer for input loading
+ 1 buffer for "next input" when submitting current one
+ 1 buffer stuck in queue
2017-07-12 14:23:34 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
2a62f48bf4
release input buffers from inside worker thread
...
buffers are released sooner, which makes them available faster for next job.
=> decreases total nb of buffers necessary
2017-07-11 15:56:40 -07:00
57236184af
buffer pool : all buffers have same size
...
to reduce memory fragmentation.
They can be used for in or out, interchangeably.
2017-07-11 15:17:25 -07:00
34b2b95631
zstdmt : intermediate outBuffer allocated from within worker
...
reduces total amount of memory needed,
since jobs in queue do not have an outBuffer pre-reserved now
2017-07-11 14:59:10 -07:00
16261e6951
buffer pool can be invoked from multiple threads
2017-07-11 14:14:07 -07:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
40156a4967
bumped version nb to v1.3.1
2017-07-08 04:55:09 -07:00
0f4fc6c20a
fixed several conversion warnings
2017-07-07 17:13:12 -07:00
9bde061a0b
fixed minor Visual compilation limitation
2017-07-07 16:14:17 -07:00
593d517ebf
fixed minor cast warning
2017-07-07 16:09:47 -07:00
ead4dd48f6
new field frameHeader.headerSize
2017-07-07 15:51:24 -07:00
46396523c0
ZSTD_getFrameHeader : control of windowSize limits is delegated to caller
...
Extracting frame header is a separate operation.
It's now possible to get frame header, whatever the window size set in it.
2017-07-07 15:32:12 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
e622330a3b
extended frameHeader.windowSize to unsigned long long
2017-07-07 14:19:01 -07:00
f04deff4fc
fixed #718 , reported by @GregSlazinski, solution suggested by @mcmilk
2017-07-06 01:42:46 -07:00
d75c0e71c4
minor code refactoring
2017-07-05 18:10:07 -07:00
49af41820d
clarified status of zstdmt_compress.h API
2017-07-05 17:21:13 -07:00
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
6383372dec
fixed : 0-copy in NULL is UB
2017-07-04 10:36:41 -07:00
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
2de2396a36
refactor ZSTDMT_compress()
2017-07-03 16:23:36 -07:00
2084b041f4
fixed comments
2017-07-03 15:52:19 -07:00
5a77361595
fixed wrong function name in comment
2017-07-03 15:21:24 -07:00
c80fc50a8d
[libzstd] Fix memcpy() on potential NULL source
...
* `ZSTD_decompressStream_generic()` `ip` may be `NULL` for one of the calls
to `memcpy()`
* Assert the source is not `NULL` for calls to `memcpy()` where I believe
the source should not be `NULL`.
2017-07-03 12:31:55 -07:00
2485f88bf8
fixed legacy version init bug
2017-07-01 09:09:34 -07:00
21fdf97e00
Merge branch 'dev' into fullbench
2017-07-01 07:01:08 -07:00
1bafe393e4
fix : ZSTDMT_compressStream_generic() can accept NULL input
...
also : converge implementations towards new version of ZSTDMT_compressStream_generic()
2017-07-01 06:59:24 -07:00
58bd0e70fc
fixed : dictionary compression with new advanced API in Multi-threading mode
2017-06-30 16:01:02 -07:00
d8b33a598d
Optimized ZSTDMT single-pass mode speed on large sources
...
by ensuring job sizes remain "not too large"
2017-06-30 15:44:57 -07:00
d5c046c609
implemented shortcut for zstd_compress_generic() in MT mode
...
added ZSTDMT_compress_advanced() API
2017-06-30 14:51:01 -07:00
7f40bb1c39
Merge pull request #742 from stellamplau/stack-space
...
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
2017-06-30 14:50:23 -07:00
32df49e9f8
Fix typo
2017-06-30 12:56:24 -07:00
b0513b519c
Add comment to HUF_DECOMPRESS_WORKSPACE_SIZE
2017-06-30 12:53:56 -07:00
4c71f59c77
Clarify typedef of rankVal_t and rankValCol_t
2017-06-30 09:52:20 -07:00
28f711ef95
Rename ALIGN and ALIGN_MASK to HUF_ALIGN and HUF_ALIGN_MASK
2017-06-30 09:38:11 -07:00
70ad6829e7
Delegate HUF_decompress4X_hufOnly to workspace version
2017-06-29 16:22:32 -07:00
104c4d57c1
Fix bitshift error
2017-06-29 15:40:49 -07:00
a3d9926c40
compression optimization opportunity
...
switch to single-pass mode directly into output buffer
when outputSize >= ZSTD_compressBound(inputSize).
Speed gains observed with fullbench (~+15% on level 1)
2017-06-29 14:44:49 -07:00
fedc94de8c
Fix pointer casting warning
2017-06-29 13:04:15 -07:00
c6a5275a28
Fix alignment warnings with pointer casting
2017-06-29 12:39:34 -07:00
99e315999c
Reduce stack usage of HUF_readDTableX4 and HUF_readDTableX2
2017-06-29 11:49:59 -07:00
97f2bf66da
minor : fix typo
2017-06-29 11:31:40 -07:00
acbef3decd
ZSTD_getFrameContentSize() is promoted to "stable" status
2017-06-29 05:19:51 -07:00
590937df20
Merge pull request #739 from facebook/refPrefix
...
ZSTD_refPrefix
2017-06-29 04:36:03 -07:00
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
037466245f
refactor ZSTD_check_compressionLevel_monotonicIncrease_memoryBudget()
...
use less macro statements
the initial version was meant to work with STATIC_ASSERT
but since it doesn't work and needs assert()
it's possible to rewrite it using normally compiled code
which is better for compiler.
Downside : the error message is less precise.
There is a DEBUGLOG(3,) to compensate.
2017-06-28 20:24:08 -07:00
2bf428df45
Merge branch 'advancedAPI2' into refPrefix
2017-06-28 16:35:49 -07:00
1ca76039af
fixed -Wdeclaration-after-statement
2017-06-28 15:40:21 -07:00
813535105b
added function to control monotonic memory budget increase of ZSTD_defaultCParameters[0]
...
It's a runtime test, based on assert(),
played once, on first ZSTD_getCParams() usage,
when ZSTD_DEBUG is enabled.
2017-06-28 15:34:56 -07:00
adbe74a8ac
adjusted compression levels to guarantee a monotonically increasing memory budget
2017-06-28 13:22:37 -07:00
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
2e4274262d
controlled dictMode
2017-06-27 17:09:12 -07:00
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
c7fb884eea
fixed minor conversion warning
2017-06-26 18:02:23 -07:00
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
ecb0f46866
add controls over streaming buffers
2017-06-21 17:25:01 -07:00
dce789281b
fixed : decompression of skippable frames in streaming mode
2017-06-21 15:53:42 -07:00
204b6b7ef6
fixed streaming buffered allocation with CDict compression
2017-06-21 15:13:00 -07:00
1e4129b27b
fixed dangling pointer risk, detected by @terrelln
2017-06-21 13:26:10 -07:00
83095970e6
free cdictLocal faster, suggested by @terrelln
2017-06-21 12:26:40 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
9c56b12938
Merge pull request #723 from paulcruz74/dev
...
Adding zstd -l
2017-06-21 09:41:55 -07:00
e51d51bdf7
fixed memcpy() overlap
2017-06-20 17:44:55 -07:00
466f92eaa6
removed one useless streaming compression stage, detected by @terrelln
2017-06-20 16:25:29 -07:00
c3bce24ef4
fixed potential dangling pointer, detected by @terrelln
2017-06-20 16:09:11 -07:00
78b8234554
fixed comments, following suggestion by @terrelln
2017-06-20 14:26:48 -07:00
b44ab82f7a
ensure new ZSTD_strategy starts at value 1
2017-06-20 14:11:49 -07:00
c08e649e95
first implementation of bench.c with new API ZSTD_compress_generic()
...
Doesn't speed optimize this buffer-to-buffer scenario yet.
Still internally defers to streaming implementation.
Also : fixed a long standing bug in ZSTDMT streaming API.
2017-06-19 18:25:35 -07:00
695a0a3449
fixed IA64 compilation error, by @mcmilk
2017-06-19 15:27:30 -07:00
fe234bf48b
fix attempts : fullbench for VS2008
2017-06-19 15:23:19 -07:00
55f9cd4942
[libzstd] Fix UBSAN failure
2017-06-19 15:12:28 -07:00
bf99150be3
update new api presentation in zstd.h and manual
2017-06-19 12:56:25 -07:00
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
92bafda406
INSTALL_DATA instead of INSTALL_LIB for libzstd.a
...
INSTALL_LIB can be passed -s flag to strip symbols. Static libraries should not be stripped, only dynamic ones should be stripped.
2017-06-17 00:23:41 -07:00
381e66cfbd
added ZSTD_clampCParams()
...
now ZSTD_adjustCParams() is always successful,
it always produces a valid CParams
2017-06-16 17:34:54 -07:00
aee916e37c
fixed +/-1 error for pledgedSrcSizePlusOne
2017-06-16 17:02:35 -07:00
d3de3d51a3
fix attempt 2 : Visual sign conversion warning
2017-06-16 16:51:33 -07:00
944be54774
fixed attempt : minor Visual sign conversion warning
2017-06-16 14:05:01 -07:00
b26728c9c8
added ZSTD_startNewCompression()
2017-06-16 14:00:46 -07:00
a0ba849fe6
changed frameContentSize field to pledgedSrcSizePlusOne
...
pledgedSrcSize is proper : it's a promise, not yet fulfilled.
It will be controlled at the end.
PlusOne is meant to have 0 (default) == unknown
2017-06-16 13:29:17 -07:00
2cf7755da7
fix : pledgedSrcSize correctly reset to unknown in "continue" mode
2017-06-16 12:34:41 -07:00
9e73f2f320
fix : correctly reset pledgedSrcSize to unknown status
...
when starting a new compression with an existing context
2017-06-16 12:24:01 -07:00
33873f0e74
fixed : new advanced AIP : setting nbThreads to the same value > 1
2017-06-16 12:04:21 -07:00
559ee82e90
fixed : calling ZSTD_compress_generic() to end-flush a stream in multiple steps
2017-06-16 11:58:21 -07:00
bd18c885a3
added ZSTD_CCtx_reset
2017-06-16 10:17:50 -07:00
cc9f9b7f4c
protection : ZSTD_CONTENTSIZE_UNKNOWN automatically disables contentSizeFlag
2017-06-15 18:17:34 -07:00
05ae4b2190
added protection : MT incompatible with Static allocation
2017-06-15 18:03:34 -07:00
a9b77c83e5
cleaning up code for analyzing frames
2017-06-15 14:13:28 -07:00
f129fd3970
disabled MT code path when ZSTD_MULTITHREAD is not defined
2017-06-11 18:46:09 -07:00
23aace9778
added control stage to MT mode
2017-06-11 18:32:36 -07:00
f35e2de61c
linked newAPI to ZSTDMT
2017-06-05 18:32:48 -07:00
c59162e053
minor fix for -Wdocumentation
2017-06-05 00:12:13 -07:00
8bcbf42617
fixed g++ prototype mismatch
2017-06-04 23:52:00 -07:00
8c910d2097
updated ZSTDMT streaming API
...
ZSTDMT streaming API is now similar
and has same capabilites as single-thread streaming API.
It makes it easier to blend them together.
2017-06-03 01:15:02 -07:00
58e8d793e1
made debug definitions common within zstd_internal.h
2017-06-02 18:20:48 -07:00
8ddf4c22d5
fixed missing initialization
2017-06-02 17:16:49 -07:00
33a7e679e5
significant zlib wrapper code refactoring
...
code indentation
variable scope and names
constify
Only coding style changes.
The logic should remain the same.
2017-06-02 17:10:49 -07:00
4effccbf56
zlib_wrapper's uncompress() uses ZSTD_isFrame() for routing
...
more generic and safer than using own routing for magic number comparison
2017-06-02 14:27:11 -07:00
Nick Terrell
Stella Lau
Yann Collet
Dmitriy Titarenko
Yann Collet
Roman Gershman
Paul Cruz
cyan4973
Yuri