3128e03be6
updated license header
...
to clarify dual-license meaning as "or"
2017-09-08 00:09:23 -07:00
376f435914
[dictBuilder] Set default compression level to 3
2017-08-24 16:21:05 -07:00
20f715d709
Fix displayLevel overflow
2017-08-23 15:56:15 +05:00
bd9c8ca146
Merge pull request #811 from terrelln/segmentSize
...
[cover] Fix end condition for small dictionary
2017-08-22 14:36:30 -07:00
29c2d9a4d0
[cover] Turn down notification for ZDICT subroutines
2017-08-21 14:28:31 -07:00
98de3f6847
[cover] Add dictionary size to compressed size
2017-08-21 14:23:17 -07:00
9a54a315aa
[cover] Convert score to U32 and check for zero
2017-08-21 13:30:07 -07:00
d49eb40c03
[cover] Stop when segmentSize is less than d
2017-08-21 13:10:03 -07:00
f306d400c0
[cover] Fix divide by zero
2017-08-21 11:12:11 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
b71363b967
check pthread_*_init() success condition
2017-07-19 01:05:40 -07:00
2bd6440be0
pinned down error code enum values
...
Note : all error codes are changed by this new version,
but it's expected to be the last change for existing codes.
Codes are now grouped by category, and receive a manually attributed value.
The objective is to guarantee that
error code values will not change in the future
when introducing new codes.
Intentionnal empty spaces and ranges are defined
in order to keep room for potential new codes.
2017-07-13 17:12:16 -07:00
590937df20
Merge pull request #739 from facebook/refPrefix
...
ZSTD_refPrefix
2017-06-29 04:36:03 -07:00
7d3816183f
exposed ZSTD_MAGIC_DICTIONARY in zstd.h
...
makes it easier to explain ZSTD_dictMode
2017-06-27 13:50:34 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
ee970398b2
Merge branch 'dev' into advancedAPI2
2017-05-22 12:33:56 -07:00
a1280406b0
[libzstd] Allow users to define custom visibility
2017-05-19 18:01:59 -07:00
fa3671eac7
changed ZSTD_BLOCKSIZE_ABSOLUTEMAX into ZSTD_BLOCKSIZE_MAX
...
Also :
change ZSTD_getBlockSizeMax() into ZSTD_getBlockSize()
created ZSTD_BLOCKSIZELOG_MAX
2017-05-19 10:51:30 -07:00
f376d47c11
[CLI] Switch dictionary builder on CLI to cover
2017-05-02 11:18:27 -07:00
020b960e13
[cover] Make optimization faster
2017-05-02 11:02:48 -07:00
f2d9ef1dc0
[cover] Optimize case where d <= 8
2017-05-02 11:02:43 -07:00
865918dd04
Fix typo in zdict.h
2017-05-02 11:02:37 -07:00
5152fb2cb2
Convert all tabs to spaces
2017-03-29 18:51:58 -07:00
4cf0093571
restored bonus rule
2017-03-26 14:51:00 -07:00
69017bf253
Merge branch 'dev' into LegacyDictBuilder
2017-03-26 14:39:13 -07:00
582760818f
minor refactor
...
add const
changed if for easier to add new conditions
2017-03-26 03:04:56 -07:00
858f72eeb8
fixed dictBuilder issue
...
dictionary loading would fail during entropy analysis
2017-03-26 02:50:00 -07:00
ecee9f2ef8
fixed conversion warnings
2017-03-26 00:59:14 -07:00
4c41d37fcc
changed test for new syntax
...
--dictID= and --maxdict=
2017-03-24 18:36:56 -07:00
d41f707e88
minor improvement : remove duplicates with 1 char prefix difference
2017-03-24 17:56:45 -07:00
96aa3019b2
changed advanced commands --maxdict= and --dictID=
...
now works with the `=` variant, which is the recommended one.
Old variant `--dictID #` still works, for compatibility with existing scripts.
Long term objective is to remove the old variant..
2017-03-24 16:04:29 -07:00
9da3b215ec
Ensure all limits derived from same constants
...
Now uses ZDICT_DICTSIZE_MIN and ZDICT_CONTENTSIZE_MIN
from zdict.h.
Also : reduced values to 256 and 128 respectively
2017-03-24 15:02:09 -07:00
f332ece468
dictBuilder fails to create dictionary on certain input
...
Properly expressed with an error code (see zstd_errors.h)
and a cli return code != 0
2017-03-23 16:24:02 -07:00
042ba122ae
Change g_displayLevel to int and fix DISPLAYUPDATE flush
2017-03-23 11:21:59 -07:00
976e325b2e
Fix COVER_optimizeTrainFromBuffer() resource leaks
...
Thanks to @nemequ for reporting the resource leaks.
2017-03-02 15:54:39 -08:00
545987996a
Fix deprecation warnings for clang with C++14
2017-02-08 17:38:17 -08:00
71c5263c00
Attribute cover dictionary code
2017-02-07 11:35:07 -08:00
43474313f8
Fix documentation about memory usage
2017-01-27 18:43:05 -08:00
2fe9126591
Add multithread support to COVER
2017-01-27 11:56:02 -08:00
8d984699db
Document memory requirements for COVER algorithm
2017-01-09 18:20:10 -08:00
555e281637
Handle large input size in 32-bit mode correctly
2017-01-09 18:20:06 -08:00
3a1fefcf00
Simplify COVER parameters
2017-01-02 17:51:38 -08:00
96b39f65fa
Add COVER dictionary builder
2017-01-02 13:22:51 -08:00
aca113f4f5
fixed ZSTD_sizeof_?Dict()
2016-12-23 22:25:03 +01:00
1b5d4a7d53
ZDICT_finalizeDictionary() flipped comparison
2016-12-22 18:14:57 -08:00
bcbe77e994
ZDICT_finalizeDictionary() flipped comparison
...
`ZDICT_finalizeDictionary()` had a flipped comparison.
I also allowed `dictBufferCapacity == dictContentSize`.
It might be the case that the user wants to fill the dictionary
completely up, and then let zstd take exactly the space it needs
for the entropy tables.
2016-12-22 18:01:14 -08:00
78a0072d5a
Fix failing test due to deprecation warning
2016-12-22 17:36:16 -08:00
d76d1a9ef0
added ZDICT_finalizeDictionary()
2016-12-22 20:18:43 +01:00
0819abe3c1
added ZSTD_createDDict_byReference() body
2016-12-21 19:25:15 +01:00
1496c3dc47
Fix : size estimation when some samples are very large
2016-12-18 11:58:23 +01:00
d46ecb58a5
added dll compilation tests
2016-12-17 16:28:12 +01:00
8de46ab51a
Export all API functions
2016-12-16 13:27:30 -08:00
0a5a5fb7fd
Fix #418 : printing selected segments in zdict debug mode can segfault with certain pathological patterns
2016-11-02 13:57:55 -07:00
52c1bf93fe
improved dicitonary segment merge
2016-10-18 16:34:58 -07:00
2b361cf2f1
minor opt
2016-10-14 16:09:07 -07:00
df6797447f
update dictionary builder warning comments
2016-09-27 15:14:32 +02:00
47094ea66b
added comment on filePos
2016-09-26 18:03:33 +02:00
97b378a6f8
Streaming : dictionary compression on multiple files / segments can correctly provide srcSize into header (when provided) using pledgedSrcSize.
2016-09-21 17:20:19 +02:00
d56dbc02d3
removed g_displayLevel
2016-09-02 17:28:41 -07:00
855766d73d
clarified dictionary in format description
2016-09-02 17:04:49 -07:00
d725427a3c
g_time => local displayTime
2016-09-02 15:32:39 -07:00
4ded9e591c
added boilerplate
2016-08-30 11:06:28 -07:00
3b15f1f10f
minor refactor
2016-08-30 09:58:50 -07:00
87c18b2ebd
fixed multiple minor warnings for XCode
2016-08-26 01:43:47 +02:00
da3fbcb302
Added ZDICT_getDictID()
2016-08-19 14:23:58 +02:00
a5dbf9f629
Merge pull request #297 from borzunov/dev
...
Export functions related to dictionary compression from DLL
2016-08-18 15:05:01 +02:00
49d105cfcf
better warning and error messages in case of dictionary training failure ( #292 )
2016-08-18 15:02:11 +02:00
0f6f17a14f
Rename ZSTDLIB_API to ZDICTLIB_API in zdict.h
2016-08-18 16:47:06 +05:00
1f48382b1a
Export functions related to dictionary compression from DLL
2016-08-18 16:12:49 +05:00
e9b414d825
fixed msan warning ( #281 )
2016-08-11 22:09:09 +02:00
e0b4a2d40f
fixed dictionary generation, reported by Bartosz Taudul
2016-08-03 03:36:03 +02:00
bf2bc112bb
bench : controlled display update when loading lot of files
2016-08-02 23:48:13 +02:00
f116e87f59
fixed analyzer warning
2016-08-01 19:15:18 +02:00
3ca750372d
updated doc ( #269 )
2016-08-01 02:26:20 +02:00
70a9ff4af3
fixed too large selectivity level, reported by Ilona Papava
2016-07-30 01:09:14 +02:00
c0ce4f1211
slightly improved compression speed
2016-07-30 00:55:13 +02:00
ed57d8530a
new seqStore
2016-07-29 21:22:17 +02:00
d469a98c01
Clarified API comments, from suggestions by Bryan O'Sullivan
2016-07-28 04:55:03 +02:00
4110534886
ZSTD_maxCLevel() is promoted to "stable" API ( #254 , by @FrancescAlted)
2016-07-27 15:09:11 +02:00
55a8bea0b5
fixed dictionary generation
2016-07-27 14:48:47 +02:00
c154d9d6a2
better support for large dictionaries (> 128 KB)
2016-07-27 14:37:00 +02:00
07626dfa51
improved dictbuilder notifications on selectivity
2016-07-27 13:28:46 +02:00
f796f7ab45
removed fastscan mode
2016-07-27 12:53:54 +02:00
dd25a27702
added tutorial warning messages for dictBuilder
2016-07-27 12:43:09 +02:00
cf05b9d477
ZSTD_getBlockSizeMax()
2016-07-18 16:52:10 +02:00
e557fd5e92
minor compression level corrections
2016-07-17 16:21:37 +02:00
f6ff53cd4e
implemented dictID reserved ranges
2016-07-15 17:03:38 +02:00
961b6a0e34
ZSTD_compressBlock() limits block size depending on windowLog parameter
2016-07-15 11:58:49 +02:00
e9ed5cdc94
fixed minor coverity warning
2016-07-14 21:02:57 +02:00
17508f1a16
fixed a few minor coverity warnings
2016-07-14 17:18:20 +02:00
69c2cdb45c
fixed conversion warning
2016-07-14 16:52:45 +02:00
5e80dd3261
fixed minor coverity warnings
2016-07-13 19:21:57 +02:00
3c174f4da9
fixed minor coverity warning
2016-07-13 17:25:53 +02:00
fbc69f8649
changed for #245
2016-07-13 13:52:58 +02:00
29652e2618
sample set limitation closer to 2 GB
2016-07-06 16:25:46 +02:00
99b045b70a
dictBuilder protection vs huge sample sets (>2 GB)
2016-07-06 16:12:38 +02:00
445d49d898
fixed conversion warning
2016-07-06 13:27:22 +02:00
a295b3170f
fixed conversion warning
2016-07-06 13:13:12 +02:00
517e1ba623
fixed dictBuilder issue with HC levels. Reported by Bartosz Taudul.
2016-07-06 12:35:09 +02:00
c093208ab8
fix : potential leak ( #229 )
2016-06-30 14:07:30 +02:00
23042929da
Fixed : dictBuilder fails if first sample is too small
2016-06-22 11:05:34 +02:00
ad39b7a718
zdict stores standard rep-offset.
...
It can use custom ones, but the proper formula and impact on statistics is not done yet.
2016-06-16 01:14:41 +02:00
52a0622beb
RepsCodes are saved into Dict
...
(uncomplete : need decompression to regenerate them)
2016-06-16 01:05:04 +02:00
130fe11394
merged `huf_static.h` into `huf.h` . Requires `HUF_STATIC_LINKING_ONLY` macro.
2016-06-05 00:42:28 +02:00
49bb0041af
removed `ZSTD_highbit()` from `zstd_internal.h`, as it is only used by `zstd_compress.c`
2016-06-04 20:17:38 +02:00
e69b8ccceb
merged `zdict_static.h` into `zdict.h`. Now requires `ZDICT_STATIC_LINKING_ONLY` macro.
2016-06-04 18:56:23 +02:00
370b751e24
Expose function to add entropy tables to pre-built dictionary.
...
In some cases a custom dictionary building algorithm tailored for a specific
input can be more effective than the one produced by `ZDICT_trainFromBuffer`,
but with the current API it's not possible encode the entropy tables into the
custom-built dictionary.
This commit extracts the logic to add entropy tables to a dictionary from
`ZDICT_trainFromBuffer` and exposes it as a function
`ZDICT_addEntropyTablesFromBuffer`.
2016-05-30 19:50:09 -07:00
290aaa7521
Added : ability to manually select the dictionary ID of a newly created dictionary
2016-05-30 21:18:52 +02:00
c46fb924df
added dictionary ID (incomplete)
2016-05-29 05:01:04 +02:00
d5ff2c3d9a
ordering of #include
2016-04-28 14:40:45 +02:00
69fcd7c0ae
getFileSize moved to common/util.h
2016-04-28 12:23:33 +02:00
23a0889301
separation of lib/ into common/, compress/, decompress/, dictBuilder/, legacy/
2016-04-22 12:43:18 +02:00
Yann Collet
Nick Terrell
Dmitriy Titarenko
Yann Collet
Sean Purcell
Yann Collet
Alexander Borzunov
Giuseppe Ottaviano
inikep