d195eec97e
fixed msan error
...
cold dictionary is detected through a comparison with dictEnd,
which was not initialized at the beginning of first DCtx usage.
2018-09-13 12:29:52 -07:00
674dd21bd0
final parameter tuning
2018-09-12 17:25:34 -07:00
419dfd4ea3
clean traces
2018-09-12 16:40:28 -07:00
2618253da2
fixed PREFETCH() macro
...
for corner cases and platforms without this instruction
2018-09-12 16:15:37 -07:00
44d3b83bb1
conditional dict content prefetching
...
based on nbSeq.
2018-09-12 15:35:21 -07:00
5fb5ed3b31
adjust heuristic decisions
2018-09-12 12:32:09 -07:00
c49ccbc8e7
largeNbDicts : can select a nb of blocks
...
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
2018-09-12 11:31:28 -07:00
4de344d505
added conditional prefetch
...
depending on amount of work to do.
2018-09-12 10:29:47 -07:00
63a519dbf6
implemented first prefetch
...
based on dictID.
dictContent is prefetched up to 32 KB
(no contentSize adaptation)
2018-09-11 17:23:44 -07:00
51a246da82
Merge pull request #1310 from facebook/Dworkspace
...
reduce DDict size by 2KB
2018-09-10 13:11:11 -07:00
3675ef4762
added comment about minimum size of FSE tables
...
required for DDict creation,
which use this space as workspace during Hufman table building stage.
2018-09-10 11:24:17 -07:00
7ff74c89fa
Merge pull request #1307 from facebook/spec
...
spec clarification
2018-09-07 15:42:04 -07:00
112c18d6d5
Merge pull request #1309 from facebook/islink
...
fixed UTIL_isLink() for Mac OS-X
2018-09-06 18:01:10 -07:00
f97ca36eab
strengthened conditions for using workplace into fse table space
...
ensure that the structure layout is as expected.
will trigger an error if it changes in the future.
Another solution would be to use a union,
this would be cleaner and get rid of these static asserts.
However, in order to keep the current code unmodified,
it would be necessary to use an un-named unions.
And apparently, un-named unions are only possible on "recent" compilers (C99+).
2018-09-06 17:54:13 -07:00
b036b9a177
Merge branch 'islink' into Dworkspace
2018-09-06 17:33:27 -07:00
deb03d8a26
fixed UTIL_isLink() for Mac OS-X
2018-09-06 17:32:16 -07:00
87406548f0
reduced DDict size, by -2KB
...
corresponding to the removal of workspace
which is needed while building huffman table
and is now either present in DCtx,
or temporarily borrowed from available FSE table space.
2018-09-06 17:07:53 -07:00
79106d8de9
Merge pull request #1308 from facebook/codeofconduct
...
added code of conduct
2018-09-06 14:38:32 -07:00
161ecddf3c
added code of conduct
...
as requested by Facebook OSS program.
2018-09-06 11:20:39 -07:00
50b216146f
Merge pull request #1304 from facebook/largeNbDicts
...
contrib/largeNbDicts
2018-09-06 09:50:56 -07:00
c57a856d64
fixed minor static analyzer warning
2018-09-05 14:33:51 -07:00
55a8f84a2c
spec clarification
...
following #1305 comments from @ulikunitz
2018-09-05 12:31:33 -07:00
d492ef4e07
Merge pull request #1306 from jennifermliu/dev
...
Change default f of FASTCOVER to 20
2018-09-04 18:03:36 -07:00
21721b75a3
Change default f to 20
2018-09-04 17:15:14 -07:00
7a02df8dbe
Merge pull request #1302 from jennifermliu/splitpoint
...
Always use splitPoint=1.0 for non-optimize cover and fastcover and some minor changes on comments
2018-09-04 16:59:41 -07:00
1d487d587f
updated documentation
2018-09-04 14:57:45 -07:00
11b8b8c100
silenced false-positive scan-build warning
2018-08-31 10:01:06 -07:00
cc1f14a16b
Merge pull request #1303 from cmcginty/dev
...
Zstandard - better `make list`
2018-08-30 18:09:33 -07:00
986ead646b
Add optional description field support to "list" target
2018-08-30 17:00:17 -07:00
0ff67511e6
fixed link order for old compilers
2018-08-30 16:43:28 -07:00
f76253bb70
minor : createDictionaryBuffer() can create dictionaries of different sizes
2018-08-30 16:24:44 -07:00
39c55a118f
fixed minor compatibility issues with older compilers
2018-08-30 16:00:57 -07:00
f87383507d
Update comment about default dictionary builder
2018-08-30 15:46:39 -07:00
944c9986e0
Update comment on default steps of cover and fastcover
2018-08-30 15:37:29 -07:00
39ef91a599
-std=c99 for largeNbDicts
2018-08-30 14:59:23 -07:00
16db0337b1
Always use splitPoint=1.0 for non-optimize cover and fastcover
2018-08-30 14:59:22 -07:00
e45d82ab0b
Merge branch 'dev' of github.com:facebook/zstd into dev
2018-08-30 14:57:16 -07:00
4086b2871b
largeNbDicts compatible with multiple source files
...
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
2018-08-30 14:38:49 -07:00
a5a77965d3
make all includes contrib/largeNbDicts
2018-08-29 16:17:22 -07:00
d89fa814c1
added a README
...
for documentation
2018-08-28 18:19:19 -07:00
6444c50035
increases randomness of ddict ptrs
2018-08-28 18:13:46 -07:00
6c398df241
level, block size and nb dicts can be set on command line
2018-08-28 18:05:31 -07:00
31ebb26945
Merge pull request #1301 from terrelln/lit-size
...
[zstd] Fix seqStore growth
2018-08-28 17:10:25 -07:00
0c66a44d1b
first working test program
...
measures :
- compression ratio with / without dictionary
- create one dictionary per block
- memory budget for dictionaries
- decompression speed, using one different dictionary per block
current limitations :
- only one file
- 4K blocks only
- automatic dictionary built with 4K size
dictionary can be selected on command line, with -D
2018-08-28 15:47:07 -07:00
e3b5286197
Fix decodecorpus
2018-08-28 13:56:47 -07:00
e984d01912
Small test fixes
2018-08-28 13:42:01 -07:00
5e580de6da
[zstd] Fix seqStore growth
...
We could undersize the literals buffer by up to 11 bytes,
due to a combination of 2 bugs:
* The literals buffer didn't have `WILDCOPY_OVERLENGTH` extra
space, like it is supposed to.
* We didn't check the literals buffer size in `ZSTD_sufficientBuff()`.
2018-08-28 13:24:44 -07:00
5a4e6c9f3d
[fuzzer] Test growing the seqStore_t
2018-08-28 13:20:37 -07:00
b37a0a6bde
Merge pull request #1298 from facebook/bench
...
Refactored bench.c
2018-08-28 12:25:02 -07:00
0491037db9
Merge branch 'bench' into largeNbDicts
2018-08-28 11:26:46 -07:00
Yann Collet
Yann Collet
Nick Terrell
Jennifer Liu
Casey McGinty
Nick Terrell