74b1c75d64
btopt : minor adjustment of update frequencies
2018-05-10 16:32:36 -07:00
ac6105463a
opt: minor improvements to log traces
...
slight improvement when using fractional-bit evaluation (opt:dictionay)
2018-05-09 15:46:11 -07:00
c39061cb7b
fixed declaration-after-statement warning
2018-05-09 12:07:25 -07:00
4d5bd32a00
added traces to look at symbol costs
...
evaluation looks correct.
2018-05-09 12:00:12 -07:00
c0da0f5e9e
switchable bit-approximation / fractional-bit accuracy modes
...
also : makes it possible to select nb of fractional bits.
2018-05-09 10:48:09 -07:00
ba2ad9b6b9
implemented fractional bit cost evaluation
...
for FSE symbols.
While it seems to work, the gains are negligible compared to rough maxNbBits evaluation.
There are even a few losses sometimes, that still need to be explained.
Furthermode, there are still cases where btlazy2 does a better job than btopt,
which seems rather strange too.
2018-05-08 17:43:13 -07:00
1aff63b114
opt: shift all costs by 8 bits (* 256)
...
making it possible to represent fractional bit costs.
2018-05-08 16:19:04 -07:00
6a3c34aa58
opt: estimate cost of both Hufman and FSE symbols
...
For FSE symbols : provide an upper bound,
in nb of bits,
since cost function is not able to store fractional bit costs.
2018-05-08 16:11:21 -07:00
338f738c24
pass entropy tables to optimal parser
...
for proper estimation of symbol's weights
when using dictionary compression.
Note : using only huffman costs is not good enough,
presumably because sequence symbol costs are incorrect.
2018-05-08 15:37:06 -07:00
a155061328
minor code refactor for readability
...
removed some useless operations from optimal parser
(should not change performance, too small a difference)
2018-05-08 12:32:44 -07:00
ef1abd3c07
Merge pull request #1120 from lamby/897904-libzstd-please-make-the-build-reproducible
...
Make the build reproducible
2018-05-04 12:39:40 -07:00
2dbe408a49
Make the build reproducible
...
Whilst working on the Reproducible Builds effort [0], we noticed
that zstd could not be built reproducibly.
This is due to the manual page encoding the number of CPUs from the
build machine and thus varies across builds.
This was originally filed in Debian as #897904 [1].
[0] https://reproducible-builds.org/
[1] https://bugs.debian.org/897904
Signed-off-by: Chris Lamb <lamby@debian.org>
2018-05-04 08:39:51 -07:00
5ec92c4d96
Merge pull request #1118 from terrelln/playtests-typo
...
Fix playTests.sh typo
2018-05-03 14:26:26 -07:00
263134ce66
Fix playTests.sh typo
...
Thanks to @mestia for reporting and finding the bug!
Fixes #1116 .
2018-05-03 13:03:08 -07:00
b095bffac6
ignore Windows build/test artefacts
2018-05-01 05:45:46 -07:00
1fc15dfd9d
Merge pull request #1114 from pseiderer/ps-devel-001
...
Split library install target into pc, static, shared and include only target
2018-04-30 14:59:59 -07:00
82ad249645
Clarifications of Zstandard format specification
...
from IETF RFC review
2018-04-30 12:36:55 -07:00
64bfdca5b9
Split library install target into pc, static, shared and include only target
...
Signed-off-by: Peter Seiderer <ps.report@gmx.net>
2018-04-30 20:32:32 +02:00
a11f0c67bb
Merge pull request #1113 from terrelln/dict-fix
...
Fix parameter adjustment with dictionary
2018-04-26 10:31:25 -07:00
ca77822ddf
Fix parameter adjustment with dictionary
...
The new advanced API basically set `requestedParams = appliedParams` when
using a dictionary. This halted all parameter adjustment, which can hurt
compression ratio if, for example, the window log is small for the first
call, but the rest of the files are large.
This patch fixes the bug, and checks that the `requestedParams` don't change
in the new advanced API when using a dictionary, and generally in the fuzzer.
2018-04-25 16:32:29 -07:00
12f60b8c98
clarified documentation related to refPrefix()
2018-04-25 10:17:06 -07:00
449cccf6c9
Merge pull request #1112 from facebook/lz4dec
...
fixed decoding bogus lz4 frame
2018-04-24 16:50:54 -07:00
ace856a835
updated documentation of streaming compression api
2018-04-24 14:44:27 -07:00
62487b5e76
fixed decoding bogus lz4 frame
...
FIO would keep presenting data after an LZ4F decoding error
resulting in a NULL pointer dereference
when associated with older liblz4 version (< v1.8.1.2)
2018-04-23 18:50:16 -07:00
2c3ad05812
Fix the problem that enables DYNAMIC_BMI2 macro by mistake on ARM architecture with Clang ( #1110 )
2018-04-23 15:41:50 -07:00
9de03cd748
Merge pull request #1107 from terrelln/reset-params
...
Add ZSTD_CCtx_resetParameters() function
2018-04-13 13:23:47 -07:00
e8c9dc5cea
Fix documentation
2018-04-13 12:43:38 -07:00
c0987986e5
Only reset CDict in ZSTD_CCtx_resetParameters()
2018-04-13 11:26:40 -07:00
9f76eebd17
Add ZSTD_CCtx_resetParameters() function
...
* Fix docs for `ZSTD_CCtx_reset()`.
* Add `ZSTD_CCtx_resetParameters()`.
Fixes #1094 .
2018-04-12 16:54:07 -07:00
3c3f59e68f
Enforce pledgeSrcSize whenever known ( #1106 )
...
The test fails before the patch and passes after.
Fixes #1095 .
2018-04-12 16:02:03 -07:00
1f25b17c7d
Merge pull request #1105 from terrelln/get-param
...
Add ZSTD_CCtx(Param)?_getParameter() function
2018-04-12 13:25:57 -07:00
280a236e9e
Add ZSTD_CCtx(Param)?_getParameter() function
...
Closes #1096 .
2018-04-12 11:50:12 -07:00
04212178b5
doc : clarified advanced API usage
...
sticky parameters only work with `ZSTD_compress_generic()`
2018-04-10 11:40:36 -07:00
ad5ba6cdcf
updated comment on parameters that can be changed during compression
2018-04-09 17:39:07 -07:00
1da629f2ad
Merge pull request #1104 from terrelln/fast-train
...
Allow negative compression levels in training
2018-04-09 14:16:20 -07:00
569e2abccd
Allow negative compression levels in training
...
* Set `dictCLevel` in `zstdcli.c`.
* Only set to default level if the compression level `== 0`, not `<= 0`.
2018-04-09 12:12:03 -07:00
77e373bff5
Merge pull request #1103 from kilobyte/dev
...
Fix riscv.
2018-04-09 11:12:51 -07:00
00d52282d0
Use -pthread rather than -lpthread.
...
It can have other effects, such as pulling in extra libraries. Without it,
riscv build fails with undefined reference to `__atomic_compare_exchange_1'.
2018-04-09 01:50:49 +02:00
4195b36dd7
Merge pull request #1100 from bket/stable_sort
...
zstd requires a stable sort.
2018-04-05 11:39:27 -07:00
f35b8ba9da
updated ZSTD_p_chainLog description
2018-04-05 11:05:11 -07:00
b1e4d891cf
Merge pull request #1099 from bket/fix_typo
...
fix typo in programs/zstd.{1,1.md}
2018-04-05 10:30:35 -07:00
462aed6811
zstd requires a stable sort.
...
On OpenBSD qsort() is not guaranteed to be stable, their mergesort() is.
This fixes issue #1088 . All the hard work has been done by @terrelln.
2018-04-05 07:59:16 +02:00
e5ea8d272a
fix typo in programs/zstd.{1,1.md}
...
s/nodictID/no-dictID/g
2018-04-05 06:44:46 +02:00
55f67502f4
Merge pull request #1098 from terrelln/nd-mt
...
Only load extra table positions for CDicts
2018-04-02 15:38:20 -07:00
295ab0dbfa
Only load extra table positions for CDicts
...
Zstdmt uses prefixes to load the overlap between segments. Loading extra
positions makes compression non-deterministic, depending on the previous
job the context was used for. Since loading extra position takes extra
time as well, only do it when creating a `ZSTD_CDict`.
Fixes #1077 .
2018-04-02 14:41:30 -07:00
e866f0919f
Merge pull request #1089 from bket/portability
...
examples/Makefile assumes GNU make.
2018-04-02 14:16:16 -07:00
5b616fa269
Merge pull request #1090 from bket/openbsd
...
Fix building zstd on OpenBSD.
2018-04-02 14:15:26 -07:00
daf3ab0aad
Make sure that make check runs on OpenBSD. ( #1091 )
...
OpenBSD uses md5 instead of md5sum, and has no device called full.
With this patch, make check runs until #1088 . With the assumption made
in the issue make check runs succesfully.
2018-04-02 14:12:18 -07:00
754a821f17
Merge pull request #1080 from facebook/getFrameHeader
...
added ZSTD_getFrameHeader_advanced()
2018-04-01 14:27:46 -07:00
9d3048346d
Fix building zstd on OpenBSD.
2018-03-31 10:46:20 +02:00
Yann Collet
Yann Collet
Chris Lamb
Nick Terrell
cyan4973
Peter Seiderer
taigacon
Nick Terrell
Adam Borowski
Björn Ketelaars