eb7bbab36a
Remove ZSTD_p_refDictContent and dictContentByRef
2017-08-25 11:11:45 -07:00
15fdeb9e41
Enforce nbThreads<=1 for estimateCCtxSize
2017-08-24 16:28:49 -07:00
1c81f725ff
Remove duplicated testing code
2017-08-23 15:47:15 -07:00
6f1a21c7e9
Remove formatting-only changes
2017-08-23 10:24:19 -07:00
8fd1636776
Remove unused functions
2017-08-22 13:33:58 -07:00
73c73bf16a
Reduce code duplication in zstreamtest
2017-08-21 12:41:19 -07:00
3587556873
[cover] Test small maxdict
2017-08-21 11:16:47 -07:00
91b30dbe84
Remove test parameter
2017-08-21 10:09:06 -07:00
f181f33bdf
Disable tests and refactor
2017-08-21 01:59:08 -07:00
023b24e6d4
Add cctx param tests
2017-08-20 22:55:07 -07:00
d6394cc4c3
fixed test-zstd-nolegacy
2017-08-20 10:15:44 -07:00
32fb407c9d
updated a bunch of headers
...
for the new license
2017-08-18 16:52:05 -07:00
8049556928
Merge pull request #778 from terrelln/bad-huff
...
[libzstd] Fix bug in Huffman decompresser
2017-08-07 14:05:58 -07:00
abe12b3399
[libzstd] Fix bug in Huffman decompresser
...
The zstd format specification doesn't enforce that Huffman compressed
literals (including the table) have to be smaller than the uncompressed
literals. The compressor will never Huffman compress literals if the
compressed size is larger than the uncompressed size. The decompresser
doesn't accept Huffman compressed literals with 4 streams whose compressed
size is at least as large as the uncompressed size.
* Make the decompresser accept Huffman compressed literals whose size
increases.
* Add a test case that exposes the bug. The compressed file has to be
statically generated, since the compressor won't normally produce files
that expose the bug.
2017-08-07 12:37:48 -07:00
e1abc2a367
Switch the sleep function to UTIL_sleepMilli
2017-08-07 11:49:13 -07:00
1e366f9dea
Add test for deadlock
2017-08-02 11:27:50 -07:00
5adceeed01
Allow queueSize=0 in pool.c and update poolTests
2017-07-31 10:10:16 -07:00
38ba7002f2
fixed minor warning on unused variable in shell function
2017-07-20 18:39:04 -07:00
5e6c5203f3
fixed fuzzer test for non OS-X platforms
2017-07-20 15:11:56 -07:00
1ca1288689
added --memtest=# command to fuzzer
...
to jump directly to relevant test section
2017-07-19 16:01:16 -07:00
44b0838253
Merge pull request #770 from terrelln/test-mode
...
[zstdcli] Fix -t in streaming mode
2017-07-18 15:40:59 -07:00
d0b27483ae
[zstdcli] Fix -t in streaming mode
2017-07-18 14:45:49 -07:00
cc1522351f
[libzstd] Fix bug in Huffman encoding
...
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.
The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.
Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
1#Synthetic 50% : 10000000 -> 3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
2#Synthetic 50% : 10000000 -> 3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
3#Synthetic 50% : 10000000 -> 3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
4#Synthetic 50% : 10000000 -> 3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
5#Synthetic 50% : 10000000 -> 3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 243.7 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 226.7 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
1#silesia.tar : 211988480 -> 73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
2#silesia.tar : 211988480 -> 70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
3#silesia.tar : 211988480 -> 66993813 (3.164), 246.5 MB/s , 981.4 MB/s
4#silesia.tar : 211988480 -> 66306481 (3.197), 229.2 MB/s , 972.4 MB/s
5#silesia.tar : 211988480 -> 64757852 (3.274), 149.3 MB/s , 963.6 MB/s
Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
2017-07-18 13:20:40 -07:00
fa3aa04ccd
Merge pull request #761 from paulcruz74/file-rename
...
renamed pool.c to poolTests.c
2017-07-14 09:09:45 -07:00
3a60efd3a9
policy change : ZSTDMT automatically caps nbThreads to ZSTDMT_NBTHREADS_MAX ( #760 )
...
Previously, ZSTDMT would refuse to create the compressor.
Also : increased ZSTDMT_NBTHREADS_MAX to 256,
updated doc,
and added relevant test
2017-07-13 10:17:23 -07:00
052a95f77c
fix : ZSTDMT_compress_advanced() correctly generates checksum
...
when params.fParams.checksumFlag==1.
This use case used to be impossible when only ZSTD_compress() was available
2017-07-11 17:18:26 -07:00
ef0ff7fe7f
zstdmt: removed margin for improved memory usage
2017-07-11 08:54:29 -07:00
4616fad18b
improved ZSTDMT_compress() memory usage
...
does not need the input buffer for streaming operations
also : reduced a few tests time length
2017-07-10 17:16:41 -07:00
670b1fc547
optimized memory usage for ZSTDMT_compress()
...
Previously, each job would reserve a CCtx right before being posted.
The CCtx would be "part of the job description",
and only released when the job is completed (aka flushed).
For ZSTDMT_compress(), which creates all jobs first and only join at the end,
that meant one CCtx per job.
The nb of jobs used to be == nb of threads,
but since latest modification,
which reduces the size of jobs in order to spread the load of difficult areas,
it also increases the nb of jobs for large sources / small compression level.
This resulted in many more CCtx being created.
In this new version, CCtx are reserved within the worker thread.
It guaranteea there cannot be more CCtx reserved than workers (<= nb threads).
To do that, it required to make the CCtx Pool multi-threading-safe :
it can now be called from multiple threads in parallel.
2017-07-10 16:30:55 -07:00
3510efb02d
fix : custom allocator correctly propagated to child contexts
2017-07-10 14:21:40 -07:00
ee3423d709
extended fuzzer MT memory tests
2017-07-10 14:09:16 -07:00
88da8f1816
fix : propagate custom allocator to ZSTDMT though ZSTD_CCtx_setParameter()
...
also : compile fuzzer with MT enabled
2017-07-10 14:02:33 -07:00
f9524cf366
added --memtest to fuzzer
2017-07-10 13:48:41 -07:00
e32fb0c1fe
added ZSTD_sizeof_CCtx() test
2017-07-10 12:29:57 -07:00
89190ef07d
renamed pool.c to poolTests.c
2017-07-10 11:32:30 -07:00
ed0243a63c
removed zbufftest from list of `all` tests
2017-07-07 16:16:14 -07:00
990449b89d
new field : ZSTD_frameHeader.frameType
...
Makes frame type (zstd,skippable) detection more straighforward.
ZSTD_getFrameHeader set frameContentSize=ZSTD_CONTENTSIZE_UNKNOWN to mean "field not present"
2017-07-07 15:21:35 -07:00
7758ed8458
fixed fullbench, part 2
2017-07-06 02:48:00 -07:00
9b2c1acfc0
fixed fullbench
2017-07-06 02:22:57 -07:00
27e883371d
fixed wrong assert() condition
...
A single job created by ZSTDMT_compress() can be < 256KB
if data to compress is < 256 KB
(in which case it is delegated to single thread mode)
2017-07-04 19:33:16 -07:00
2cb9774f5e
more precise estimation of amount to flush at end of stream (single thread mode)
...
also : can use DEBUGLEVEL variable in /tests
2017-07-04 12:39:26 -07:00
5051dd39ca
Merge pull request #743 from facebook/fullbench
...
compress_generic() automatic optimization opportunities
2017-07-03 21:26:38 -07:00
95c4a6e2c8
Merge pull request #745 from terrelln/libfuzzer
...
[fuzz] Add libFuzzer targets
2017-07-03 15:15:20 -07:00
bea0f0cfa0
[fuzz] Move from fuzz/ to tests/fuzz/
2017-07-03 12:40:12 -07:00
4b26306cb8
blindfix : fullbench's one-time leak, detected by valgrind
2017-07-01 08:03:59 -07:00
c07e43c2b5
added --show-leak-kind=all to valgrind tests
2017-07-01 07:05:11 -07:00
b5bb7c6d95
fixed Visual compilation of fullbench-dll
2017-06-29 19:59:37 -07:00
e7e5a8cef7
made fullbench compatible with multi-threading
...
fullbench 61/62 measure speed of ZSTD_compress_generic with 2 threads
2017-06-29 18:56:24 -07:00
afb0aca739
zstreamtest : big tests are only enabled in 64-bits mode
...
to avoid requesting too much memory in 32-bits mode during MT tests
2017-06-29 18:19:09 -07:00
2e84bec9ac
updated fullbench to also measure ZSTD_compress_generic()
...
will make it possible to visualize
optimization opportunity for ZSTD_e_end
2017-06-29 13:03:10 -07:00
7303ed5d74
minor : specify compression level in tags
2017-06-29 11:30:31 -07:00
590937df20
Merge pull request #739 from facebook/refPrefix
...
ZSTD_refPrefix
2017-06-29 04:36:03 -07:00
811deaea6f
Merge pull request #736 from terrelln/cover-default-api
...
[zdict] Make COVER the default algorithm
2017-06-28 20:25:36 -07:00
ad5f29918e
Merge pull request #737 from facebook/advancedAPI2
...
simplified API for *initStatic*()
2017-06-28 20:25:22 -07:00
62f7efc04a
added a test to ensure ZSTD_refPrefix() doesn't impact the following compression job
2017-06-28 16:25:13 -07:00
7ac4724bd2
removed fnum from DISPLAY statements
2017-06-28 13:00:49 -07:00
e667d33b0b
fixed generation of buggy test, corrected DISPLAY statements for errors
2017-06-28 12:19:37 -07:00
33a6639039
fixed ZSTD_refPrefix with Multithread-enabled CCtx
2017-06-28 11:09:43 -07:00
1fd2df9c2c
fixed zstreamtest
...
modifying compression parameters is not allowed after ZSTD_CCtx_loadDictionary()
but it is still allowed after ZSTD_refPrefix()
2017-06-28 09:57:59 -07:00
2e4274262d
controlled dictMode
2017-06-27 17:09:12 -07:00
b7372933b8
implemented ZSTD_refPrefix()
2017-06-27 15:49:12 -07:00
fecc721fd9
added parameter ZSTD_p_refDictContent
2017-06-27 11:46:39 -07:00
09a5bbe22e
Merge pull request #735 from terrelln/xz-default
...
[zstdcli] Support xz by default when liblzma is available
2017-06-27 09:42:02 -07:00
5b7fd7c422
[zdict] Make COVER the default algorithm
2017-06-26 21:09:22 -07:00
20eeb243d1
Merge pull request #729 from paulcruz74/corpus
...
Corpus
2017-06-26 17:47:28 -07:00
dde10b23fe
refactored ZSTD_estimateDStreamSize()
...
now uses windowSize as argument.
Also : created ZSTD_estimateDStreamSize_fromFrame()
2017-06-26 17:44:26 -07:00
09ae03a570
ZSTD_estimateCDictSize_advanced()
...
ZSTD_estimateCDictSize() now uses same arguments as ZSTD_createCDict()
ZSTD_estimateCDictSize_advanced() uses same arguments as ZSTD_createCDict_advanced()
2017-06-26 16:47:32 -07:00
0c9a915a28
ZSTD_estimateCStreamSize_advanced()
2017-06-26 16:02:25 -07:00
31af8290d1
ZSTD_estimateCCtx_advanced()
...
ZSTD_estimateCCtx() is now a "simple" function,
taking int compressionLevel as single argument.
ZSTD_estimateCCtx_advanced() takes a CParams argument,
which is both more complete and more complex to generate.
2017-06-26 15:52:39 -07:00
6aeb50ea0d
[zstdcli] Add unlzma and unxz symlinks
2017-06-26 11:24:36 -07:00
298fe9f23b
added cli tests to test-all
2017-06-23 17:10:41 -07:00
3a295a91f8
added additional condition so large offsets into the dictionary are not generated past windowSize
2017-06-23 15:54:51 -07:00
2085375816
fixed bug detected by the API test
2017-06-23 13:44:24 -07:00
8cd134559d
type warnings
2017-06-23 12:00:48 -07:00
4219acc60a
fixed bus error bug
2017-06-23 11:22:29 -07:00
ef269c1b68
Merge pull request #725 from facebook/advancedAPI2
...
New Advanced API
2017-06-23 09:50:47 -07:00
cca1ed14db
Merge branch 'advancedAPI2' of github.com:facebook/zstd into advancedAPI2
2017-06-23 00:13:03 -07:00
32f546c763
zstreamtest : disabled multi-threading tests with --newapi --no-big-tests
...
--no-big-tests is typically used in combination with qemu-user-static
qemu-user-static allocated 4 GB of RAM upfront.
On 2 GB VM, this can degenerate into a crash.
It's not a problem as long as memory is not used.
But with multi-threading enabled, memory fragmentation kicks in,
so the amoung of RAM effectively touched increases,
and can pass beyond the 2 GB limit of the VM.
In single-threaded mode, there is no such issue :
memory requirement is smaller, and remains well-located,
so very little fragmentation is expected.
This modification should make `qemu-arm-static zstreamtests --newapi --no-big-tests`
work fine on Travis CI.
2017-06-23 00:09:02 -07:00
2e8cc6f12a
added sizeof for clarity
2017-06-22 15:52:33 -07:00
6d7f442120
changed makefile test
2017-06-22 15:44:01 -07:00
b325a2e4db
changed assignment
2017-06-22 15:36:28 -07:00
2935239544
added description for dictionary cli tests
2017-06-22 15:23:59 -07:00
97a8f89bec
made changes to cli test so it runs properly
2017-06-22 15:13:41 -07:00
829eb29033
added cli test for decodecorpus inside tests/Makefile. Also changed calculation of offset
2017-06-22 14:43:44 -07:00
028fb9d01b
added cli test case in Makefile
2017-06-22 13:53:39 -07:00
98751f69e7
should be updating seed whenever multiple files are generated
2017-06-22 10:23:36 -07:00
84cfa07d2d
changed format of command to --use-dict=#
2017-06-22 10:04:14 -07:00
f99c2c1a22
reduced --no-big-tests even more
...
to pass tests on qemu-aarch64
2017-06-21 23:35:58 -07:00
04094f37e9
fixed offset in this case os that it always goes past src start
2017-06-21 18:47:40 -07:00
49f8459d11
fixed minor cast warning
2017-06-21 18:43:39 -07:00
0950b3159a
more meaningful names for count variables
2017-06-21 18:30:27 -07:00
0b6eedeace
malloc samples instead of static allocation
2017-06-21 18:24:19 -07:00
4d3bdcf130
reduced CLevelMax for --no-big-tests
2017-06-21 18:20:02 -07:00
52a13abf3f
removed comment regarding windowSize requirement
2017-06-21 18:13:52 -07:00
32e36c2d9a
removed requirement that offset be less than windowSize
2017-06-21 18:00:47 -07:00
bfc2f00080
--no-big-tests for zstreamtest
...
Apply --no-big-tests for tsan tests
2017-06-21 17:57:14 -07:00
eeff2d0e3c
added const x2
2017-06-21 17:48:37 -07:00
04253e21bb
added const
2017-06-21 17:48:01 -07:00
2ceaec1aad
got rid of unnecessary casting to void*
2017-06-21 17:46:37 -07:00
ccae9ec807
added comment addressing use of bits in frame header descriptor
2017-06-21 17:43:21 -07:00
dce789281b
fixed : decompression of skippable frames in streaming mode
2017-06-21 15:53:42 -07:00
c16748b137
added test cases for new command --list
2017-06-21 12:09:53 -07:00
7bd1a2900e
added ZSTD_dictMode_e to control dictionary loading mode
2017-06-21 11:50:33 -07:00
db3606e249
added test for files with no frame content size
2017-06-20 17:43:36 -07:00
acf0df678c
fixed strange spacing
2017-06-20 15:41:43 -07:00
69bc4fab25
free the context object once done with decompression
2017-06-20 15:37:14 -07:00
ceb4f65620
allocate statically so we don't have to deal with freeing
2017-06-20 15:31:54 -07:00
a73c2a444a
added tests with null files, changed condition to check that the file is more than 0 bytes
2017-06-20 14:33:08 -07:00
58c19b4202
spacing matters for test scripts
2017-06-20 14:14:53 -07:00
ff8f83bd47
fixed fuzzer test
2017-06-20 12:17:32 -07:00
b07d0af2ba
added additional tests for error detection
2017-06-20 11:54:44 -07:00
43ffcdde40
const dictOffset
2017-06-20 11:27:39 -07:00
1e0d125e76
const value
2017-06-20 11:15:26 -07:00
0034129270
made changes to accommodate single file use case
2017-06-20 11:07:38 -07:00
6a99de2dd8
change test to list info on .zst files
2017-06-20 10:45:06 -07:00
36e14b1963
added description to advanced usage, changed to use-dict
2017-06-20 10:34:34 -07:00
15102069a1
removed testing code from generation code
2017-06-20 10:25:16 -07:00
ff5589c091
added comments for clarity
2017-06-20 10:17:38 -07:00
551fb7ca05
changed error condition
2017-06-20 10:14:03 -07:00
b8237c6c66
refactoring for readability
2017-06-20 10:12:13 -07:00
6f17de2132
changed computation for readability
2017-06-20 09:50:28 -07:00
4a4f9ba262
const value that doesn't change
2017-06-19 17:46:21 -07:00
3a6856edca
changed scope to avoid errors
2017-06-19 17:43:16 -07:00
7d2b5613b4
adhere to min dict size
2017-06-19 17:37:14 -07:00
ea2af1ffc4
changed dictionary to stay reasonable size during test
2017-06-19 17:34:50 -07:00
fdbb07283a
added test to runTestMode
2017-06-19 17:29:15 -07:00
bdec7b8ce7
fixed for loop declaration, fixed test case
2017-06-19 17:18:43 -07:00
23098b53e8
wrote a test for dictionary corpus
2017-06-19 17:13:00 -07:00
7a26390989
changing for loop declaration
2017-06-19 16:39:57 -07:00
147ef05f12
for loop declaration not allowed
2017-06-19 16:38:51 -07:00
791352ad0e
scope change
2017-06-19 16:32:38 -07:00
365e265a8e
changed scope so that warning disappears
2017-06-19 16:28:40 -07:00
f4f928595b
changed scope to get rid of mixed code/variables warning
2017-06-19 16:27:05 -07:00
7ea025cc7d
added in struct to keep functions more orderly
2017-06-19 16:23:00 -07:00
b91b810c64
made some small changes to dictInfo struct and initDictInfo
2017-06-19 16:05:05 -07:00
1ee4f6b36c
cleaning up code
2017-06-19 15:55:01 -07:00
2a52d72712
added comment to anotate decompressed error code
2017-06-19 14:21:35 -07:00
73ebb646d8
small stylistic changes
2017-06-19 14:20:35 -07:00
34f4e5cab2
changed if statement styling
2017-06-19 14:07:56 -07:00
1fc1a35dc4
added calls to free to match calls to malloc, made some stylistic changes, added init function for new struct
2017-06-19 14:02:27 -07:00
8f4fa43fe8
got files to decompress
2017-06-19 13:21:50 -07:00
688952062b
minor declaration statement warning fix
2017-06-19 11:53:42 -07:00
d7a3bffba9
new api : setting compression parameters is refused if a dictionary is already loaded
2017-06-19 11:53:01 -07:00
da4ddc1f76
fixed decompression so that it uses compressed version
2017-06-19 11:40:45 -07:00
33467add60
made changes so that HUF_writeCTable no longer fails
2017-06-19 11:27:50 -07:00
ed1d039127
newapi fuzzer tests : random flush orders in main loop
2017-06-19 11:07:33 -07:00
09ac51b56b
small style changes
2017-06-19 09:24:38 -07:00
f5deae8a67
new api fuzzer tests : frame parameters are randonly set
2017-06-18 23:41:38 -07:00
ea3630a889
new api fuzzer set : dictionary is randomly set
2017-06-18 23:31:55 -07:00
8dee0ec99e
new api fuzzer tests : compression parameters are randomly set
2017-06-18 23:25:15 -07:00
Stella Lau
Nick Terrell
Yann Collet
Yann Collet
Paul Cruz
cyan4973