facebook
/
zstd
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,276 Commits
2 Branches
0 Tags
28 MiB
Commit Graph

4276 Commits (bdc83f523b536ade61adb46393d0ac87f18b49bc)

Author SHA1 Message Date
Yann Collet b4d460f32c pool.c : blindfix for Visual warnings 2017-07-20 01:13:14 -07:00
Stella Lau 13a01ffb27 Fix off-by-one in size calculations 2017-07-19 17:24:09 -07:00
Stella Lau 2427a154cb Minor refactoring 2017-07-19 16:56:28 -07:00
Paul Cruz dcf609f835 make adaptCompressionLevel oscillate less 2017-07-19 16:36:33 -07:00
Yann Collet 1ca1288689 added --memtest=# command to fuzzer 
to jump directly to relevant test section
 2017-07-19 16:01:16 -07:00
Paul Cruz 2a22c7915e call ZSTD_compressBegin() once 2017-07-19 16:00:54 -07:00
Paul Cruz 6767abe652 fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0) 2017-07-19 14:54:15 -07:00
Stella Lau 030264ca51 Experiment with integrating ZSTD_count with findBestMatch 2017-07-19 14:14:26 -07:00
Yann Collet 3974d2b38a blind fix for Windows Multithreading module 
adds a fake 0 return value for mutex/cond init
 2017-07-19 13:33:21 -07:00
Paul Cruz 42382c1216 added some debug statements, adjusted end condition 2017-07-19 13:30:07 -07:00
Paul Cruz 6945b3c43d removed previous version of completion for compression 2017-07-19 11:51:50 -07:00
Paul Cruz 5a85c57e30 set up new calculations compression completion progress 2017-07-19 11:47:17 -07:00
Paul Cruz f1ac518b59 split compression into smaller blocks 2017-07-19 11:23:40 -07:00
Paul Cruz 338951cd48 moved compression adapt to avoid warning 2017-07-19 10:23:46 -07:00
Paul Cruz 4497ecf297 change compression level only right before actually performing compression. When waiting, only update waiting statistics. 2017-07-19 10:14:00 -07:00
Paul Cruz e11bf55d0b added mechanism for measuring how much of a job has been created 2017-07-19 10:10:47 -07:00
Paul Cruz 559ea4ff25 split up read process into smaller chunks 2017-07-19 09:59:17 -07:00
Paul Cruz 6119cd2164 added additional print for help menu 2017-07-19 09:43:17 -07:00
Yann Collet b71363b967 check pthread_*_init() success condition 2017-07-19 01:05:40 -07:00
Stella Lau 4352e09cb0 Avoid recounting match lengths with ZSTD_count 2017-07-18 18:35:25 -07:00
Stella Lau 1fa223859f Switch to using ZSTD_count instead of function pointer 2017-07-18 18:05:10 -07:00
Paul Cruz 3d7f1afadd changed createCCtx() to split into initialization and creation 2017-07-18 17:32:36 -07:00
Paul Cruz 2c4e4ddc50 added mutex for stats struct 2017-07-18 15:55:58 -07:00
Yann Collet 44b0838253 Merge pull request #770 from terrelln/test-mode 
[zstdcli] Fix -t in streaming mode
 2017-07-18 15:40:59 -07:00
Paul Cruz ad66faf16a added progress check for filewriting, put important shared data behind mutex when being read from/written to 2017-07-18 15:23:11 -07:00
Yann Collet 2aba13b770 Merge pull request #769 from terrelln/real-block-split 
[libzstd] Fix bug in Huffman encoding
 2017-07-18 14:58:26 -07:00
Nick Terrell d0b27483ae [zstdcli] Fix -t in streaming mode 2017-07-18 14:45:49 -07:00
Stella Lau 19258f51c1 Make the meaning of LDM_MEMORY_USAGE consistent across tables 2017-07-18 14:25:39 -07:00
Paul Cruz a34bc30237 setting up basic readme 2017-07-18 13:31:02 -07:00
Paul Cruz 29c36cf051 rename completion variable, split up fwrite operations in order to track progress 2017-07-18 13:30:29 -07:00
Nick Terrell cc1522351f [libzstd] Fix bug in Huffman encoding 
Summary:
Huffman encoding with a bad dictionary can encode worse than the
HUF_BLOCKBOUND(srcSize), since we don't filter out incompressible
input, and even if we did, the dictionaries Huffman table could be
ill suited to compressing actual data.

The fast optimization doesn't seem to improve compression speed,
even when I hard coded fast = 1, the speed didn't improve over hard coding
it to 0.

Benchmarks:
$ ./zstd.dev -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1890.0 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 372.6 MB/s ,1830.2 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1400.2 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 198.0 MB/s ,1280.1 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 107.8 MB/s ,1200.0 MB/s
$ ./zstd -b1e5
Benchmarking levels from 1 to 5
 1#Synthetic 50%     :  10000000 ->   3139163 (3.186), 524.8 MB/s ,1870.2 MB/s
 2#Synthetic 50%     :  10000000 ->   3115138 (3.210), 370.0 MB/s ,1810.3 MB/s
 3#Synthetic 50%     :  10000000 ->   3222672 (3.103), 223.3 MB/s ,1380.1 MB/s
 4#Synthetic 50%     :  10000000 ->   3276678 (3.052), 196.1 MB/s ,1270.0 MB/s
 5#Synthetic 50%     :  10000000 ->   3271570 (3.057), 106.8 MB/s ,1180.1 MB/s
$ ./zstd.dev -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1096.5 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 321.2 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 243.7 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 226.7 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 150.3 MB/s , 963.6 MB/s
$ ./zstd -b1e5 ../silesia.tar
Benchmarking levels from 1 to 5
 1#silesia.tar       : 211988480 ->  73651685 (2.878), 429.7 MB/s ,1087.1 MB/s
 2#silesia.tar       : 211988480 ->  70158785 (3.022), 318.8 MB/s ,1029.1 MB/s
 3#silesia.tar       : 211988480 ->  66993813 (3.164), 246.5 MB/s , 981.4 MB/s
 4#silesia.tar       : 211988480 ->  66306481 (3.197), 229.2 MB/s , 972.4 MB/s
 5#silesia.tar       : 211988480 ->  64757852 (3.274), 149.3 MB/s , 963.6 MB/s

Test Plan:
I added a test case to the fuzzer which crashed with ASAN before the patch
and succeeded after.
 2017-07-18 13:20:40 -07:00
Paul Cruz ae47eab2fd changed test cases to use -s setting on the diffs 2017-07-18 12:58:50 -07:00
Yann Collet 77d67fb167 Merge pull request #766 from terrelln/real-block-split 
[libzstd] Pull optimal parser state out of seqStore_t
 2017-07-18 08:26:24 -07:00
Yann Collet 14c83b05c7 Merge pull request #765 from terrelln/real-block-split 
[libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences()
 2017-07-17 19:25:55 -07:00
Stella Lau fc41a87964 Experiment with using a lag when hashing 2017-07-17 18:13:09 -07:00
Paul Cruz 5af04c57b0 change parameters for compression level adapt 2017-07-17 17:59:50 -07:00
Paul Cruz b3c9e02bb6 added signal to other threads whenever error occurs 2017-07-17 15:34:58 -07:00
Nick Terrell 7a28b9e4a3 [libzstd] Pull optimal parser state out of seqStore_t 2017-07-17 15:29:11 -07:00
Stella Lau a00e406231 Remove version archive 2017-07-17 15:17:32 -07:00
Stella Lau 15a041adbf Add function to get valid entries only from table 2017-07-17 15:16:58 -07:00
Yann Collet 3381bf4b84 Merge pull request #764 from terrelln/real-block-split 
[libzstd] Refactor ZSTD_compressSequences()
 2017-07-17 14:46:01 -07:00
Paul Cruz 6be22f1f84 swap buffers instead of copying memory over 2017-07-17 14:39:10 -07:00
Paul Cruz 708238e07e open file outside of adaptCCtx, pass to the output thread 2017-07-17 14:01:13 -07:00
Nick Terrell e198230645 [libzstd] Remove ZSTD_CCtx* argument of ZSTD_compressSequences() 2017-07-17 12:27:24 -07:00
Stella Lau 4bb42b02c1 Add basic chaining table 2017-07-17 11:53:54 -07:00
Nick Terrell 634f012420 [libzstd] Refactor ZSTD_compressSequences() 2017-07-17 11:36:11 -07:00
Paul Cruz 044e40db5a removed freeCCtx() calls from createCCtx() so that it is not called twice during errors 2017-07-17 11:19:23 -07:00
Paul Cruz 50ce4eaeb6 added error detection for pthread initialization, added compression completion measurement, fixed const values 2017-07-17 10:12:44 -07:00
Stella Lau ca300ce6e0 Decouple hash table from compression function 2017-07-14 17:17:00 -07:00
Paul Cruz 1ab3f06f00 updated tests to use different seeds when executing different tests 2017-07-14 16:29:29 -07:00