Commit Graph

691 Commits (bd547232bc0aa43591b9eabf51a78ae3cb10fdcf)

Author SHA1 Message Date
Shawn Landden daffe435c0 meson: support differn't legacy levels.
Default to v0.4.0+
2017-12-21 15:47:38 -08:00
Shawn Landden 3ddfa42fe8 meson: fix build
used absolute paths which are deprecated in meson, also missing some sources
that got split

also move source files each to their own line so future diffs are clearer.
2017-12-19 22:02:03 -08:00
Yann Collet c173dbd6e7 no longer supported starting C++17 2017-12-04 18:00:53 -08:00
Yann Collet 0d58aaf6f0 /contrib: fixed license header
removed last reference to PATENTS file
2017-10-02 02:07:17 -07:00
Yann Collet f1a3ce34b7 Merge pull request #826 from terrelln/license
[linux-kernel] Update license
2017-08-31 12:52:23 -07:00
Nick Terrell f9252d8347 [linux-kernel] Update license 2017-08-31 12:48:36 -07:00
Yann Collet b0cb081dc8 last batch of header files changed to reflect new license (#825)
only remains to update contrib/linux-kernel (@terrelln)
2017-08-31 12:20:50 -07:00
Yann Collet e21384fffb fixed more file headers after license change (#825) 2017-08-31 12:11:57 -07:00
Yann Collet e9dc204f42 fixed a bunch of headers after license change (#825) 2017-08-31 11:24:54 -07:00
Yann Collet 394bdd7db9 changed license for examples
intentionnally this time
2017-08-29 09:24:11 -07:00
Yann Collet dd63533b29 Merge pull request #797 from paulcruz74/documentation
Updating README.md
2017-08-16 00:37:55 -07:00
Paul Cruz 733ca51360 Updating README.md 2017-08-15 17:48:23 -07:00
Yann Collet 2daeab12f4 Merge pull request #795 from terrelln/squashfs-tools2
[kernel] Update squashfs-tools patch
2017-08-15 11:59:41 -07:00
Nick Terrell 57e2df6651 [kernel] Update squashfs-tools patch 2017-08-14 22:43:36 -07:00
Paul Cruz fc90469587 updated program name print statement 2017-08-10 16:11:59 -07:00
Paul Cruz 75c3ad947c Merge remote-tracking branch 'upstream/dev' into dev 2017-08-10 14:46:31 -07:00
Yann Collet 8915b9b518 Merge pull request #786 from terrelln/squashfs-tools
[linux-kernel] Update patches for v4 and v5
2017-08-09 16:00:11 -07:00
Nick Terrell 8b6702a00d [linux-kernel] Update patches for v5 2017-08-09 13:03:40 -07:00
Yann Collet a1fe656165 Merge pull request #779 from paulcruz74/adapt-approach-4
Patching warnings, adding ability to bound compression variation
2017-08-07 14:50:38 -07:00
Paul Cruz 7069bb9458 Merge branch 'adapt-approach-4' into bug-fixes 2017-08-07 13:18:53 -07:00
Paul Cruz e100a311eb removed direct assignment of 22, used ZSTD_maxCLevel() instead 2017-08-07 13:11:07 -07:00
Nick Terrell 7393b49fbd [linux-kernel] Update patches for v4 2017-08-04 16:57:03 -07:00
Paul Cruz 01237e3b35 changed multi to zstd-adaptive in the help menu 2017-08-03 15:13:49 -07:00
Paul Cruz 8be7bba08c added mutex for compression level to avoid data race 2017-08-02 10:27:33 -07:00
Paul Cruz 69ef22c0ac added detach statements to prevent resource leak 2017-08-01 17:36:13 -07:00
Paul Cruz f8c7b191e7 Merge branch 'adapt-approach-4' into dev 2017-07-31 15:42:57 -07:00
Paul Cruz 0295737ad7 change signal to broadcast for jobCompressed condition varaible since multiple threads waiting 2017-07-31 13:43:03 -07:00
Paul Cruz cf92221671 Merge remote-tracking branch 'upstream/dev' into dev 2017-07-31 11:43:40 -07:00
Paul Cruz c2f1bda8a1 Merge branch 'adapt-approach-4' into dev 2017-07-31 11:43:01 -07:00
Paul Cruz 9ea7df03de add install target in makefile 2017-07-31 11:04:17 -07:00
Paul Cruz f60cd3f99b print defaults and range, remove EXT 2017-07-31 09:47:09 -07:00
Paul Cruz e22b60cb76 removed ternary operation, added assert statement, check to make sure initial compression level is within bounds 2017-07-28 17:46:51 -07:00
Paul Cruz cb9af53e77 delete empty line 2017-07-28 17:28:25 -07:00
Paul Cruz 51788225db remove exe extension from makefile, reinclude pthread flag 2017-07-28 17:27:36 -07:00
Yann Collet ee27f6937c Merge pull request #758 from stellamplau/ldm
Add working prototype of a long distance matcher
2017-07-28 17:07:21 -07:00
Paul Cruz 4d904ac800 add flags for multithreading 2017-07-28 16:12:58 -07:00
Paul Cruz 0f4cb67b00 add tests for compression bounds, fix another warning 2017-07-28 15:55:02 -07:00
Paul Cruz ff54fced64 patched style errors, add ability to bound compression level variation 2017-07-28 15:30:46 -07:00
Stella Lau 8fae41c412 Return error code in verify() and minor code cleanup 2017-07-27 17:14:05 -07:00
Stella Lau 1294a4a897 Fix typo 2017-07-27 15:49:46 -07:00
Stella Lau 627621839c Add checks in initialization code 2017-07-27 15:37:37 -07:00
Stella Lau c105f605e6 Update README 2017-07-27 11:11:35 -07:00
Yann Collet e1222544be Merge pull request #753 from paulcruz74/adapt-approach-3
adaptive compression v1
2017-07-27 10:00:10 -07:00
Paul Cruz 2320e7378a remove unused variable, add documentation for context fields 2017-07-26 17:02:47 -07:00
Stella Lau 9eaf3d22d0 Allow HASH_ONLY_EVERY_LOG to be configured in ldm.h 2017-07-26 16:43:25 -07:00
Paul Cruz ab5a78547e fix leaky abstraction regarding measuring completion 2017-07-26 16:40:05 -07:00
Paul Cruz 715f36ca81 added definitions for conversion constants, moved forced compression check to top of adaptCompressionLevel, used ZSTD_BLOCKSIZE_MAX 2017-07-26 15:52:15 -07:00
Paul Cruz 6c1c1242fc set the window log value before performing compression 2017-07-26 14:29:59 -07:00
Stella Lau 40759bade9 Add README and clean up code 2017-07-26 13:24:03 -07:00
Paul Cruz a959cc881a moved reset of completion to right after wait 2017-07-26 10:34:48 -07:00
Paul Cruz 305d5ee70f change to >= convergence counter 2017-07-26 10:20:29 -07:00
Paul Cruz be92a38d6a decrease completion requirements for change, move create thread wait, merge cases where compression thread should wait 2017-07-26 10:05:10 -07:00
Stella Lau e9161637b2 Allow parameters to be modified from a separate file 2017-07-25 18:13:27 -07:00
Paul Cruz 0b18d21e03 building on readme, added another help tip in the menu 2017-07-25 17:47:02 -07:00
Paul Cruz 7cc74e0b27 adding more to readme 2017-07-25 16:55:16 -07:00
Paul Cruz 8dbb07d822 updated progress bar with better representation of time, added const 2017-07-25 16:03:43 -07:00
Paul Cruz 9a132707af changing time units to seconds 2017-07-25 15:26:26 -07:00
Stella Lau 629c300118 Rename and remove unneeded files 2017-07-25 15:17:36 -07:00
Paul Cruz 31a9ed9883 updated const values, added more comments 2017-07-25 14:53:40 -07:00
Paul Cruz 5cfbf609a4 removed old debug statements no longer being used 2017-07-25 14:31:48 -07:00
Paul Cruz 0882cd1981 progress bar -- don't print num jobs, time elapsed shown in seconds 2017-07-25 14:26:55 -07:00
Paul Cruz 310c12d07e moved debug statements to a compiler flag 2017-07-25 14:08:39 -07:00
Paul Cruz e02c79f833 started using decrease cooldown so that compression level would not decrease several times in a row 2017-07-25 11:16:27 -07:00
Paul Cruz 85d7c919f6 created independent function for controlling how completion relates to compression level change 2017-07-25 10:32:14 -07:00
Paul Cruz 6f1e260edd added mechanism for getting rid of spikes 2017-07-25 10:01:10 -07:00
Paul Cruz 700758d676 added help statement for -p, switched it to hide progress bar now that progress bar is default 2017-07-24 16:26:20 -07:00
Paul Cruz df3754b6ed add quiet option, make progress bar default 2017-07-24 16:19:07 -07:00
Stella Lau 0295a27133 Experiment with not using a checksum 2017-07-24 15:26:44 -07:00
Paul Cruz 4dc83ca64c compression thread should take measurements independently based on whether or not the create/write thread will actually bottleneck performance 2017-07-24 15:14:58 -07:00
Paul Cruz 0ee3f8c2f8 adding more debug 2017-07-24 15:06:11 -07:00
Paul Cruz 8328f8192a updating debug statements again 2017-07-24 14:40:23 -07:00
Paul Cruz d3d759301f changing position of endline for debug 2017-07-24 13:47:39 -07:00
Stella Lau 08a6e9a141 Minor code cleanup 2017-07-24 13:22:00 -07:00
Stella Lau 6eefa32911 Deduplicate code 2017-07-24 12:40:59 -07:00
Stella Lau 8ed9220102 Experiment with eviction policies and minor code cleanup 2017-07-24 12:05:43 -07:00
Paul Cruz e508f632d6 updated comments and debug statements 2017-07-24 11:01:36 -07:00
Stella Lau eb16da647d Minor clean up 2017-07-24 10:20:53 -07:00
Stella Lau 1a188fe864 Fix overflow bug when calculating hash 2017-07-24 10:20:53 -07:00
Stella Lau 0b8fb1703b Experiment with 64-bit hash insertion policy 2017-07-24 10:20:30 -07:00
Stella Lau 273c17b350 Experiment with 64-bit hash and checksum 2017-07-24 10:19:50 -07:00
Paul Cruz 483d936b87 reduced competition for completion mutex by separating mutex use based on which values is updated 2017-07-23 14:09:16 -07:00
Paul Cruz 880f08d104 change how completion is measured in compression thread 2017-07-23 10:18:54 -07:00
Paul Cruz 08d9e42ec6 removed useless measurements 2017-07-21 18:02:55 -07:00
Paul Cruz 95bef759b3 switched over to model where reading only waits on compression thread 2017-07-21 17:49:39 -07:00
Paul Cruz 6455ec482c taking the maximum of the completion level reads in order to determine which one was waiting more 2017-07-21 16:05:01 -07:00
Paul Cruz 05fe8dd47c updating debug statements 2017-07-21 14:06:24 -07:00
Paul Cruz db109f8fef measure multiple completion levels during each wait 2017-07-21 13:38:24 -07:00
Paul Cruz 721c6a8b97 added bounding to compression level change 2017-07-21 09:30:24 -07:00
Paul Cruz e929d3b787 added priority decision making for adapt compression level 2017-07-21 09:26:35 -07:00
Paul Cruz 9259c7afa4 semi working version that stabilizes 2017-07-20 18:45:33 -07:00
Paul Cruz 82e488770c fixed bug where writeSize could be zero 2017-07-20 16:38:02 -07:00
Paul Cruz a19916425d reworked adaptCompressionLevel to only account for completion information 2017-07-20 16:19:16 -07:00
Nick Terrell 7d3ac0710d [linux] Update patches for v3 2017-07-20 13:33:55 -07:00
Paul Cruz 7ab758a640 changed how completion is actually sampled 2017-07-20 10:53:51 -07:00
Stella Lau 13a01ffb27 Fix off-by-one in size calculations 2017-07-19 17:24:09 -07:00
Stella Lau 2427a154cb Minor refactoring 2017-07-19 16:56:28 -07:00
Paul Cruz dcf609f835 make adaptCompressionLevel oscillate less 2017-07-19 16:36:33 -07:00
Paul Cruz 2a22c7915e call ZSTD_compressBegin() once 2017-07-19 16:00:54 -07:00
Paul Cruz 6767abe652 fixing error when file size is multiple of job size (in which case, the srcSize of the last job is 0) 2017-07-19 14:54:15 -07:00
Stella Lau 030264ca51 Experiment with integrating ZSTD_count with findBestMatch 2017-07-19 14:14:26 -07:00
Paul Cruz 42382c1216 added some debug statements, adjusted end condition 2017-07-19 13:30:07 -07:00
Paul Cruz 5a85c57e30 set up new calculations compression completion progress 2017-07-19 11:47:17 -07:00
Paul Cruz f1ac518b59 split compression into smaller blocks 2017-07-19 11:23:40 -07:00
Paul Cruz 338951cd48 moved compression adapt to avoid warning 2017-07-19 10:23:46 -07:00
Paul Cruz 4497ecf297 change compression level only right before actually performing compression. When waiting, only update waiting statistics. 2017-07-19 10:14:00 -07:00
Paul Cruz e11bf55d0b added mechanism for measuring how much of a job has been created 2017-07-19 10:10:47 -07:00
Paul Cruz 559ea4ff25 split up read process into smaller chunks 2017-07-19 09:59:17 -07:00
Paul Cruz 6119cd2164 added additional print for help menu 2017-07-19 09:43:17 -07:00
Stella Lau 4352e09cb0 Avoid recounting match lengths with ZSTD_count 2017-07-18 18:35:25 -07:00
Stella Lau 1fa223859f Switch to using ZSTD_count instead of function pointer 2017-07-18 18:05:10 -07:00
Paul Cruz 3d7f1afadd changed createCCtx() to split into initialization and creation 2017-07-18 17:32:36 -07:00
Paul Cruz 2c4e4ddc50 added mutex for stats struct 2017-07-18 15:55:58 -07:00
Paul Cruz ad66faf16a added progress check for filewriting, put important shared data behind mutex when being read from/written to 2017-07-18 15:23:11 -07:00
Stella Lau 19258f51c1 Make the meaning of LDM_MEMORY_USAGE consistent across tables 2017-07-18 14:25:39 -07:00
Paul Cruz a34bc30237 setting up basic readme 2017-07-18 13:31:02 -07:00
Paul Cruz 29c36cf051 rename completion variable, split up fwrite operations in order to track progress 2017-07-18 13:30:29 -07:00
Paul Cruz ae47eab2fd changed test cases to use -s setting on the diffs 2017-07-18 12:58:50 -07:00
Stella Lau fc41a87964 Experiment with using a lag when hashing 2017-07-17 18:13:09 -07:00
Paul Cruz 5af04c57b0 change parameters for compression level adapt 2017-07-17 17:59:50 -07:00
Paul Cruz b3c9e02bb6 added signal to other threads whenever error occurs 2017-07-17 15:34:58 -07:00
Stella Lau a00e406231 Remove version archive 2017-07-17 15:17:32 -07:00
Stella Lau 15a041adbf Add function to get valid entries only from table 2017-07-17 15:16:58 -07:00
Paul Cruz 6be22f1f84 swap buffers instead of copying memory over 2017-07-17 14:39:10 -07:00
Paul Cruz 708238e07e open file outside of adaptCCtx, pass to the output thread 2017-07-17 14:01:13 -07:00
Stella Lau 4bb42b02c1 Add basic chaining table 2017-07-17 11:53:54 -07:00
Paul Cruz 044e40db5a removed freeCCtx() calls from createCCtx() so that it is not called twice during errors 2017-07-17 11:19:23 -07:00
Paul Cruz 50ce4eaeb6 added error detection for pthread initialization, added compression completion measurement, fixed const values 2017-07-17 10:12:44 -07:00
Stella Lau ca300ce6e0 Decouple hash table from compression function 2017-07-14 17:17:00 -07:00
Paul Cruz 1ab3f06f00 updated tests to use different seeds when executing different tests 2017-07-14 16:29:29 -07:00
Stella Lau 6e443b4960 Move hash table access for own functions 2017-07-14 14:27:55 -07:00
Stella Lau 2d8e6c6608 Add more statistics 2017-07-14 12:31:01 -07:00
Stella Lau 55f960e8db Add percentages to offset histogram 2017-07-14 11:00:20 -07:00
Stella Lau 4db7f12ef3 Add offset histogram 2017-07-14 10:52:03 -07:00
Paul Cruz 0c8b9436b7 removed goto statements for the most part 2017-07-13 16:38:20 -07:00
Stella Lau 175a6c6029 [ldm] Minor refactoring 2017-07-13 16:16:31 -07:00
Stella Lau 361c06df75 Add min/max offset to stats 2017-07-13 15:29:41 -07:00
Paul Cruz 65a4ce2635 added tests for forced compression level 2017-07-13 14:57:24 -07:00
Paul Cruz 0d9665cef5 added additional tests for performance, allowed force compression level for testing purposes 2017-07-13 14:46:54 -07:00
Stella Lau 2b3c7e4199 [ldm] Make some functions shared 2017-07-13 14:39:35 -07:00
Paul Cruz 9165e97fc6 added some tests for correctness, time, and compression ratio 2017-07-13 13:50:23 -07:00
Stella Lau 9306feb8fa [ldm] Switch to using lib/common/mem.h and move typedefs to ldm.h
Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

Blame Revision:
2017-07-13 13:44:48 -07:00
Stella Lau 50421d9474 [ldm] Remove old main files 2017-07-13 11:45:00 -07:00
Stella Lau 68c4560701 [ldm] Add TODO and comment for segfaulting in compress function 2017-07-13 10:38:19 -07:00
Paul Cruz 766663f1f1 added altering dictionary size depending on compression level 2017-07-13 10:15:27 -07:00
Stella Lau 92bed4a7e0 [ldm] Add CHAR_OFFSET in hash function and extend header size 2017-07-12 18:47:26 -07:00
Paul Cruz 7c886db0a8 changed to stderr 2017-07-12 17:28:53 -07:00
Paul Cruz b5b18cf664 changed to malloc, added comment about adaptive compression level, and changed ternary operators 2017-07-12 17:10:58 -07:00
Paul Cruz 954d999abf fixed up freeCCtx() removed BYTE since it wasn't being used 2017-07-12 16:50:43 -07:00
Paul Cruz 3c16edd26a added copyright header, removed clean from makefile 2017-07-12 16:40:24 -07:00
Stella Lau 8de82b6eb0 [ldm] Clean up versions 2017-07-12 16:31:31 -07:00
Paul Cruz 74d3a6f5ae passes tests with adaptive compression level 2017-07-12 16:18:41 -07:00
Paul Cruz 5353d350ae working with fixed compression level and fixed dictionary size 2017-07-12 16:02:20 -07:00
Stella Lau 8ff8cdb15b [ldm] Clean up code 2017-07-12 15:12:07 -07:00
Paul Cruz 356ddb649f working with flush job->src.size and fixed cLevel 2017-07-12 12:21:21 -07:00
Stella Lau 3a48ffd4fd Fix sumToHash to use hash space more efficiently 2017-07-12 10:53:19 -07:00
Stella Lau e0d4162464 Minor fix for non-rolling hash 2017-07-12 09:50:24 -07:00
Stella Lau 50502519fb Switch to using rolling hash only 2017-07-12 09:47:00 -07:00
Stella Lau 583dda17a8 Update rolling hash 2017-07-11 18:13:26 -07:00
Paul Cruz 0a401852c4 added debug statement 2017-07-11 16:50:50 -07:00
Paul Cruz 72a183efad changed dictionary size, added debugging statements 2017-07-11 15:49:52 -07:00
Paul Cruz 7c54e09347 updated DEBUG statements 2017-07-11 15:15:41 -07:00
Paul Cruz a3c077b8c6 added error message, updated copying dictionary into the input buffer 2017-07-11 15:00:52 -07:00
Paul Cruz 34afb9b23e changed to using ZSTD_compressBegin_usingDict() and fixed strange issue with ZSTD_compressContinue() 2017-07-11 11:50:00 -07:00
Paul Cruz 7ec5928626 fixed an error where -c argument wasn't working for single files 2017-07-11 10:23:25 -07:00
Stella Lau f6c5d07fe2 Save v3 2017-07-11 09:23:44 -07:00
Stella Lau 6c3673f4c3 Add rolling hash 2017-07-10 22:27:43 -07:00
Paul Cruz f918545491 made some progress on improving compression ratio, but problems exist with speed limits, and for some reason higher compression levels are really slow 2017-07-10 18:16:42 -07:00
Paul Cruz 01fc7c4244 changed how the detection of the last job works 2017-07-10 16:27:58 -07:00
Paul Cruz c36552ef8a dst buffer should use ZSTD_compressBound to determine how much space it needs 2017-07-10 16:10:19 -07:00
Paul Cruz 7aa36df6df fixed memory leak that was happening when creating jobs 2017-07-10 16:03:09 -07:00
Stella Lau ef2b728316 Clean up and refactor compress function 2017-07-10 15:48:47 -07:00
Paul Cruz e410d63d45 made input buffer an internal part of the compression context 2017-07-10 15:37:14 -07:00
Stella Lau e4155b11d7 Add warning flags to makefile and clean up code to remove warnings 2017-07-10 13:08:19 -07:00
Stella Lau 10a71d9f1c Add compression context 2017-07-10 12:38:27 -07:00
Paul Cruz cc7f8e4d71 small changes 2017-07-10 11:10:11 -07:00
Paul Cruz 7e09b508ff changed name 2017-07-10 11:05:37 -07:00
Paul Cruz ed72ea5438 removed single from Makefile 2017-07-10 10:58:03 -07:00
Paul Cruz ced3ec5714 removed scripts 2017-07-10 10:53:02 -07:00
Paul Cruz 82f0d64bee removed single.c 2017-07-10 10:51:50 -07:00
Paul Cruz 62ebbabd32 updated error checking in each thread 2017-07-10 09:36:22 -07:00
Stella Lau ae9cf235d6 Add LDM_DCtx 2017-07-10 07:38:09 -07:00
Stella Lau 5432214ee3 Minor refactoring 2017-07-10 06:50:49 -07:00
Stella Lau b94b468e84 Merge branch 'ldm' of https://github.com/stellamplau/zstd into ldm 2017-07-10 06:32:46 -07:00
Stella Lau 474e06ac5b Minor refactoring 2017-07-10 06:32:29 -07:00
Stella Lau eb280cd568 Add folder for old versions 2017-07-10 06:32:05 -07:00
Stella Lau 719ccdc5a5 Update mainfile 2017-07-09 22:45:54 -07:00
Stella Lau acdeb9f302 Add compression statistics 2017-07-07 17:09:28 -07:00
Paul Cruz c3ae23d459 added ability to compress without specifying out filename 2017-07-07 17:07:05 -07:00
Paul Cruz 7163ffafde playing around with adapt param 2017-07-07 15:56:00 -07:00
Paul Cruz 1c9d6b2c6b rewrote time elapsed with UTIL 2017-07-07 15:42:20 -07:00
Paul Cruz c0c236a28b changed to using compressCCtx 2017-07-07 15:13:40 -07:00
Stella Lau 4076be09ec [ldm] Update to hash every position 2017-07-07 14:52:40 -07:00
Stella Lau 7945f9ee47 Fix offset overflow bug 2017-07-07 14:14:01 -07:00
Paul Cruz 11fc0f4119 changed completed -> compressed 2017-07-07 13:55:38 -07:00
Paul Cruz 09d7c6a994 changed completed variables to compressed for clarity 2017-07-07 13:18:55 -07:00
Stella Lau f791fc27e3 Add header with compress and decompress size 2017-07-07 12:44:29 -07:00
Paul Cruz 8c0eb62920 removed unnecessary comments, uncommented DEBUGLOG for later use 2017-07-07 11:47:16 -07:00
Paul Cruz 70a4153bd3 added ability to force output to stdout, wrote an additional test for this functionality 2017-07-07 11:32:14 -07:00
Paul Cruz 532f439961 cleaned up code for arguments a bit 2017-07-07 10:58:43 -07:00
Paul Cruz f7e6b358d0 added tests that check to ensure stdout is working 2017-07-07 10:29:06 -07:00
Paul Cruz 4679132f59 updated avg compression rate, also hiding progress bar behind a flag now 2017-07-07 10:25:38 -07:00
Paul Cruz 00bc5df4e0 added compression rate to status bar 2017-07-07 09:35:39 -07:00
Paul Cruz f351848b76 added data amount 2017-07-06 20:40:00 -07:00
Paul Cruz 2939301023 fixed problem with progress bar not persisting, added time elapsed 2017-07-06 20:30:20 -07:00
Paul Cruz 57ec0232a8 added help menu 2017-07-06 18:09:10 -07:00
Paul Cruz b6cc084716 added really simple progress update in the corner 2017-07-06 17:48:18 -07:00
Stella Lau 3bbfa1249e Update compressor and decompressor 2017-07-06 16:47:08 -07:00
Paul Cruz ff9f2cd057 added some basic logic for altering compression level 2017-07-06 16:06:53 -07:00
Stella Lau b96ad327a4 Add simple compress and decompress functions 2017-07-06 15:23:15 -07:00
Paul Cruz a407ccc215 added ability to congregate statistics into single print statement rather than using debug 2017-07-06 13:09:17 -07:00
Igor Vuk e6e25c9507 Fix typos in README.md 2017-07-06 20:43:14 +02:00
Paul Cruz f57849b9c6 added ability to set initial compression level 2017-07-06 11:05:51 -07:00
Paul Cruz 592a0d9495 changed to work with std out 2017-07-06 10:49:26 -07:00
Paul Cruz 94fe291b83 small changes 2017-07-06 10:29:16 -07:00
Stella Lau 8aa34a7608 Switch to mmapping files 2017-07-06 07:30:49 -07:00
Paul Cruz 79d4657ce5 small changes 2017-07-05 17:44:36 -07:00
Paul Cruz 6f3ad1b22e fixed the problem with pipeline tests by changing how jobs move through the threads 2017-07-05 17:24:21 -07:00
Paul Cruz cc714f3bd3 added print statements and debuglog 2017-07-05 16:54:34 -07:00
Paul Cruz 3f52ca94bf added more tests, changed makefile 2017-07-05 14:36:09 -07:00
Paul Cruz faeb6e0b1b added filenameTable for multiple files 2017-07-05 14:19:56 -07:00
Stella Lau 88f3d8641e Initial long distance matcher commit 2017-07-05 13:57:07 -07:00
Paul Cruz f0b9a153f3 added tests to run.sh 2017-07-05 13:23:34 -07:00
Paul Cruz b42108386a added some basic parsing for args 2017-07-05 12:20:16 -07:00
Paul Cruz 898c1a5b46 removed references to file size computation and file size function 2017-07-05 11:54:21 -07:00
Paul Cruz a2680e5b96 removed calculation of file size and replaced with limited number of available jobs 2017-07-05 11:52:55 -07:00
Paul Cruz dd8a591d5d moved main logic for job creation into a separate function 2017-07-05 10:48:04 -07:00
Paul Cruz 9ccd55f3a8 free ctx fields when error occurs during creation 2017-07-05 10:20:56 -07:00
Paul Cruz 5df4cb0530 renamed files 2017-07-05 09:57:50 -07:00
Paul Cruz c9f49198b8 fixed TODOs 2017-07-05 09:49:27 -07:00
Paul Cruz 9a147d8671 removed unnecessary checks for null pointer on free 2017-07-05 09:37:52 -07:00
Paul Cruz a47ebb1607 removed print statements 2017-07-05 09:23:46 -07:00
Paul Cruz 0b70152a9b working I believe 2017-07-03 20:05:42 -07:00
Paul Cruz cd50382c03 fixed some issues with segfaults 2017-07-03 19:28:48 -07:00
Paul Cruz 95ea54b4cf added code for waiitng for all jobs to finish 2017-07-03 19:24:22 -07:00
Paul Cruz dd447bb9a7 fixed an error with number of jobs 2017-07-03 19:00:55 -07:00
Paul Cruz ff9ac637d9 removed unnecessary files 2017-07-03 17:44:40 -07:00
Paul Cruz dd96efa9ef added print statements for debugging, fixed long memset by changing to calloc 2017-07-03 17:44:22 -07:00
Paul Cruz 0887e98d4b finished main portion of code, now need to debug 2017-07-03 17:28:59 -07:00
Paul Cruz 00b5e6c512 continuing work on v2 2017-07-03 14:18:46 -07:00
Paul Cruz ac79928960 version one complete, can compress a file given input and output names 2017-07-03 14:11:55 -07:00
Nick Terrell ce91b64f00 [linux-kernel] Update patches for v2
* Reduce stack usage of many zstd functions, none use over 388 B anymore.
* Remove an incorrect `const` in `xxhash`.
2017-06-28 22:01:46 -07:00
Nick Terrell f570609286 [pzstd] Fix deadlock in the case of errors
Fixes #720.
2017-06-22 18:09:42 -07:00
Nick Terrell aff86be75d [linux] Write all the patch summaries 2017-06-21 20:23:44 -07:00
Nick Terrell 0a94fead41 [linux] Rename last 2 diffs to patch names 2017-06-21 15:27:07 -07:00
Nick Terrell f3ad4062ba [linux] Add summaries to xxhash and zstd patches 2017-06-21 15:27:07 -07:00
Nick Terrell 34b0a0883a [linux] Rename diffs to patch names 2017-06-21 15:27:07 -07:00
Nick Terrell b789860db0 [linux] Switch BtrFS compression level to 1 2017-06-21 15:27:02 -07:00
Nick Terrell 631d3664b0 [linux] Add xxhash and zstd in-kernel tests 2017-06-19 19:09:41 -07:00
Nick Terrell e7093ca18c [linux] Add libfuzzer targets 2017-06-19 19:09:15 -07:00
Nick Terrell a48e129d48 [linux] Port libzstd UBSAN fix 2017-06-19 15:48:30 -07:00
Nick Terrell 9ad11bea4e [linux] Fix decompression memory allocation 2017-06-19 12:32:04 -07:00
Paul Cruz d75c4ddb1b added a project support notice for VS2005 2017-06-02 10:28:56 -07:00
Paul Cruz 306701a447 adding zstd/zstd.vcproj because it was ignored during initial commit 2017-06-01 21:25:39 -07:00
Paul Cruz 6dc508539c Moved VS2005 to the contrib directory 2017-06-01 21:18:48 -07:00
Nick Terrell 53a67ec1a6 [linux-kernel] Fix duplicate symbols when built-in to kernel 2017-05-25 18:34:13 -07:00
Nick Terrell e63fff9b97 [linux-kernel] Fix clang-format edge case with goto labels 2017-05-25 13:56:46 -07:00
Nick Terrell 74b12f4014 [linux-kernel] Fix some up clang-format edge cases 2017-05-24 13:52:36 -07:00
Nick Terrell ddc68e174b [linux-kernel] Fix a weird clang-format corner case 2017-05-17 15:06:41 -07:00
Nick Terrell a2b21c552f [linux-kernel] Update tests Makefile 2017-05-17 14:56:37 -07:00
Nick Terrell 8d8e65c6c3 [linux-kernel] Update zstd.diff 2017-05-17 14:52:12 -07:00
Nick Terrell 06fa1b0560 [linux-kernel] Move ZSTD_STATIC_ASSERT to zstd_internal.h 2017-05-17 14:51:50 -07:00
Nick Terrell 67101c6c77 [linux-kernel] clang-format lib/zstd/ 2017-05-17 14:40:46 -07:00
Nick Terrell 588579f3a1 [linux-kernel] Rename MEM_* functions to ZSTD_* 2017-05-17 14:39:57 -07:00
Nick Terrell 581e5fbe83 Add clang-format config 2017-05-17 13:49:19 -07:00
Nick Terrell d1a5790f8e [linux-kernel] Dual license with GPLv2 2017-05-17 12:08:15 -07:00
Nick Terrell fb10c91b4f [linux-kernel] Update zstd.diff 2017-05-16 17:21:22 -07:00
Nick Terrell 358280f107 Remove cross-platform macros and dead code 2017-05-16 17:16:00 -07:00
Nick Terrell 8eb7a32527 [linux-kernel] Fix whitespace edge case with script 2017-05-16 16:55:52 -07:00
Nick Terrell 0ce0782c97 [linux-kernel] Update kernelize.sh script for edge cases 2017-05-16 16:55:03 -07:00
Nick Terrell 0d26da2f25 [linux-kernel] Don't undef current (use curr) 2017-05-16 16:48:58 -07:00
Nick Terrell f3c9bd014c [linux-kernel] Remove unused variables in test 2017-05-16 16:13:51 -07:00
Nick Terrell 90a1c94965 [linux-kernel] Update header comments 2017-05-16 16:10:17 -07:00
Nick Terrell 1d42ca8f6d Add zstd.diff 2017-05-16 16:02:05 -07:00
Nick Terrell b3ebe9b77c [linux-kernel] Separate xxhash into its own module 2017-05-16 15:52:35 -07:00
Yann Collet d47709b6ea Merge pull request #654 from iburinoc/splittable
[RFC] Splittable Format and API
2017-05-08 13:41:56 -07:00
Nick Terrell cf4f9403fa [kernel] Update README with SquashFS patch
Take patch from PR #682 by @iburinoc and update benchmarks.
2017-05-03 11:07:39 -07:00
Nick Terrell 99972fa9a8 [btrfs] Fix typo in pr_warn() message 2017-05-02 13:57:40 -07:00
Nick Terrell 79a85e1371 [btrfs] Update patch to use ZSTD_init{C,D}Stream() 2017-05-02 13:41:57 -07:00
Sean Purcell 470993c9b1 Add raw seek table construction API and parallel compression example 2017-04-28 12:17:09 -07:00
Yann Collet ccc0b0eafa Merge pull request #680 from iburinoc/sqfs-kernel
SquashFS linux kernel patch
2017-04-28 10:27:42 -07:00
Sean Purcell c7e107197a Fix missing squashfs_finish_page's 2017-04-27 17:19:20 -07:00
Sean Purcell eeeb201e70 Fix case where pages run out before end of stream 2017-04-27 16:54:05 -07:00
Yann Collet aca392610b Merge pull request #679 from terrelln/dev
[kernel] Rename ZSTD_create*() to ZSTD_init*()
2017-04-27 16:31:03 -07:00
Sean Purcell 32c658a289 SquashFS linux kernel patch 2017-04-27 15:04:31 -07:00
Nick Terrell 0bd5d25d02 [pzstd] Add logging statements to tests 2017-04-27 09:55:19 -07:00
Nick Terrell 5087c1761f Rename ZSTD_create*() to ZSTD_init*() 2017-04-26 13:08:20 -07:00
Sean Purcell 11dc940e72 Add parallel processing example for seekable API 2017-04-21 12:23:06 -07:00
Sean Purcell 35186e65b0 Address comments and make sure all prototypes are rendered by gen_html 2017-04-20 16:48:54 -07:00
Sean Purcell 0f7bd772e6 Update seekable API to simplify IO 2017-04-18 16:48:30 -07:00
Sean Purcell 9626cf1ac6 Address @terrelln's comments 2017-04-13 17:48:35 -07:00
Sean Purcell 2785b28e05 Reduce the limit on frame decompressed size to 2 GB 2017-04-12 14:09:13 -07:00
Sean Purcell 5ee1135f30 s/chunk/frame/ 2017-04-12 11:15:50 -07:00
Sean Purcell e80f1d74b3 Address PR comments and minor fixes 2017-04-12 11:15:46 -07:00
Sean Purcell d048fefef7 Move seekable format content to /contrib 2017-04-11 14:38:56 -07:00
Nick Terrell b633377d0e Add BtrFS benchmarks 2017-04-11 12:40:53 -07:00
Nick Terrell b4dd3378f1 Add BtrFS patch for 4.10 kernel 2017-04-11 11:50:24 -07:00
Nick Terrell adb1974aec Switch comments to kernel style + limit to 80 cols 2017-04-10 16:52:59 -07:00
Nick Terrell bed6c0a4b0 Remove ZSTDLIB_API from decompress.c 2017-04-10 14:49:34 -07:00
Yann Collet b5b79b3942 Merge pull request #647 from terrelln/linux
Add linux kernel modules to contrib/
2017-04-05 07:35:48 -07:00
Nick Terrell 9c7371b3ee Improve comments and ordering 2017-04-04 19:02:28 -07:00