senhuang42
e0f26afce9
Add sequence compression format param
2020-11-16 10:49:16 -05:00
senhuang42
f51af9a609
Always ensure sequenceRange updates properly, add more error forwarding
2020-11-16 10:49:16 -05:00
senhuang42
1a449688fd
Various minor logical refactors to improve clarity
2020-11-16 10:49:16 -05:00
senhuang42
e5fe485dcc
Fix cSize calculation for noCompressBlocks
2020-11-16 10:49:16 -05:00
senhuang42
6145ebb400
Rebased, roundtrips silesia.tar
2020-11-16 10:49:16 -05:00
senhuang42
b5b61cc216
Refactor for better debugging info
2020-11-16 10:49:16 -05:00
senhuang42
293fad6b45
Corrections and edge-case fixes to be able to roundtrip dickens
2020-11-16 10:49:16 -05:00
senhuang42
7eb6fa7be4
Multi-block compression scaffolding - works on single-block files
2020-11-16 10:49:16 -05:00
senhuang42
75b01f34b9
Add support for uncompressible blocks
2020-11-16 10:49:16 -05:00
senhuang42
e04da68157
Enable usage of ZSTD_sequenceRange for single-block compression
2020-11-16 10:49:16 -05:00
senhuang42
337fac216d
Add logic to handle ZSTD_sequenceRange
2020-11-16 10:49:16 -05:00
senhuang42
85822ddd53
Add last literals handling like getSequences()
2020-11-16 10:49:16 -05:00
senhuang42
2cff8df1a2
Pull block compression out of main compressSequences() function
2020-11-16 10:49:16 -05:00
senhuang42
cfced9344a
Implement ZSTD_updateSequenceRange
2020-11-16 10:49:16 -05:00
senhuang42
b116e1f211
Modify SequenceRange to have posInSequence
2020-11-16 10:49:16 -05:00
senhuang42
d99b675112
Add function definition for sequenceRange updater
2020-11-16 10:49:16 -05:00
senhuang42
74e95c05cc
Add ZSTD_SequenceRange to count ranges in array of ZSTD_Sequence
2020-11-16 10:49:16 -05:00
senhuang42
89f3848310
Add support for repcodes
2020-11-16 10:49:16 -05:00
senhuang42
3e930fd044
Code cleanup, add debuglog statments
2020-11-16 10:49:16 -05:00
senhuang42
086513b5b9
Implement first pass at compressSequences()
2020-11-16 10:49:16 -05:00
senhuang42
a9327b1e9b
Add initial function prototype for ZSTD_compressSequences_ext (to be renamed later)
2020-11-16 10:33:35 -05:00
animalize
52f8c07a3f
Clamp compression level in ZSTD_getCParams_internal() function
2020-11-14 13:26:08 +08:00
senhuang42
9d936d61d2
Reduce number of memcpy() calls
2020-11-13 19:43:30 -05:00
senhuang42
be4ac6c5bc
Use existing repcode update function to implement updates
2020-11-12 16:51:12 -05:00
senhuang42
674c9b9235
Add in proper block repcode histories
2020-11-12 15:34:37 -05:00
senhuang42
06c7f14066
Let block reps persist
2020-11-12 12:24:44 -05:00
senhuang42
396275068c
Fix incorrect repcode setting
2020-11-12 11:57:01 -05:00
senhuang42
1a8af0de73
Improve unit test
2020-11-12 11:09:09 -05:00
senhuang42
4d4fd2c55f
Overhaul repcode handling logic
2020-11-12 10:59:35 -05:00
sen
f62edf0fe9
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api
...
Add enum to define ZSTD_Sequence type and update sequence extraction API
2020-11-06 13:00:31 -05:00
senhuang42
7d1dea070c
Update unit tests
2020-11-06 11:10:37 -05:00
senhuang42
779df995c6
Implement mergeGeneratedSequences()
2020-11-06 10:55:46 -05:00
senhuang42
51abd58208
Rename getSequences() to generateSequences()
2020-11-06 10:53:22 -05:00
Luke Pitt
eac309c71b
Add ZSTD_getDictID_fromCDict function to experimental section
2020-11-04 11:37:37 +00:00
senhuang42
f782cac3d4
Change block delimiter removing to linear time approach
2020-11-02 17:06:20 -05:00
senhuang42
3434049c1f
Use ZSTD_memmove() instead of memmove()
2020-11-02 11:43:19 -05:00
senhuang42
d4d0346b40
Update name of enum, clarify documentation
2020-11-02 11:38:17 -05:00
senhuang42
e6178f837f
Revert unnecessary seqCollector adjustment
2020-11-02 10:59:20 -05:00
senhuang42
e8501e00b8
Fix incorrect index increment in merge algorithm
2020-11-02 10:58:41 -05:00
senhuang42
a36fdada57
Add algorithm to remove all delimiters
2020-11-02 10:46:52 -05:00
senhuang42
435a3a0428
Update seqCollector definition
2020-11-02 10:19:26 -05:00
senhuang42
3327932609
Update ZSTD_getSequences function signature
2020-11-02 10:17:59 -05:00
Nick Terrell
7205e609a9
Merge pull request #2354 from terrelln/stable-buffer
...
Add ZSTD_c_stable{In,Out}Buffer and optimize when set
2020-10-30 15:06:56 -07:00
sen
c37c714ef1
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api
...
Refine external ZSTD_Sequence API
2020-10-30 15:47:25 -04:00
Nick Terrell
d4e021fe35
[lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set
...
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
2020-10-30 10:55:34 -07:00
Nick Terrell
24f72789e2
[lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set
...
Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.
2020-10-30 10:55:34 -07:00
Nick Terrell
fcf81cee5e
[lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set
...
We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.
2020-10-30 10:55:34 -07:00
Nick Terrell
6d5dc93d4e
[lib] Compress directly into output when ZSTD_c_stableOutBuffer is set
...
When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.
2020-10-30 10:55:34 -07:00
Nick Terrell
987cb4ca6a
[lib] Take the shortcut when ZSTD_c_stableOutBuffer is set
...
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
2020-10-30 10:55:34 -07:00
Nick Terrell
809b2f2071
[lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2()
...
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.
An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
2020-10-30 10:55:34 -07:00
