Commit Graph

556 Commits (e649fad7aacebb4df62257fa430f301b4a00ccdd)

Author SHA1 Message Date
Peter (Stig) Edwards 4a9e0502e6
-Wformat-security not needed with -Wformat=2 2019-02-01 09:29:08 +00:00
Peter (Stig) Edwards 2b7120ec71
-Wformat-security not needed with -Wformat=2 2019-02-01 09:28:41 +00:00
Dmitry V. Levin 8b2210411a contrib/pzstd/Makefile: fix build of tests
Apparently, Options.o cannot be linked in without $(PROGDIR)/util.o
2018-12-28 19:02:22 +00:00
Yann Collet ededcfca57 fix confusion between unsigned <-> U32
as suggested in #1441.

generally U32 and unsigned are the same thing,
except when they are not ...

case : 32-bit compilation for MIPS (uint32_t == unsigned long)

A vast majority of transformation consists in transforming U32 into unsigned.
In rare cases, it's the other way around (typically for internal code, such as seeds).

Among a few issues this patches solves :
- some parameters were declared with type `unsigned` in *.h,
  but with type `U32` in their implementation *.c .
- some parameters have type unsigned*,
  but the caller user a pointer to U32 instead.

These fixes are useful.

However, the bulk of changes is about %u formating,
which requires unsigned type,
but generally receives U32 values instead,
often just for brevity (U32 is shorter than unsigned).
These changes are generally minor, or even annoying.

As a consequence, the amount of code changed is larger than I would expect for such a patch.

Testing is also a pain :
it requires manually modifying `mem.h`,
in order to lie about `U32`
and force it to be an `unsigned long` typically.
On a 64-bit system, this will break the equivalence unsigned == U32.
Unfortunately, it will also break a few static_assert(), controlling structure sizes.
So it also requires modifying `debug.h` to make `static_assert()` a noop.
And then reverting these changes.

So it's inconvenient, and as a consequence,
this property is currently not checked during CI tests.
Therefore, these problems can emerge again in the future.

I wonder if it is worth ensuring proper distinction of U32 != unsigned in CI tests.
It's another restriction for coding, adding more frustration during merge tests,
since most platforms don't need this distinction (hence contributor will not see it),
and while this can matter in theory, the number of platforms impacted seems minimal.

Thoughts ?
2018-12-21 18:09:41 -08:00
Yann Collet 34f01e600f fixed multiple conversions
from 64-bit to 32-bit
2018-12-13 14:02:22 -08:00
Yann Collet 9c3265a53f
Merge pull request #1417 from facebook/advancedAPI
Advanced API
2018-12-10 18:48:15 -08:00
Yann Collet 3583d19c4e changed parameter names from ZSTD_p_* to ZSTD_c_*
for naming consistency
2018-12-05 17:26:02 -08:00
Lzu Tao beb13bd87e Move contrib/meson to build/meson 2018-12-01 23:18:59 +07:00
Lzu Tao c0e71cae55 Add enable_lz4 build option and fix lzma dependency 2018-12-01 23:18:59 +07:00
Lzu Tao 5c4965c351 Add pedantic flag 2018-12-01 23:18:59 +07:00
Lzu Tao 6f3f1a8d3a No install zstd_manual.html 2018-12-01 23:18:59 +07:00
Lzu Tao f660825d9f Install missed zstdgrep and zstdless 2018-12-01 23:18:59 +07:00
Lzu Tao 3f27e2a072 Install zstdmt.1 manpage [skip ci] 2018-12-01 23:18:59 +07:00
Lzu Tao d3134a3ed3 Rename meson variables 2018-12-01 23:18:59 +07:00
Lzu Tao 1985e427c7 Add manpage install warning [skip ci]
We link new manpages with gz compressed format of the target manpage.
I have not tested it on Windows. So just place a warning here.
2018-12-01 23:18:59 +07:00
Lzu Tao 9c862c6a53 Fix manpage symlinks [skip ci] 2018-12-01 23:18:59 +07:00
Lzu Tao d79df2a370 Apply new InstallSymlink script 2018-12-01 23:18:59 +07:00
Lzu Tao ef2e761937 Helper script to install symlink in meson 2018-12-01 23:18:59 +07:00
Lzu Tao 3175188407 No need these helpers 2018-12-01 23:18:59 +07:00
Lzu Tao 337f914dc8 Fix lib soversion and no install cover.h header 2018-12-01 23:18:59 +07:00
Lzu Tao c9f0144302 Fix meson tests build 2018-12-01 23:18:59 +07:00
Lzu Tao 5a36a57cf5 Bump to 1.3.8 and fix run_command function
The run_command is run from an unspecified directory. Therefore we cannot assume
which directory it is running our command.
2018-12-01 23:18:59 +07:00
Lzu Tao 8a160680d1 Update legacy support to 5 2018-12-01 23:18:59 +07:00
Lzu Tao f727808731 Minor fix for meson build
Use files function instead of constructing path with meson.current_source_dir()
2018-12-01 23:18:59 +07:00
Lzu Tao 9a721e5216 Update meson build system
NOTE: This commit only tested on Linux (Ubuntu 18.04). Windows
build may not work as expected.

* Use meson >= 0.47.0 cause we use install_man function
* Add three helper Python script:
  * CopyFile.py: To copy file
  * CreateSymlink.py: To make symlink (both Windows and Unix)
  * GetZstdLibraryVersion.py: Parse lib/zstd.h to get zstd version
  These help emulating equivalent functions in CMake and Makefile.
* Use subdir from meson to split meson.build
  * Add contrib build
  * Fix other build
* Add new build options
  * build_programs: Enable programs build
  * build_contrib: Enable contrib build
  * build_tests: Enable tests build
  * use_static_runtime: Link to static run-time libraries on MSVC
  * zlib_support: Enable zlib support
  * lzma_support: Enable lzma support
2018-11-28 01:08:34 +07:00
Lzu Tao 9bd8f6a00c Rename and update build instruction in README file to README.md 2018-11-28 01:08:34 +07:00
Lzu Tao 2abd5139a5 Add meson build guide 2018-11-28 01:08:34 +07:00
Yann Collet 5adbad4059 Merge branch 'dev' into advancedAPI 2018-11-14 13:00:37 -08:00
Yann Collet b83d1e7714 removed some `static const` variables
and replaced by traditional macro constants.

Unfortunately, C doesn't consider `static const` to mean "constant"
2018-11-13 16:56:32 -08:00
Yann Collet b830ccca5c changed benchfn api
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
2018-11-13 13:12:50 -08:00
Yann Collet d38063f8ae separated bench module into benchfn and benchzstd
it shall be possible to use benchfn
without any dependency on zstd.
2018-11-13 11:01:59 -08:00
Yann Collet 483759a3de Improves decompression speed when using cold dictionary
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).

Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)

The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.

This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).

Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
2018-11-08 17:00:23 -08:00
Rohit Jain 705e0b18ab Making changes to make it compile on my laptop 2018-10-11 15:51:57 -07:00
Yann Collet 123fac6b6d fix pzstd compatibility with mingw
some details changed with introduction of gcc7
2018-09-21 17:36:00 -07:00
Yann Collet 00ce26725b
Merge pull request #1324 from ko-zu/fixclangcode
Fix largeNbDicts bench for clangbuild
2018-09-17 14:10:17 -07:00
Nick Terrell 8f27e8cf3d
Merge pull request #1322 from azat-archive/seekable-fixes-pull
Fixes read write past end of input buffer.
2018-09-17 11:04:51 -07:00
ko-zu b053bec2f4 Fix largeNbDicts bench for clangbuild
Remove unsigned to size_t promotion to fix implicit down conversion errors in clangbuild target.
2018-09-17 13:09:08 +09:00
Azat Khuzhin d707692e05
seekable_decompression: support offset greater then UNIT_MAX 2018-09-16 18:05:32 +03:00
Azat Khuzhin b52867a97f
zstdseek_decompress: fix decompression with data left in input buffer 2018-09-16 18:05:32 +03:00
Yann Collet c49ccbc8e7 largeNbDicts : can select a nb of blocks
will automatically truncate or repeat input as needed,
to create the requested nb of blocks.
default: nb of files, eventually increased appropriately if blockSize is set
2018-09-12 11:31:28 -07:00
Yann Collet 50b216146f
Merge pull request #1304 from facebook/largeNbDicts
contrib/largeNbDicts
2018-09-06 09:50:56 -07:00
Yann Collet c57a856d64 fixed minor static analyzer warning 2018-09-05 14:33:51 -07:00
Yann Collet 1d487d587f updated documentation 2018-09-04 14:57:45 -07:00
Yann Collet 11b8b8c100 silenced false-positive scan-build warning 2018-08-31 10:01:06 -07:00
Yann Collet 0ff67511e6 fixed link order for old compilers 2018-08-30 16:43:28 -07:00
Yann Collet f76253bb70 minor : createDictionaryBuffer() can create dictionaries of different sizes 2018-08-30 16:24:44 -07:00
Yann Collet 39c55a118f fixed minor compatibility issues with older compilers 2018-08-30 16:00:57 -07:00
Yann Collet 39ef91a599 -std=c99 for largeNbDicts 2018-08-30 14:59:23 -07:00
Yann Collet 4086b2871b largeNbDicts compatible with multiple source files
splitting is disabled by default, but can be re-enabled using usual command -B#
update commands to look like zstd ones
2018-08-30 14:38:49 -07:00
Yann Collet a5a77965d3 make all includes contrib/largeNbDicts 2018-08-29 16:17:22 -07:00