diff --git a/pages/Persistence.md b/pages/Persistence.md new file mode 100644 index 0000000..b165ca8 --- /dev/null +++ b/pages/Persistence.md @@ -0,0 +1,192 @@ +This page aims to give an overview of the serialization options Minetest provides. + +Your choice for persistence depends on the data you want to persist: + +* Structure & types of the data: How is the data structured? What data types occur? +* Size of the data, frequency of changes to the data: Can it fit into memory? How expensive can updates be? +* Granularity needed: How often must the data be persisted? +* Required queryability of the data: Are indices for fast lookups required? +* Required data access, objects the data is tied to: Items? Players? + +[toc] + +## (De-)Serialization + +### JSON + +Minetest provides a JSON serializer based on `jsoncpp`. + +The following types are supported by JSON: + +* booleans +* numbers +* strings +* tables (objects/arrays) + +NOTE: Negative and positive infinity are represented by numbers which exceed the double number range by a lot (plus/minus `1e9999`). + +WARNING: The empty table `{}` (ambiguous: might be either `{}` or `[]`) and `nan` aren't supported by JSON and will be turned into `null`. + +Tables may either: +1. Contain only positive integer keys (represented as array) or +2. Contain only string keys (represented as object/dictionary) +obviously, plenty of Lua tables aren't representable this way (boolean keys, mixed hash/list keys, fractional keys, inf keys) + +#### `minetest.write_json(data, [styled])` + +If the data is serializable as outlined above:: Returns a JSON string representing `data`. Formats the JSON nicely to improve readability if `styled` is truthy. +Else:: Returns `nil` and one of the following errors if it fails: +* `Can't use indexes with fractional part in JSON` +* `Lua key to convert to JSON is not a string or number` +* `Can't mix array and object values in JSON` + +WARNING: Hash tables containing only positive integer keys - a table of `minetest.hash_node_position` hashes for instance - +will be turned into *an array with holes*, that is, all `nil` values from `1` to the maximum index will be filled with `null`, +which means terrible performance and possibly very large output generated from very small input. +*Do not allow direct access to `minetest.write_json`* as it can be trivially used to DoS your server. + +TIP: Sparse hash maps, should always be converted to objects, using only string keys, for JSON. You might do this just for serialization. + +WARNING: The JSON serializer is furthermore limited by its recursion depth: Only a recursion depth up to *16* is allowed. Very nested data structures can't be (de)serialized. Circular references will cause the JSON serializer to recurse until the maximum recursion depth is exceeded. + +TIP: Use the `json = assert(minetest.write_json(data))` idiom to not silently ignore errors when serializing. + +#### `minetest.parse_json(json, [nullvalue])` + +If `json` is valid JSON:: Returns the Lua value represented by the JSON string `json`, with `null` values deserialized to `nullvalue` (which defaults to `nil`). +Else:: Returns `nil` and logs an error. + +TIP: If you must use JSON (to interact with a web interface, for instance), consider using your JSON library of choice; +Lua-only implementations such as [`lunajson`](https://github.com/grafi-tt/lunajson/) are available. + +### Lua + +Minetest alternatively offers a serializer implemented in Lua, which serializes to Lua source code, available as `minetest.serialize` and `minetest.deserialize`. The following values are supported: + +* `nil` +* booleans +* numbers +* strings +* tables, including circular references +* functions: *not recommended*; uses `string.dump` internally + * Deserialization of functions will error unless mod security is disabled + * Lua-implementation and platform-specific bytecode: Functions serialized by PUC Lua won't work on LuaJIT and vice versa; no cross-platform portability + * Upvalues or the context of the function (function environment) aren't preserved + * C functions which lack Lua bytecode (such as `math.sqrt` or most Minetest API functions) can't be (de)serialized at all + +Userdata objects (like `ItemStack` for example) aren't supported. Threads (coroutines) aren't supported either. + +#### `minetest.serialize(data)` + +Serializes `data`, which may be a value of any of the above types. + +Returns a Lua chunk in `string` form that returns `data` if executed. + +Will error with `Can't serialize data of type ` if it encounters an unsupported type. + +Fully supports circular references. + +#### `minetest.deserialize(str, [safe])` + +If `str` is not of type `string`:: Returns `nil, "Cannot deserialize type ''. Argument must be a string."` +If `str` starts with `"\27"`:: Returns `nil, "Bytecode prohibited"` +If `str` triggers a Lua error at run- or load-time:: Returns `nil, error` +If `str` returns without errors:: Returns the first value returned by executing the chunk `str` without arguments + +If `safe` is truthy, serialized functions will be deserialized to `nil`. +This will trigger an error if functions are used as table keys (`{[function()end] = true}`). +Otherwise, serialized functions will get an empty function environment set - only being able to operate on literals and arguments. + +TIP: Use of the `data = assert(minetest.deserialize(lua, safe))` idiom is recommended. + +WARNING: [`minetest.deserialize` errors on large objects on LuaJIT](https://github.com/minetest/minetest/issues/7574) + +## Engine-provided default persistence + +Nodes (consisting of nodename, param1, param2, meta) are persisted automatically as part of mapblocks. +A handful of player properties (HP, breath, position, look direction, inventory, meta) are persisted as well. +Granularity is controlled by the `server_map_save_interval` setting. + +## Storage options + +### Database server / the ominous cloud + +You can use Minetest's HTTP library to communicate with webservers, which might store data for you. + +Other ways of Inter-Process Communication that can be leveraged to communicate with a database include *sockets*, +provided through the `luasockets` library (requiring an insecure environment and an accessible installation). +If the database server runs on the same machine, you might decide to use file bridges for IPC. + +### Lightweight database library + +Requires an insecure environment and an installation of the database library that is accessible to Minetest. +SQLite3, available through the `lsqlite3` luarocks package, +is a popular choice here and used for instance by the [sban](https://github.com/shivajiva101/sban) mod. + +### String stores + +#### Entity staticdata + +Tied to entities. The serialized string must be returned by `get_staticdata` and is passed to `on_activate`. + +#### File store + +Usually tied to world or mod paths. The simplest approach reads the file at load time and writes it on shutdown. +As `on_shutdown` may however not be called in the case of a crash - +or even worse, a power outage might abruptly shut down the server without calling anything - +this provides a rather poor granularity, as all changes to the data during the uptime may be lost. + +You may simply serialize your data and write it to a file on every update. +If your data is rather larger or gets updated frequently, a full serialization might negatively impact performance. +Performance can be improved at the expense of granularity by saving periodically and choosing "long" periods. +A transaction log improves performance by only storing changes, at the expense of disk space. + +TIP: A mix of both approaches can provide satisfying results, logging only changes and rewriting the logfile periodically to keep disk space waste acceptable. + +For special cases like logging, an append-only file may be the ideal solution if using the global `minetest.log` is not desirable. + +### Key-value store + +#### Filesystem + +On systems that provide a decent filesystem implementation (that is, everything except Windows), +you can use filenames/filepaths as keys and files as values. +On poor filesystems, you might be heavily limited by absolute path character limits; +lots of small files might lead to fragmentation. + +A nested hierarchical key-value store is possible through directory structures, which can be managed and traversed using: + +* `minetest.mkdir` +* `minetest.rmdir` +* `minetest.cpdir` +* `minetest.mvdir` +* `minetest.get_dir_list` + +If you want to mitigate the risk of data loss, you can use `minetest.safe_file_write` when (re)writing files. + +#### Configuration files + +The `Settings` object allows you to operate on configuration files, getting & setting key-value entries and saving the file. +The main `Settings` object `minetest.settings` can be used to persist a few settings "globally" - bleeding everywhere. +This is horribly abused by the mainmenu to store stuff like the last selected game. Don't be like the mainmenu; +distinguish persistent game data and settings/configuration properly and _namespace_ your data properly. + +That said, `Settings` can be used to read from & write to simple & limited key-value store files. +The main advantage of them is that they are presumably easy to edit for users, +but this is usually not a requirement for game data which is "edited" by in-game interactions; +indeed, you might not want to tempt players to cheat in singleplayer by editing their "saves". + +#### MetaData + +Minetest provides metadata objects which all provide a simple string key-value store, tied to four different game "objects": + +1. ItemStacks: `ItemStackMetaData`: Fully sent to clients; serialized within inventories, which may be serialized within mapblocks +2. Node positions: `NodeMetaData`: Sent to clients, but fields can be marked as private; serialized somewhere within mapblocks +3. Players: `PlayerMetaData`: SQLite-backed key-value storage, however only available while the player is online +4. Mods: `ModStorage`: SQLite-backed key-value storage; older versions use a JSON-backed store unsuitable for large data (as serialization will block the main thread) + +Utilities for setting & getting non-string datatypes like integers and floats are provided; the datatype is however not stored with the entries. +The granularity of all these key-value stores is determined by the `server_map_save_interval` setting. + +--- +*This article is originally based on an article from the minetest_docs project: [persistence.adoc](https://github.com/minetest/minetest_docs/blob/master/doc/persistence.adoc) by Lars Müller, licensed under CC-BY 4.0* \ No newline at end of file