buildat/doc/design.txt

Buildat
=======
A minecraftlike with vast extendability.

License: Apache 2.0

Client
------
Built using Urho3D.

Module code is transfered from the server and run in a safe Lua sandbox.

Extensions are non-sandboxed code installed separately on each client.

Server
------
Built using C++, with suitable parts from Urho3D, with most functionality in
runtime-compiled C++ modules.

Module structure
----------------
module
|-- deps.txt      << Module and extension dependencies
|-- <module>.cpp  << Server-side code
|-- api.h         << Structures for interfacing between modules
|-- client_lua
|   `-- init.lua  << Client-side code (by convention)
`-- client_data
    `-- media.png << Data files (by convention)

Module behavior
---------------
No script or data transfer to the client is initiated by the core. Conventions
followed by builtin modules:
- module/client_lua/{init,*}.lua - builtin/client_lua
- module/client_data/*           - builtin/client_data

Modules can be unloaded at runtime. Handling of client-side state is left up to
the C++ modules themselves.

The first module to be loaded is called __loader. It loads all other modules.

C++ modules can use the core/ and interface/ headers. Everything else is
considered unstable.

C++ modules are run in threads, and everything they can officially access is
thread-safe.

C++ modules can provide direct library functionality inlined in their include/
files. See builtin/network as an example.

Startup sequence and what the module should do:
- constructor     : Don't access other modules. Throw on fatal errors.
- init()          : Subscribe to events; access other external things.
- "core:start"    : Start doing whatever the module wants to actively do.
- "core:unload"   : Module will be unloaded immediately after event handler.
- "core:continue" : Continue doing stuff after a reload.

Metainformation: meta.json
-------------------------
Example:
{
	"cxxflags": "",
	"ldflags": "-lsasl2",
	"dependencies": [
		{"module": "network"},
		{"module": "plants", "optional": true},
	],
	"reverse_dependencies": [
		{"module": "stuff", "optional": true},
	],
}

Any fields can be left out. The minimum meta.json content is an empty object {}.

Extension structure
-------------------
extension
`-- init.lua  << Loaded when the module is required
`-- init.cpp  << Compiled as a Lua module and loaded if init.lua doesn't exist

Extension behavior
------------------
Extensions use the new Lua 5.1/5.2 module interface.

If an extension wish to provide an interface to sandboxed code, it should
implement table "safe", which contains the safe interface.

Extensions and modules use require "buildat/extension/<name>" to use extensions.

The __menu extension is specially loaded automatically at client startup if no
server address is provided on the command line. __menu can then connect to a
server. When disconnecting from a server, the whole client window is closed and
reopened.

Network protocol
----------------
(Type, length, data) tuples on TCP. In the future TLS can be taken into use. A
name->type registry is used for determining numeric packet types.

Data is freeform. Types 0...99 are reserved for initialization.

Core uses cereal's portable binary serialization, except for low-level packet
streaming.

Voxels
------
A way to define voxel properties:
- Polyvox can output about 2**23 = 8388608 different face materials (uint32_t ->
  float conversion)
- So given that each voxel has six faces, we can have about 8388608/6 =
  1398101 (~2**20) different voxels
- Pseudorandom selection of voxel textures doesn't lower this value; they
  can be randomized at the texture coordinate generation phase (which are
  generated from face materials at known positions in the mesh)
- If voxels can be rotated, it directly lowers this value according to the
  number of possible positions; if there are 4*6 rotational positions, we
  can have 1398101/24 = 58254 different voxels
- It probably doesn't make sense to distinguish rotation from voxel type in
  volume data, which means the voxel type namespace size is 1398101
- If a block spans multiple voxels, each of those is a distinct voxel type
- This means that there can be 1398101/24/8 = 7281 distinct
  rotatable 8-voxel blocks
- A voxel instance only contains a 21-bit type id stored as the lowest bits
  of an uint32_t; highest bits are reserved and should be ignored.
- It is not possible to add game-specific data to voxels, but games can
  utilize the 11 MSBs of the uint32_t
	- Adding a field with a runtime-specified size is impossible in
	  template-based PolyVox
	- It is recommended that games only use the 8 MSBs. The rest 3 are
	  reserved for future development of the engine.
- In-memory storage:
	- PolyVox::SimpleVolume<interface::VoxelInstance>
	- Anything like PolyVox::LargeVolume isn't useful because the world will not
	  be handled as a large contiguous space, but instead as many nodes that
	  each contain a chunk of voxels. Custom-made in-memory compression can be
	  added to them later and is not needed now. (Deinterlaced run-length
	  encoding according to the most common block dimensions will probably work
	  well.)
- On-disk storage:
	- Zlib is fine; probably a compression level of 6 is fine. Tests show that
	  levels 1..3 perform poorly with this kind of data.
- Special voxels are just regular voxels, but with a handler module name defined
  as a property, into which the engine calls at construction and destruction
  time, giving the generated node, voxel type id and position as parameters
	- A torch could be similar to air, but the handler would create a child
	  node with a model and a light at its position
- Basic voxel types are defined by buildat:
	- interface::VoxelName
	- interface::VoxelTypeId (uint32_t)
	- interface::VoxelDefinition
	- interface::VoxelRegistry
	- interface::VoxelInstance
- How are voxel texture definitions stored?
	- They are references to one of many texture atlases (many are needed in
	  case of high-resolution textures)
	- They are an index into a texture atlas id
	- A texture atlas definition is a list of texture resource names
	- One texture atlas contains textures of one resolution

A way to define block properties:
- interface::-based because the client has to support blocks directly:
	- interface::BlockName (typedef ss_)
	- interface::BlockTypeId (uint32_t)
	- interface::BlockDefinition
	- interface::BlockRegistry
- Definitions refer to voxel definitions
- In practice, voxel definitions are generated by definining a block that needs
  them
- Can be defined to be rotatable or not
- Should there be an option to make a block rotatable only to 4 positions?
- Saved volume data should be designed so that if a block's properties are
  modified, it will not be invalidated; eg. if a block is changed to not be
  rotatable and be of different voxel size, each of the voxels can be loaded
  as their non-rotated counterparts as some suitable block segment
	- This can be done by resolving IDs from namespaced names:
	  "dirt;s=0,1,1;r=2;R=5"
	- Or just have a VoxelName type which consists of these partitions:
		- block_name: Name of the block this was instanced from
		- segment.{x,y,z}: Which segment of the block this was instanced from
		- rotation_primary: 4 possible rotations when looking at a face
		- rotation_secondary: 6 possible directions for a face to point to

The voxel world:
- builtin/voxelworld handles most things on the server and the client
- The client contains built-in helpers for resource-intensive tasks
- VoxelRegistry: owned by builtin/voxelworld
	- Yes, on the client too
	- The main world registry is called "main"
	- An identically named block registry also always exists
	- For now, voxel registries are synced only when a new client connects
	- If these registries are synchronized in the background, it probably is not
	  feasible to allow making anonymous ones
- Voxel data is always just a string in a user variable in each node
	- Nodes contain:
		- the used voxel registry name ("buildat_voxel_registry_name")
		- the data itself              ("buildat_voxel_data")
		- the used data format         ("buildat_voxel_data_format")
		- data modification version    ("buildat_voxel_mod_version")
	- The data can be raw or compresed, and it can be cached by node id and data
	  modification version
- The client is allowed to see all voxel data for each node that gets synced
  over network.
	- Data can be hidden in local components or other storage on the server.
- Stationary and non-stationary voxel nodes are stored like any node in the
  world
	- How can the world be sanely loaded when it only consists of arbitrarily
	  positioned nodes?
		- Maybe all nodes, including the chunk nodes, could be grouped in
		  sections when saved on disk
- Terminology:
	- "static node": Stationary voxel-aligned node used for main world content
	- "dynamic node": Freely positioned node (containing voxels)
	- "static voxel": Voxel stored in a static node
	- "dynamic voxel": Voxel stored in a dynamic node
- Save format tables:
	- section_size(w, h, d) <- only one row
	- section(sx, sy, sz, save_enabled, generated)
	- node(sx, sy, sz, data) <- (sx, sy, sz) are section coordinates
- How to handle references from nodes to other nodes?
	- Using each node id only once in a world's lifetime is not feasible: If
	  60 nodes are created per second, the networked namespace lasts only
	  for 2**24/60/3600/24 = 3.2 days (FIRST_LOCAL_ID = 0x01000000)
	- The save format uses a 64-bit node id namespace, which is converted to and
	  from the in-memory 24-bit id namespace.
		- The only way to implement this is to integrate this into Urho3D's
		  Node's Get/SetNetParentAttr() because it contains the node hierarchy
		  serialization code.
			- Basically add Node::Get/SetPersistentID() and Node::persistent_id_
			- Also the external parts (reference implementation in Connection)
			  need to be altered
			- 64-bit ids will be used over the network too to simplify things
	- What happens when loading a node, but not a node that it refers to?
		- Nodes can be added to the scene without adding any components or
		  setting any attributes
		- A disk-to-memory id mapping can be maintained in memory so that nodes
		  loaded at different times can be connected
			- In practice, these are stored in Urho3D's Nodes as
			  Node::persistent_id_
- Section storage:
	- By default, generated sections are save-disabled. If a game determines
	  (eg. from player actions) that a section should be saved, it explicitly
	  tells it to builtin/voxelworld, which then marks the section as
	  save-enabled. This flag has multiple states: it is temporarily enabled if
	  the section contains a node that has a save-enabled flag set. In this case
	  the whole section is save-enabled during the time that the node is located
	  inside the section.
	- Overshooting trees and other pseudo objects have to be saved as extra
	  static nodes if they cross the edges of the section
	- Save-enabled sections in the memory are saved. Others are discraded.
	- When saving a section:
		- The section properties are saved
		- All nodes in the section are saved
	- When loading a section:
		- The section properties are loaded
		- All nodes in the section are loaded
	- When generating a section:
		- The "voxelworld:generation_request" event is emitted
- Methods:
	- set_voxel(p, v)
		- Set a static voxel
- World generation does not use any special interface because the same things
  should be possible without being triggered by builtin/voxelworld
	- Just send an event; "voxelworld:generation_request"
	- A generation accelerator module can exist to speed up generation by using
	  a single buffer (PolyVox::RawVolume<uint32_t>) and allowing direct
	  registration of generation callbacks. It could also automate the creation
	  of extra static nodes when needed by padding the buffer enough for common
	  things.