buildat/doc/design.txt

Buildat
=======
A minecraftlike with vast extendability.

License: Apache 2.0

Client
------
Built using Urho3D.

Module code is transfered from the server and run in a safe Lua sandbox.

Extensions are non-sandboxed code installed separately on each client.

Server
------
Built using C++, with suitable parts from Urho3D, with most functionality in
runtime-compiled C++ modules.

Module structure
----------------
module
|-- deps.txt      << Module and extension dependencies
|-- <module>.cpp  << Server-side code
|-- api.h         << Structures for interfacing between modules
|-- client_lua
|   `-- init.lua  << Client-side code (by convention)
`-- client_data
    `-- media.png << Data files (by convention)

Module behavior
---------------
No script or data transfer to the client is initiated by the core. Conventions
followed by builtin modules:
- module/client_lua/{init,*}.lua - builtin/client_lua
- module/client_data/*           - builtin/client_data

Modules can be unloaded at runtime. Handling of client-side state is left up to
the C++ modules themselves.

The first module to be loaded is called __loader. It loads all other modules.

C++ modules can use the core/ and interface/ headers. Everything else is
considered unstable.

C++ modules are run in threads, and everything they can officially access is
thread-safe.

C++ modules can provide direct library functionality inlined in their include/
files. See builtin/network as an example.

Startup sequence and what the module should do:
- constructor     : Don't access other modules. Throw on fatal errors.
- init()          : Subscribe to events; access other external things.
- "core:start"    : Start doing whatever the module wants to actively do.
- "core:unload"   : Module will be unloaded immediately after event handler.
- "core:continue" : Continue doing stuff after a reload.

Metainformation: meta.json
-------------------------
Example:
{
	"cxxflags": "",
	"ldflags": "-lsasl2",
	"dependencies": [
		{"module": "network"},
		{"module": "plants", "optional": true},
	],
	"reverse_dependencies": [
		{"module": "stuff", "optional": true},
	],
}

Any fields can be left out. The minimum meta.json content is an empty object {}.

Extension structure
-------------------
extension
`-- init.lua  << Loaded when the module is required
`-- init.cpp  << Compiled as a Lua module and loaded if init.lua doesn't exist

Extension behavior
------------------
Extensions use the new Lua 5.1/5.2 module interface.

If an extension wish to provide an interface to sandboxed code, it should
implement table "safe", which contains the safe interface.

Extensions and modules use require "buildat/extension/<name>" to use extensions.

The __menu extension is specially loaded automatically at client startup if no
server address is provided on the command line. __menu can then connect to a
server. When disconnecting from a server, the whole client window is closed and
reopened.

Network protocol
----------------
(Type, length, data) tuples on TCP. In the future TLS can be taken into use. A
name->type registry is used for determining numeric packet types.

Data is freeform. Types 0...99 are reserved for initialization.

Core uses cereal's portable binary serialization, except for low-level packet
streaming.

Voxels
------
Terminology:
- "voxel": Low-cost thing using the space of a cube, stored in large amounts
- "block": Defines an arrangement of voxels (often 1x1x1)
- "volume": 3-dimensional volume of voxels
- "node": Scene node
- "static node": Stationary voxel-aligned node containing a volume, used for
  containing most of the generated world
- "chunk": The volume in a static node
- "dynamic node": Freely positioned node containing a volume

A way to define voxel properties:
- Polyvox can output about 2**23 = 8388608 different face materials (uint32_t ->
  float conversion)
- So given that each voxel has six faces, we can have about 8388608/6 =
  1398101 (~2**20) different voxels
- Pseudorandom selection of voxel textures doesn't lower this value; they
  can be randomized at the texture coordinate generation phase (which are
  generated from face materials at known positions in the mesh)
- If voxels can be rotated, it directly lowers this value according to the
  number of possible positions; if there are 4*6 rotational positions, we
  can have 1398101/24 = 58254 different voxels
- It probably doesn't make sense to distinguish rotation from voxel type in
  volume data, which means the voxel type namespace size is 1398101
- If a block spans multiple voxels, each of those is a distinct voxel type
- This means that there can be 1398101/24/8 = 7281 distinct
  rotatable 8-voxel blocks
- A voxel instance only contains a 21-bit type id stored as the lowest bits
  of an uint32_t; highest bits are reserved and should be ignored.
- It is not possible to add game-specific data to voxels, but games can
  utilize the 11 MSBs of the uint32_t
	- Adding a field with a runtime-specified size is impossible in
	  template-based PolyVox
	- It is recommended that games only use the 8 MSBs. The rest 3 are
	  reserved for future development of the engine.
- In-memory storage:
	- PolyVox::SimpleVolume<interface::VoxelInstance>
	- Anything like PolyVox::LargeVolume isn't useful because the world will not
	  be handled as a large contiguous space, but instead as many nodes that
	  each contain a chunk of voxels. Custom-made in-memory compression can be
	  added to them later and is not needed now. (Deinterlaced run-length
	  encoding according to the most common block dimensions will probably work
	  well.)
- On-disk storage:
	- Zlib is fine; probably a compression level of 6 is fine. Tests show that
	  levels 1..3 perform poorly with this kind of data.
- Special voxels are just regular voxels, but with a handler module name defined
  as a property, into which the engine calls at construction and destruction
  time, giving the generated node, voxel type id and position as parameters
	- A torch could be similar to air, but the handler would create a child
	  node with a model and a light at its position
- Basic voxel types are defined by buildat:
	- interface::VoxelName
	- interface::VoxelTypeId (uint32_t)
	- interface::VoxelDefinition
	- interface::VoxelRegistry
	- interface::VoxelInstance
- How are voxel texture definitions stored?
	- They are references to one of many texture atlases (many are needed in
	  case of high-resolution textures)
	- They are an index into a texture atlas id
	- A texture atlas definition is a list of texture resource names
	- One texture atlas contains textures of one resolution

A way to define block properties:
- interface::-based because the client has to support blocks directly:
	- interface::BlockName (typedef ss_)
	- interface::BlockTypeId (uint32_t)
	- interface::BlockDefinition
	- interface::BlockRegistry
- Definitions refer to voxel definitions
- In practice, voxel definitions are generated by definining a block that needs
  them
- Can be defined to be rotatable or not
- Should there be an option to make a block rotatable only to 4 positions?
- Saved volume data should be designed so that if a block's properties are
  modified, it will not be invalidated; eg. if a block is changed to not be
  rotatable and be of different voxel size, each of the voxels can be loaded
  as their non-rotated counterparts as some suitable block segment
	- This can be done by resolving IDs from namespaced names:
	  "dirt;s=0,1,1;r=2;R=5"
	- Or just have a VoxelName type which consists of these partitions:
		- block_name: Name of the block this was instanced from
		- segment.{x,y,z}: Which segment of the block this was instanced from
		- rotation_primary: 4 possible rotations when looking at a face
		- rotation_secondary: 6 possible directions for a face to point to

The voxel world:
- builtin/voxelworld handles most things on the server and the client
- The client contains built-in helpers for resource-intensive tasks
- VoxelRegistry: owned by builtin/voxelworld
	- Yes, on the client too
	- The main world registry is called "main"
	- An identically named block registry also always exists
	- For now, voxel registries are synced only when a new client connects
	- If these registries are synchronized in the background, it probably is not
	  feasible to allow making anonymous ones
- Voxel data is always just a string in a user variable in each node
	- Nodes contain:
		- the used voxel registry name ("buildat_voxel_registry_name")
		- the data itself              ("buildat_voxel_data")
		- data modification version    ("buildat_voxel_mod_version")
	- The data can be raw or compresed, and it can be cached by node id and data
	  modification version. The first byte distinguishes different data formats.
	- Data uses the PODVector<uint8_t> type in Urho3D::Variant because the
	  String type fails to work with zeroes. It is visible to Lua as
	  VectorBuffer.
	- Consider splitting data to multiple user variables if it does not compress
	  very well (say, to less than 500 bytes)
- The client is allowed to see all voxel data for each node that gets synced
  over network.
	- Data can be hidden in local components or other storage on the server.
- Stationary and non-stationary voxel nodes are stored like any node in the
  world
	- How can the world be sanely loaded when it only consists of arbitrarily
	  positioned nodes?
		- Maybe all nodes, including the chunk nodes, could be grouped in
		  sections when saved on disk
- Save format tables:
	- section_size(w, h, d) <- only one row
	- section(sx, sy, sz, save_enabled, generated)
	- node(sx, sy, sz, data) <- (sx, sy, sz) are section coordinates
- How to handle references from nodes to other nodes?
	- Using each node id only once in a world's lifetime is not feasible: If
	  60 nodes are created per second, the networked namespace lasts only
	  for 2**24/60/3600/24 = 3.2 days (FIRST_LOCAL_ID = 0x01000000)
	- The save format uses a 64-bit node id namespace, which is converted to and
	  from the in-memory 24-bit id namespace.
		- The only way to implement this is to integrate this into Urho3D's
		  Node's Get/SetNetParentAttr() because it contains the node hierarchy
		  serialization code.
			- Basically add Node::Get/SetPersistentID() and Node::persistent_id_
			- Also the external parts (reference implementation in Connection)
			  need to be altered
			- 64-bit ids will be used over the network too to simplify things
	- What happens when loading a node, but not a node that it refers to?
		- Nodes can be added to the scene without adding any components or
		  setting any attributes
		- A disk-to-memory id mapping can be maintained in memory so that nodes
		  loaded at different times can be connected
			- In practice, these are stored in Urho3D's Nodes as
			  Node::persistent_id_
- Section storage:
	- By default, generated sections are save-disabled. If a game determines
	  (eg. from player actions) that a section should be saved, it explicitly
	  tells it to builtin/voxelworld, which then marks the section as
	  save-enabled. This flag has multiple states: it is temporarily enabled if
	  the section contains a node that has a save-enabled flag set. In this case
	  the whole section is save-enabled during the time that the node is located
	  inside the section.
	- Overshooting trees and other pseudo objects have to be saved as extra
	  static nodes if they cross the edges of the section
		- Altough actually you can just lazymode through it all and just set
		  voxels in the neighboring sections and chunks. They probably exist and
		  probably will be saved, and if not, you just get your trees cut in
		  half. Well, it's a tradeoff.
	- Save-enabled sections in the memory are saved. Others are discraded.
	- When saving a section:
		- The section properties are saved
		- All nodes in the section are saved
	- When loading a section:
		- The section properties are loaded
		- All nodes in the section are loaded
	- When generating a section:
		- The "voxelworld:generation_request" event is emitted
- Methods:
	- set_voxel(p, v)
		- Set a static voxel
- World generation does not use any special interface because the same things
  should be possible without being triggered by builtin/voxelworld
	- Just send an event; "voxelworld:generation_request"
	- A generation accelerator module can exist to speed up generation by using
	  a single buffer (PolyVox::RawVolume<uint32_t>) and allowing direct
	  registration of generation callbacks. It could also automate the creation
	  of extra static nodes when needed by padding the buffer enough for common
	  things.

- User variable updates (specifically buildat_voxel_data) have to be somehow
  catched on the client so that builtin/voxelworld can update voxel geometry
	- There's no built-in event for this, especially as we aren't using the
	  regular networking implementation of Urho3D (there's nothing there either
	  though)
	- Maybe the client-side handler for replicate:latest_node_data could call
	  into the lua environment on every update, and then the Lua side would
	  check whether the voxel data was modified
		- No; in the case of moving nodes this will cause too much unnecessary
		  processing
		- Also catching them on the server side is not possible because
		  variables are handled discreetly only when they are added;
		  modifications go through Node::WriteLatestDataUpdate().
	- Maybe builtin/voxelworld can send packets to clients when voxel data is
	  modified
		- How does it know which clients know about which node?
			- builtin/replicate could have an interface for asking this