Project Raygine #3

I have reached a point where I need to adress some issues that I've been only winging so far:

A proper serialization system.

The assets are json files, storing the serialized data of objects that need to be reinstantiated. This means, that I need to resolve the classes during deserialization. For that, a registry is needed for the classes that need to be used. However, Lua doesn't have classes. It has tables and metatables, which can be used to imitate classes and objects. So while I could go the way to imitate the OOP style of deserializing objects, maybe there's a more clever way to approach this.

Let's break this problem down; starting with the requirements:

• The serialization uses JSON for now. It could later target binary formats, but let's stick to json for the editor
• Clarification for further analysis:
• An asset is a file.
• An serialized object is a data point inside an asset.
• A serialized object has a UID that is used as key for finding the data
• A serialized object has a type uid that is used to map the on a type
• Serialized data can reference other assets
• Question: Should it be possible to reference serialized objects that are deep down inside ?
• Unity doesn't allow this. I don't believe this is a technical limitation though, so I guess this is intentional. It could be that this is regarded to be too chaotic: If a foreign asset refers to an asset inside another asset, restructurings would be a minefield. I would only allow this, if the tooling shows refererrers and gives a warning when deleting an object that is referred to.
• So I guess this isn't a requirement - but it doesn't mean that the system shouldn't support this

One important factor is the schema definition: How does the program declare what data is to be serialized and to be displayed in the inspector?

Lua has a few language features that can make structure look rather nice - but it may look strange to people who aren't used to these features. For example:

schema = {
    { name = "entity_name", type = "string", default = "Entity" },
    { name = "some_number", type = "number", default = 0}
}

It is relatively clear, what this means - the C# equivalent in Unity would be:

[Serializable]
public class EntityInfo {
    public string EntityName = "Entity";
    public float SomeNumber = 0;
}

It clearly is more concise in C# - the Lua declaration is rather repetitive. One way to reduce this, is to use helper functions:

schema = {
    a:string("entity_name", "Entity"),
    a:string("some_number", 0)
}

This is much shorter and closer to the C# version. But there are more "unconventional" ways how this could be done:

-- using the builder pattern and exploiting that Lua function calls can
-- omit brackets when there's only a single string argument. Returning a function here
-- to construct the schema:
schema = Schema()
    :string "entity_name" "Entity"
    :number "some_number" (0)

-- we can also leverage metatables:
schema = Schema()
schema.string.entity_name = "Entity"
schema.number.some_number = 0

The problem here is, that while it may look shorter, it's quite complex what's happening in the background. The first pure data version however is immediately understandable - even though it allows doing things wrong, as there isn't any way to check the correctness of the data here.

There is however another aspect: How is nesting handled? In my first draft (and working version), the "type" of a declaration can be a table that repeats the pattern:

schema = {
    { name = "entity_name" type = "string", default = "Entity" },
    { name = "look_at" type = {
        { name = "target", type = "uid" },
        { name = "up", type = "vector3", default = {0,1,0}}
    }}
}

While this works OK, there are a few drawbacks; the verbosity becomes irritating when the schema becomes more complex and if I want to reuse the same data structure, I either need to cache the value or copy paste the declarations.

Maybe it really should become a typesystem in first place where a type can be declared and then reused:

-- variant 1 (using function calls)
Type:def "game.look_at" {
    Type:ref "builtin.entity" "target"
    Type:vector3 "up" {0, 1, 0}
}

-- variant 2 (chained returns and "standard" api)
Type:def "game.look_at_camera"
    :add("number", "turn_speed", 5)
    :add("game.look_at", "look_at")

-- variant 3 (using metatables)
Type:def "game.look_at"
    :target (Type:ref "builtin.entity")
    :up "vector3"

I think, I like variant 2 the most here: It is quite readable and short. The IDE can also help here with autocompletion and showing the documentation.

What probably most people used to statically compiled languages are missing would be the heavy usage of strings that could contain typos. Renaming is also going to be more painful, but I don't see much of a way around this - unless creating specifically a language for declaring these schemas and checking them at compile / build time. Something like Protobuf?

Protobuf allows binary encodings for efficient storage, loading, saving and transmission. That is a quite nice thing to have. But in the context of a game engine editor, the far more important factor is stable serialization and mergability. JSON is quite good with that, if certain patterns for the format are followed.

Also, the reason why considerign protobuf is the compiling of the schemas and validating it; however, this can also achieved by checking the schemas in after the Lua scripts have been loaded. Another plus on using pure Lua is, that default values can be complex or even generator functions. This is something that is not possible with protobuf.

So I guess, I will stick with JSON as serialization format and Lua for schema definition for now. But I have to come up with a way to make the type definitions, before I can continue working out the engine UI and other parts. This needs to be solid.