Project Dusk #1 - Typing

I am restarting the Raygine project using "Dusk" as new project name.

Since the Lua driven approach is not delivering enough performance in the browser, the engine needs to do more on the native side. This is such a fundamental change that I don't see sense in continuing work on the editor. Unless I find a silver bullet to run Lua in the browser fast enough. Parity with the Desktop LuaJIT performance is most probably not possible. Running 4x slower than that would be ok. A factor of 10x would already be a concern. Anything above that is not acceptable. One thing left to try is Luau. It's a Lua runtime that is supposed to be faster than standard Lua albeit not as fast as LuaJIT. But it provides optional additional typing, which is something I would actually appreciate. I haven't found benchmark numbers though, so it looks like I need to do this myself:

• Compiling Luau locally
• Running a desktop benchmark
• Compiling Luau for the browser
• Running a browser benchmark

Not something I am eagerly executing; I am still new to setup emscripten projects and Luau is quite a bit bigger than Lua alone. I have to figure out which parts are required for the browser build and which parts are not.

Anyway, that's one of my work tasks I have on my todo list. In any case, a low level rewrite is high on my priority list. Which brings me to ...

Priorities

When I started work on Raygine, I had a clear vision of what I wanted to achieve, the cornerstones being

• Asset management: assets are managed in a similar fashion as in Unity - which means renaming and moving assets in the file system won't break the project. Referencing assets should work naturally the same as well.
• Serialization and deserialization of assets: Storing assets in a json format with a strong emphasis on compatibility with working with version control systems.
• Fast loads and reloads: Reloading the project and all the scripts should be doable in a few hundred milliseconds at worst. Testing a game should feel like uninterrupted coding time.
• A strong focus on the editor: The editor should be easily extensible via scripting.

I would say this was always my focus and I will keep those goals. Runtime efficiency is now however also on the list as it shows that it is important. Therefore I am adding now these points to the list:

• The engine C/C++ core should do as much as possible to keep headroom for the Lua script execution
• Moving Lua code to C/C++ should be easy and straightforward

This is quite a challenge. I have to start low level and work my way up.

Fundamentals

The most fundamental aspect to my goals is having a proper type system of objects that can be serialized and deserialized. This is the first thing I need to get working. There are again some key features I desire to have:

• Runtime type system: Types can be declared at runtime by Lua scripts. Therefore, any static type system is not an option. That's practically how all the type systems (like ProtoBuf or FlatBuffers) I found work; there are some users looking into that, but it looks like lots of work. I don't want to have to recompile the project when I add a new type as it would violate the fast reload requirement. Additionally, I have more requirements than "just serialization": I need to be able to define custom editor behavior for types. Like what can be done through attributes in Unity C# scripts.
• Memory management: The values produced by the type system need to be memory managed, meaning creation and destruction of objects should be handled by the system.
• Interfacing: The type system should be able to interface with the C/C++ side the most natural way possible. Mapping types to structs is essential for this. If I have a Vector3 struct, the type system should shouldn't be in the way of working with it on the C side. Meanwhile, it also needs to interface with Lua: access to value members and its editor meta information is a requirement.

I may also have to look into garbage collection, but this is secondary now.

I also considered using Lua as a type system since it brings lots of features to the table already - but the seamless C integration part is an issue here I wouldn't know how to overcome. Sure, userdata values could be mapped on structs, but then I have the problem again that the user can at runtime create new types.

So I am looking into a dynamically managed type system using plain C. The reason I am aiming for C is that after toying around with C++, its existing complexity when it comes to dealing with classes and class members and so on is daunting and I fear it's even in the way to achieve a dynamic runtime type system.

One merit of this idea is, that types can be stored efficiently, even when created through the Lua interface.

Anyway, I currently don't have anything to show, so it's all just ideas and considerations. But that's where I am right now.