Project Dusk #10 - Serialization

In the last blog post I laid out how I want to have an editor for creating enemy planes. Here's what I've got so far:

Fullscreen

It's not really functional (at all). I am stumbling again over myself all over the place and lose directions:

• I first tried raygui for the UI rendering (which only needs to be 2D), but I wasn't quite happy with how it works - and I therefore started writing my own C-based imgui system 🫠. I want some hierarchy UI mapping, overlayed elements to be not click through and buttons not reacting on mouse down events but only when released after being pressed on the element in first place. I considered for a moment to import dear imgui again, but ... it's C++ and kinda big, so I thought I can give it a try to roll my own
• I encountered now several times the situation of "if I only had already implemented serialization" and this is one such case again:
• If I had serialization, I could make a (simple) component based editor, serialize the data and using it as a prefab
• I encountered it now a few times already and somehow I feel I should have done this earlier...

It's not that I haven't tried it before, but it tended to grow quite complicated and looked like I would need to write lots of boilerplate code redundantly in different places. Not something that works well for me.

Over the course of the last weeks, I've learned however a new trick that I've mentioned in one of the previous blog posts and I realized that I can use the same trick for declaring structs together with the instructions how to serialize and deserialize them.

I only learned of this trick by reading some other source code, so I have no idea what it's called - I would call it:

Context based macro expansion

I think it's not too complicated to understand. Here's an excerpt that makes a few declarations using macros:

  1 // File: foo.h
  2 // Declare something using a macro
  3 
  4 SERIALIZABLE_STRUCT_START(TestStruct)
  5     SERIALIZABLE_FIELD(int, iValue)
  6     SERIALIZABLE_FIELD(Vector3, v3Value)
  7 SERIALIZABLE_STRUCT_END(TestStruct)

You may notice that the file has no include guards; it can be included multiple times - which is exactly what we are going to do. Let's first declare the struct:

  1 #ifndef SERIALIZABLE_STRUCT_START
  2 #define SERIALIZABLE_STRUCT_START(name) typedef struct name {
  3 #define SERIALIZABLE_STRUCT_END(name) } name;
  4 #define SERIALIZABLE_FIELD(type, name) type name;
  5 #endif
  6 #include "foo.h"
  7 #undef SERIALIZABLE_STRUCT_START
  8 #undef SERIALIZABLE_STRUCT_END
  9 #undef SERIALIZABLE_FIELD

The macros are going to be expanded to:

  1 typedef struct TestStruct {
  2     int iValue;
  3     Vector3 v3Value;
  4 } TestStruct;

So we have included the file and through defining the macros prior including, we have declared the struct. Now we want to produce a function that serializes the struct:

  1 #define SERIALIZABLE_STRUCT_START(name) void SerializeData_##name(const char *key, name* data, cJSON *obj) {\
  2     cJSON *element = obj == NULL ? cJSON_CreateObject() : cJSON_AddObjectToObject(obj, key);
  3 #define SERIALIZABLE_FIELD(type, name) SerializeData_##type(#name, &data->name, element);
  4 #define SERIALIZABLE_STRUCT_END(name) }
  5 #include "foo.h"
  6 #undef SERIALIZABLE_STRUCT_START
  7 #undef SERIALIZABLE_STRUCT_END
  8 #undef SERIALIZABLE_FIELD

The macros are going to be expanded to:

  1 void SerializeData_TestStruct(const char *key, TestStruct* data, cJSON *obj) {
  2     cJSON *element = obj == NULL ? cJSON_CreateObject() : cJSON_AddObjectToObject(obj, key);
  3     SerializeData_int("iValue", &data->iValue, element);
  4     SerializeData_Vector3("v3Value", &data->v3Value, element);
  5 }

The procedures "SerializeData_int" and "SerializeData_Vector3" need to be provided - just like every other "native" data type. But this is a one-time effort and can be done in a single file.

The deserialization code is quite similar:

  1 
  2 #define SERIALIZABLE_STRUCT_START(name) void DeserializeData_##name(const char *key, name* data, cJSON *obj) {\
  3     cJSON *element = key == NULL ? obj : cJSON_GetObjectItem(obj, key);
  4 #define SERIALIZABLE_FIELD(type, name) DeserializeData_##type(#name, &data->name, element);
  5 #define SERIALIZABLE_STRUCT_END(name) }
  6 #include "foo.h"
  7 #undef SERIALIZABLE_STRUCT_START
  8 #undef SERIALIZABLE_STRUCT_END
  9 #undef SERIALIZABLE_FIELD

The macros are going to be expanded to:

  1 void DeserializeData_TestStruct(const char *key, TestStruct* data, cJSON *obj) {
  2     cJSON *element = key == NULL ? obj : cJSON_GetObjectItem(obj, key);
  3     DeserializeData_int("iValue", &data->iValue, element);
  4     DeserializeData_Vector3("v3Value", &data->v3Value, element);
  5 }

This is now the third time that the original "foo.h" file is included and again, the same macros are expanded into something totally different. It should be obvious that the all the headers that define the serializable structs could be included within those blocks, declaring each time what the structs and functions should look like.

It's a bit funny that I didn't realize this could be done in C even though I programmed with it for like 20 years in this language. Macros have a quite bad reputation and I think it's partly deserved, but I think this utilization is quite nice when comparing it to the other solutions that exist to solve the same problems! For example: Protobuf solves the serialization handling, but the complexity it adds to a project is quite substantial. In other languages, serialization can be achieved using reflection - but this is not possible in C; and compared to those other languages, C is quite simple.

In a way, the macro solution is quite similar to how protobuf works, except that it works within the C standard.

One downside is however, that the macro solution is quite confusing for the IDE and it really isn't straight forward to read, understand and debug, but of all the solutions I've tried so far, it looks the most simple and promising.

I still need to work out the details (and there are quite a few) and I am not entirely sure if this will fly, but it's fun to give it a try. What's not so fun: I have now around 20 components that are in different development states since I made improvements along the way and I never updated most of the code because I wasn't sure if the changes are final. I now need to update all of them in order to be able to utilize the serialization. And that's not all: There's much more that I will need to implement if I go that route:

• The serialization needs to be able to handle asset and object references - which I am not sure how to implement right now
• In case I need polymorphism data structures, I need to come up with a "syntax" how to describe it (same for arrays, but that should be easier)
• The scene graph needs to be serialized as well and this isn't implemented at all as well. Oh, and of course, the type system needs a working serialization system too
• In case I can really make it work, it would be nice to allow building levels in an editor and serialize them as well
• I don't have an editor - that would be needed too!
• Displaying the hierarchy is really simple...
• ... but I'll need something similar for the component inspector - which means creating editors for each component (can also use the same macro trick as above) and making it editable
• ... but I don't have all the imgui widgets yet to display that

So... lot's of things. Alternatively, I create a custom simple editor for the planes and come up with another way how to import the configurations. But somehow I believe that it might be time to deal with the serialization part to make things more flexible. It would be really nice if I could visually edit the levels and create events using an editor that works on the scene graph system.

Lot's of work and detours - but well, I guess I am doing what I think is most fun. It's just a hobby project after all and I am not in a hurry 🥲