Project Dusk #9 - Update ←
2024-04-21

Project Dusk #10 - Serialization

In the last blog post I laid out how I want to have an editor for creating enemy planes. Here's what I've got so far:

It's not really functional (at all). I am stumbling again over myself all over the place and lose directions:

It's not that I haven't tried it before, but it tended to grow quite complicated and looked like I would need to write lots of boilerplate code redundantly in different places. Not something that works well for me.

Over the course of the last weeks, I've learned however a new trick that I've mentioned in one of the previous blog posts and I realized that I can use the same trick for declaring structs together with the instructions how to serialize and deserialize them.

I only learned of this trick by reading some other source code, so I have no idea what it's called - I would call it:

Context based macro expansion

I think it's not too complicated to understand. Here's an excerpt that makes a few declarations using macros: 

// File: foo.h
// Declare something using a macro

SERIALIZABLE_STRUCT_START(TestStruct)
    SERIALIZABLE_FIELD(int, iValue)
    SERIALIZABLE_FIELD(Vector3, v3Value)
SERIALIZABLE_STRUCT_END(TestStruct)

You may notice that the file has no include guards; it can be included multiple times - which is exactly what we are going to do. Let's first declare the struct: 

#ifndef SERIALIZABLE_STRUCT_START
#define SERIALIZABLE_STRUCT_START(name) typedef struct name {
#define SERIALIZABLE_STRUCT_END(name) } name;
#define SERIALIZABLE_FIELD(type, name) type name;
#endif
#include "foo.h"
#undef SERIALIZABLE_STRUCT_START
#undef SERIALIZABLE_STRUCT_END
#undef SERIALIZABLE_FIELD

The macros are going to be expanded to: 

typedef struct TestStruct {
    int iValue;
    Vector3 v3Value;
} TestStruct;

So we have included the file and through defining the macros prior including, we have declared the struct. Now we want to produce a function that serializes the struct: 

#define SERIALIZABLE_STRUCT_START(name) void SerializeData_##name(const char *key, name* data, cJSON *obj) {\
    cJSON *element = obj == NULL ? cJSON_CreateObject() : cJSON_AddObjectToObject(obj, key);
#define SERIALIZABLE_FIELD(type, name) SerializeData_##type(#name, &data->name, element);
#define SERIALIZABLE_STRUCT_END(name) }
#include "foo.h"
#undef SERIALIZABLE_STRUCT_START
#undef SERIALIZABLE_STRUCT_END
#undef SERIALIZABLE_FIELD

The macros are going to be expanded to: 

void SerializeData_TestStruct(const char *key, TestStruct* data, cJSON *obj) {
    cJSON *element = obj == NULL ? cJSON_CreateObject() : cJSON_AddObjectToObject(obj, key);
    SerializeData_int("iValue", &data->iValue, element);
    SerializeData_Vector3("v3Value", &data->v3Value, element);
}

The procedures "SerializeData_int" and "SerializeData_Vector3" need to be provided - just like every other "native" data type. But this is a one-time effort and can be done in a single file.

The deserialization code is quite similar: 


#define SERIALIZABLE_STRUCT_START(name) void DeserializeData_##name(const char *key, name* data, cJSON *obj) {\
    cJSON *element = key == NULL ? obj : cJSON_GetObjectItem(obj, key);
#define SERIALIZABLE_FIELD(type, name) DeserializeData_##type(#name, &data->name, element);
#define SERIALIZABLE_STRUCT_END(name) }
#include "foo.h"
#undef SERIALIZABLE_STRUCT_START
#undef SERIALIZABLE_STRUCT_END
#undef SERIALIZABLE_FIELD

The macros are going to be expanded to: 

void DeserializeData_TestStruct(const char *key, TestStruct* data, cJSON *obj) {
    cJSON *element = key == NULL ? obj : cJSON_GetObjectItem(obj, key);
    DeserializeData_int("iValue", &data->iValue, element);
    DeserializeData_Vector3("v3Value", &data->v3Value, element);
}

This is now the third time that the original "foo.h" file is included and again, the same macros are expanded into something totally different. It should be obvious that the all the headers that define the serializable structs could be included within those blocks, declaring each time what the structs and functions should look like.

It's a bit funny that I didn't realize this could be done in C even though I programmed with it for like 20 years in this language. Macros have a quite bad reputation and I think it's partly deserved, but I think this utilization is quite nice when comparing it to the other solutions that exist to solve the same problems! For example: Protobuf solves the serialization handling, but the complexity it adds to a project is quite substantial. In other languages, serialization can be achieved using reflection - but this is not possible in C; and compared to those other languages, C is quite simple.

In a way, the macro solution is quite similar to how protobuf works, except that it works within the C standard.

One downside is however, that the macro solution is quite confusing for the IDE and it really isn't straight forward to read, understand and debug, but of all the solutions I've tried so far, it looks the most simple and promising.

I still need to work out the details (and there are quite a few) and I am not entirely sure if this will fly, but it's fun to give it a try. What's not so fun: I have now around 20 components that are in different development states since I made improvements along the way and I never updated most of the code because I wasn't sure if the changes are final. I now need to update all of them in order to be able to utilize the serialization. And that's not all: There's much more that I will need to implement if I go that route:

So... lot's of things. Alternatively, I create a custom simple editor for the planes and come up with another way how to import the configurations. But somehow I believe that it might be time to deal with the serialization part to make things more flexible. It would be really nice if I could visually edit the levels and create events using an editor that works on the scene graph system.

Lot's of work and detours - but well, I guess I am doing what I think is most fun. It's just a hobby project after all and I am not in a hurry 🥲

🍪