Making Saves Go Faster

Airships: Conquer the Skies
24 Dec 2020, 3:34 p.m.

I'm working on an entirely new system for saving and syncing conquest games. It's currently in beta, and it's not certain that it will make it into the game yet, but I thought you might enjoy a dive into the why and how of it.

There's two closely linked things here: saving the game, and calculating checksums to make sure that multiplayer games haven't desynced. Desyncing is what happens when the realities of two players in an MP game diverge, which is of course bad. Being able to detect it helps me fix bugs that cause the game to desync. Long-term, I hope to also add in functionality for desyncs to heal automatically.

Currently, both save and sync are very simple: the entire game world - map, empires, ships, landscapes - is converted to the JSON data format, basically one long blob of text. When saving, that blob of text is written as a file to disk. When syncing, the game calculates a checksum of the text and compares it to other players' checksums to make sure they're in sync.

But that's a lot of data, especially in the later stages on a large map. Dozens of megabytes. And of course the game world can't change while it's being saved, and so when the game syncs or autosaves, there's a noticeable pause. Sometimes, a very noticeable pause that causes your multiplayer game to steadily fall behind, because it can't keep up.

The new save system does two things to fix this, both a bit fiddly:

Lazy Saving

Most of the game world doesn't actually change between saves. A ship that's just flying around without engaging in combat doesn't change. The landscapes of peaceful towns, with nothing crashing into them, also don't change. So re-creating all the JSON data for them isn't actually necessary. By splitting the save game into a bunch of smaller files, one for each landscape and ship, the game can avoid doing most of the work of saving.

The hard part is that it has to know when a landscape or ship did change, which means that these things now need to keep track of a version number. That version number needs to be increased whenever a change happens. If not, the save will be partially outdated.

Conveniently, I realised that during development and testing, the game can produce the JSON for things that are supposedly unchanged, and compare it with the information that's in the file on disk to see if it really hasn't changed. This lets me root out cases where the version isn't getting updated.

A Compact Format

JSON is a convenient format, but it's also very bulky. It's text-based and each data field is individually labelled, which makes it human-readable but verbose. Completely changing the data format of the game would be a huge effort. Instead, I created a more compact representation of JSON. (There's probably a lot of compact JSON representations out there, but this one is mine.)

There's a bunch of minor gains to be made. For example, the number 9845329 takes seven bytes to store in JSON but only four if you store it as an integer number. A small number like 109 fits into one byte.

The major gain is not having to repeat all the names of the fields. For example, each crew member on a ship stores a "weaponReload" value. On a ship with 100 crew, this means the letters "weaponReload" are repeated 100 times, which is 1200 bytes just for that. But in a compact format, you can just use "weaponReload" once and then refer back to that text using a back-reference that uses far fewer bytes. Because most of a JSON file's size is made up of field names, the total size of this new format is a fraction of the old one.

A back-reference here means that instead of spelling out the text you keep track of all the bits of text you used previously, and then refer to them by number when you need them again. For example the following sequence of words:

fruitbat, fruitbat, llama, fruitbat, iguana, squirrel, llama

can be abbreviated to:

fruitbat, 1, llama, 1, iguana, squirrel, 3

Where the numbers represent the position in the list where a word was first encountered.

What makes changing formats hard is that it needs to have 1:1 fidelity to JSON. You should be able to convert any JSON into this format and back again without losing or distorting any information.

But wait, why do we care about a compact format? We want a fast format! Well, conveniently those are very similar goals. Having to glue together far fewer bytes in less complicated ways makes this format faster. And having to write fewer bytes to disk also makes saving faster.

So by combining these two techniques, saving and syncing will take two orders of magnitude less time. It's more work, but it's also necessary.

You might be wondering why I'm messing around with save formats instead of working on the diplomacy update, but the way I see it, I can't do a big update and tell everyone to play multiplayer games when there's problems like multi-second sync pauses still happening.

BTW Airships: Conquer the Skies is currently off 35% on Steam: