The original layout of the geometry for QubeKwest was simply blocks in 16x16x16 block chunks in the world. This was designed to accommodate bulk network transmissions. After all, sending 4,096 blocks in a neat little package is a good deal more efficient than sending them one at a time. At the moment, that means that a chunk is around 65 KB of data. As I was working on the idea of how to store chunks in permanent storage on the server, I got to thinking about how obnoxious it would be to have vast numbers of 65 KB chunks all over the disk.
Instead of making an artificial storage construct for chunks on the disk drive, I got to thinking about how to best manage it and came up with an honest to goodness layer of physical space that can be used directly. This may end up being handy for generation of the world as well since it gives a slightly more impressive physical space into which I can easily render advanced features. The idea is that above chunks, we now have clusters. These hold 8x8x8 chunks (512 chunks total) and are designed to be the way of storing the world on a hard disk. These presently average something near 26 MB each in their uncompressed form, but they will likely be stored as compressed gzip streams so the world doesn’t take over your whole drive after 20 seconds of exploring.
These clusters are also designed to be a semi-sparse structure. In other words, they have a hard list of 512 Chunks in them, but those chunks are allowed to be “EmptyChunk” objects instead of real ones. This type of chunk takes approximately 4 bytes instead of 50,000. After all, if there are parts of the cluster that haven’t been generated yet, then there is no need to store massive amounts of block data that is just waiting to be replaced by the generator later on. This is especially likely in the vast depths of the underground. If you haven’t dug down close to it, it really doesn’t need to have been generated yet. If it hasn’t been generated yet, don’t waste the hard drive space yet either.
There were a few different trade offs on the sizing of a cluster before I settled on 8x8x8 chunks. My first theory was 16x16x16 chunks in a cluster (after all I’d already done the math for storing 16x16x16 blocks in a chunk), but I soon realized that would mean way too much memory to load them off the disk to make them active on the server. After that, I briefly toyed with the idea of getting rid of clusters altogether. In the end, I decided they would still be useful to prevent the need for obscene numbers of individual chunk files.
Consider for a moment a world that is a small 128x128x32 blocks. If it was nice and flat and you were running at 6m/s it would still take you over 20 seconds to walk end to end of this tiny world. If I got rid of the cluster concept, that volume of world data would be stored in 128 files. With an 8x8x8 chunk cluster, that world fits neatly into a single file with loads of room to spare for more vertical exploring. This is much friendlier to the storage system on the server, especially as the worlds get larger.