Efficient Chunk Storage

Hello Community,

I am currently working on a project that has to do the following:

- Save a CHUNK into a .mchunk in a folder, with name of the file being the X and the Z coordinates.
- Load the CHUNK into a chunk in the reverse order of the saving to the file.

I know how to name the file, and already have a save method I use in another one of my plugins.
Here is example save and load code.

Save:

Code:java

  public void loadData() {
    File file = new File("pingdata");
    try
    {
      BufferedReader br = new BufferedReader(new InputStreamReader(new FileInputStream(file)));
      String l;
      while ((l = br.readLine()) != null)
      {
        String l;
        String[] args = l.split("[,]", 2);
        if (args.length == 2) {
          String ip = args[0].replaceAll(" ", "");
          String p = args[1].replaceAll(" ", "");
          this.playerData.put(ip, p);
          br.close();
        }
      }
    }
    catch (FileNotFoundException e) {
    }
    catch (IOException e) {
    }
  }

Load:

Code:java

 public void saveData() {
    File file = new File("pingdata");
    if (!this.playerData.isEmpty())
      try
      {
        BufferedWriter bw = new BufferedWriter(new FileWriter(file));
        for (String p : this.playerData.keySet())
        {
          bw.write(p + "," + (String)this.playerData.get(p));
          bw.newLine();
        }
        bw.flush();
        bw.close();
        getLogger().info("Data saved.");
      } catch (IOException e) {
        e.printStackTrace();
      }
    else;
  }
 

Ignore the various oddities in the code, this was copied straight from my code without editing.

Anyways, I know that I could iterate through all the blocks in the chunk.
Save the block id to a hashmap at the location.
Save the block data to a hashmap at the location.
Save the block BlockState to a hashmap at the location.

That's all fine and dandy, but it's in efficient.

My thesis: Is their any way to efficiently store a chunk and read a chunk, in a plain text file.

Can anyone give me some code/ pseudo code on how I might accomplish this.

Bottom line: Saving/ Loading a chunk from/to a plain text file. (While maintaining block data and BlockState)

Thanks!
Please respond.

- Blue

 

Since you can probably expect that you are going to have a full, contiguous array of data over the entire chunk, you probably should ditch the HashMaps because those are just wasting memory holding locations, when the data's position along the file should be well enough to determine where it is. You can probably use a CSV file or something alike, where you have all the block IDs separated by commas, a semicolon, then all the block data separated by commas, a semicolon, and then finally all the block data separated by commas. I should note that BlockState also happens to contain IDs and data as well, so you should make sure to either skip those when storing block states, or just store only the BlockStates. Then when reading back from the file, you can use a CSV library or just split each line by commas.

It would probably be much easier though to use a NBT file like WorldEdit/MCEdit does. You can read up on how WorldEdit loads schematics on their github page, and either use their system, or you can choose to extend it by adding NBT tags for the block states. Remember that block states are unnecessary for a lot of blocks and can be represented entirely by their data bytes, so you may want to skip it entirely and just have special storage for chest inventories, furnace inventories, etc instead.

Either way, you need to convert a 3d array of blocks into a 1d array. This is easily accomplished by combining the X, Y, and Z coordinates into a single index by lining up all their bits. For a 128-block tall chunk, you need 7 bits, and for X and Z lengths of 16 you need 4 bits each. This comes out to a total of 15 bits for indexing your 1d array, which implies a length of 32768. It would work something like this:

Code:

int index = (y * 256) + (z * 16) + x;

This is basically how WorldEdit indexes values in the 1d array in schematics, which I would actually highly recommend over a CSV file now. Even if you don't use schematics you can still use a NBT data structure that would work very efficiently.

 

there are special streams for binary data with much better performance, check oracle for more info.

i will suggest to use memory mapped file wrapped by bigmemory array, it will allow to write and read data from arbitrary location of file.

 

Hmm, I will consider all these. (I like them, I will try them all.)
Any other ideas however?

 

Bump. Maybe CHUNK to NBT and NBT to JSON? And vice versa.