Administration Troubles

[jalaud]

I’m brand new to Minecraft and MineOS, but I’ve messed with Linux for a very long time. I started a VirtualBox VM with turnkey/debian linux from the iso and it all worked great. My son and some friends logged in and all worked as expected. I left most of the defaults (except server name) and did a few manual restore points and an archive and the files were created properly without any errors.

A few days ago, I noticed my server load crept up to 6-7* even with just 2 people logged in, so I decided to have them logout so I could do a restore point and reboot the VM. I figured maybe it was java since it was over 400% CPU used in top, but memory use and other system stats looked normal. To my surprise, after the reboot we lost about a days worth of character progress (specifically, all inventory items and xp), but the world was restored properly. Things we had just built 5 min before the reboot and items in chests, etc., were still there. I’m not sure why the server didn’t restore everything equally. Maybe because I didn’t have the “save-all” option checked as it isn’t by default? I assumed it was a basic linux daemon that could be stopped and started and resume from where it left off.

• I have an 8 core, 16 thread AMD 2700x so it wasn’t touching the host machine much at all. It was something like 30% CPU total. Also, 400% CPU and load averages at 6-7 for linux are not “too” weird on such a CPU.

The next day, I needed to reboot Windows on the host machine for an update, so I rebooted the server from the console (the blue/gray menu on tty7), shut down the VM, rebooted my PC and restarted the VM. I had the “save-all” checked this time, so I thought it would just pick up within 20 min of where it left off, since it was auto saving every 20 min. Instead, we lost a days worth of world changes, but this time we kept character progress (xp, inventory, etc).

Eventually I’m going to need to reboot the VM again, so I’m wondering what am I missing? I’m on vacation, so I’ve spent a few hours the past couple days googling to find more info. There’s tons of places that say how to backup, the differences between incremental restores, where the files are at, permissions issues and all that, but no documentation that I could find that says “this is how you properly back up, reboot and restore”. Do I need to manually restore after a reboot, before I start the server? I wouldn’t think so since there’s a “restart server on reboot” option. Does it need to be a full archive restore? When you prune restore points, does it combine previous files into the latest?

I wanted to figure this out on my own, so I created a second server from an archive, hoping to test backup/restore behavior without screwing up live players again. It starts, I set a different server name and port 25566. I verified there’s a second java process in top, my client sees it in a LAN scan, but I can’t login. It times out. Maybe it’s a firewall issue, that I need to open port 25566?

[jalaud]

One problem solved! I updated /etc/iptables-rules to add the second server port on reboot (I believe this is processed on boot) and I committed the changes in a root shell as well and now I am logged in to the second server. So now I can probably play a bit and see what saves, what doesn’t, how to gracefully restart the server, etc.

[iMelsom]

You should normally not need to run “restore” everytime you start your server. The minecraft server loads all saved blocks when you start the server. That is what save-all does, it gets the minecraft server to save all blocks in memory to disk. This is an internal Minecraft operation. You can trigger the same opeation ingame by using the command “/save-all” in the minecraft chat

MineOS’s backup and archive functions takes a copy of the saved map files according to the cron-job period you defined, and uses the standard operating system rdiff (for backup) or tar -zxvf (for archive).
This prosess is actually somewhat processor intensive, so it is best done at a slow period (f.ex in the middle of the night)

Just to hurry past the “D’oH” parts:
If you run MineOS’s restore function each time you boot, you overwrite your minecraft servers map blocks with the backed up or archived ones. So if you do a backup or archive every second day, and restore for each boot, you may loose up to two days of work.

Now, if the “D’oH” isn’t what is happening it seems your minecraft instance is not saving properly to disk. You can test this by just stopping and starting the minecraft server (without rebooting) if it starts up again with all blocks where they are, we know that mineos saves to disk, and we’ll have to look at your virtualbox setup: Then it seems like Vritualbox is failing to write to disk, and only allows MineOS and minecraft to save to memory, and discards it when rebooted.

Since I have learned that minecraft, ubuntu (which I use), and virtualbox can be somewhat particular this is my steps for rebooting:

1. in the console at the bottoom of the log window enter the command save-all (forces a save all, just for safety, minecraft usually saves on stop)
2. stop the minecraft server with the stop button, or by writing “stop” in the same place as above
3. ssh into the MineOS server and run shutdown as root or sudo
4. reboot the VM.

And do not run “restore” in the MineOS WebUI

[jalaud]

I just finished testing that with the backup server and it worked perfectly. I’m going to try it with the “production” server after I warn everyone and have them take some precautionary screen shots. No reason to upset things until I need to.

Thanks a ton for your prompt and detailed explanation! It looks like I just had a conceptual misunderstanding of the differences between save-all, restore points and archives. To add my two cents, maybe the automatic save-all should be checked by default to something reasonable. Although, it’s maybe only an issue for folks like me that start with 0 MC knowledge and assume the files are backed up on shutdown.

[iMelsom]

@jalaud

automatic save-all should be checked

This is sa safetycatch and a vestigal choice after an earlier rather serious minecraft bug where minecraft did not save by it self at all.

The default behaviour of minecraft servers is to autosave every 5 minutes, and on a stop command: https://www.minecraftforum.net/forums/support/server-support-and/1929582-save-interval

The save-all in the MineOS Web UI is there to make MineOS ask the minecraft server to do a save-all. It is not really needed any more, but occationally minecrafts internal save functions fail to work, so it is another level of safety to make sure data is not lost.

[hexparrot]

@iMelsom is right on all counts, regarding everything from the functions of the archives and backups, restores, and ‘save-all’.

My post here is just to throw more words at an already explained phenomenon because I love talking.

---

save-all

Reference

@hexparrot

The crux of this is the save-all command (or /save-all ingame). The save-all command tells Minecraft to flush the current contents of the world (what you see ingame) to disk.

Now there are a few catches. For one, save-all saves chunk data (and maybe entities), but not player inventories. Save-all is server-wide and something that is done manually , but otherwise if not done manually save-to-disk operates on an entirely different schedule, namely, when a chunk is unloaded (because no players are in the area– and –the memory is needed for new chunks.

tl;dr “save-all” only affects block files, e.g., .mcr (minecraft region files)

archive

Archive collects and compresses into a single file all the files presently on the disk, within /var/games/minecraft/servers/[servername]. Note, archive gives a misleading sense of completeness, because when a server is running, the files on disk do not always fully reflect the state of the world you’re actually playing in. Even save-all will only push the blocks to disk, but player inventories will not be.

backup

Similar to archive, but uncompressed and version controlled. Files that change from the last snapshot are added to the collection of files that grow and grow in /var/games/minecraft/backup/[servername]. In this directory are always the most recent versions of the files. Older versions are stored “secretly” in a hidden . directory, accessible to the user and any restore attempts.

putting it all together

This is a heavily contrived example of hourly backups & half hour save-all used for illustration:

AND, if you like visuals, here’s something I wrote up a decade ago! RAM & Minecraft world generation (but ignore the MB recommendations, since this was for alpha MC)

12:00 archive made, backup made – save-all scheduled, but doesn’t go when server is down.
-whatever is on the disk, the FULL entirety of the world, player inventories, are all saved, unquestionably.

12:15 server started
-files are updated, as usual when Minecraft launches. The world now differs in a non-meaningful way from what is on disk

12:25 Player A logs in at the spawn point, (Position M), goes for a 30 minute run toward the sun and stops at Position N.
-minecraft generates a lot of MCRegion files on the fly and saves them to disk. Player A’s occasional cutting down a tree as they are running is represented in game, but is not reflected on disk.

(caveat: if the java heap got filled up at any point from this run, position M might have gotten unloaded, which means M would actually get saved to disk, and a few other unloaded regions).

12:29 Player A disconnects, voluntarily or lost internet connection
-minecraft saves player’s inventory, position, that’s it.

12:30 save-all
-Spawn point (M) and all regions up to point N are saved to disk.

12:31 Player A reconnects
-current game state: 12:29 player state, 12:31 world state

12:45 Server crashes
-current game state: 12:29 player state, 12:30 world state

12:50 Admin does a restore (inadvertently overwriting new state)
-current game state: 12:00 player state, 12:00 world state

1:00 archive made, backup made – save-all scheduled, but doesn’t go when server is down.
-current game state: 12:00 player state, 12:00 world state