Archive and restore point improvements


#1

Something I’ve been waiting for a while with regard to the archive and restore point functionality. Posted about it before, but have a more refined idea now. So:

  1. Grandfather-father-son for archives and snapshots. Something like:

Hourly backups for a day.
Daily backups for a week.
Weekly backups for a month.
Monthly backups for a year.
Yearly backups thereafter.

Archives could be similar, perhaps omitting the first 1-2 lines above.

  1. Ability to comment on specific archives/snapshots.

Right now, the only thing to go by is the timestamp, which isn’t exactly user-friendly, and relies on the user keeping a reference somewhere else to relate in-game process to server timestamps. It would be extremely useful to be able to add “Finished storeroom” or “Started new railway” to a snapshot.

Comments could be added to archives/snapshots from the relevant page after having been created, or from the status page before clicking the “Create now” button. There would also be a checkbox, per-server, of whether to include anything with a comment in the father-son pruning or not. (If something has a comment, it’s possibly an important milestone, and shouldn’t be removed)

I would love to add this functionality myself, but it’s been years since I’ve done any real programming.

Thoughts?


#2

Archives are made with the normal tar binary, standard to pretty much every *nix system, and restore points are made with rdiff-backup which is a well-matured application as well.

However, neither of them have any good built-in support for naming/tagging (that I know of).

One reason why timestamps are implemented and work so well is because they are the native accounting system of rdiff-backup:


Found 2 increments:
    increments.2018-02-21T10:13:59Z.dir   Wed Feb 21 10:13:59 2018
    increments.2018-02-21T10:16:34Z.dir   Wed Feb 21 10:16:34 2018
Current mirror: Wed Feb 21 10:21:56 2018

PROS:

  • native functionality
  • intuitively makes sense to work chronologically (and from a technical perspective, transparent to the user)
  • by design, is a mechanism to determine uniqueness

CONS:

  • timestamp doesn’t intuitively map to what a user was doing in that timeframe

So, to further your idea of having a comment for archives/restore points, we need to have a few things for proper implementation:

  1. a webui update that can accept comments
  2. a webui update that can detect and display comments
  3. a backend update that can hold the mappings of a set of timestamps along with the comment/other metadata

For simplicity, I engineered the current solution because there is no need for the webui to remember how one piece of information relates to another: we just know the timestamp.

To adding functionality/convenience will likely require adding complexity in the form of some file (likely) or database (not likely) to remember this stuff. I can envision this as a space-separated value file, something like:

increments.2018-02-21T10:13:59Z.dir just built the rainbow bridge
increments.2018-02-21T10:16:34Z.dir right before my main renovations

Then, the webui can be made to load this file (or track this file) and push the remainder of the line to the webui.

For more complex abilities, probably it’ll be better to do CSV (comma separated) if the need for additional stored data:

increments.2018-02-21T10:16:34Z.dir,important,did some stuff i really care about

Seems pretty viable.


#3

Thanks very much for the detailed feedback!

CSV makes more sense to me as well. I can’t imagine it needing a db.

Thoughts on backup pruning? If something that will be implemented, there may need to be a step 4 where if a comment is added an isIncludedInBackupPruning flag is added. (or maybe the prune just checked if the comment string is empty, dunno which is easier)


#4

Here are a few key areas I think would be excellent for this kind of feature addition:

Rather than modify known-working and well-tested functions, it’s better to add new functions. “stop_and_backup” is one such function that … stops, then creates a restore point. There are two functions already existing for this: ‘stop’, and ‘backup’. But now there’s a function that can do both without the fear of introducing unexpected behavior.

stop_and_backup:

So in your case, one might create a function like:
[this is completely untested spitballing here!]

  self.backup_and_tag = function(importance, message, callback) {
    async.waterfall([
      async.apply(self.backup),
      function(cb) { cb(self.list_increments); },
      function(increment_array, cb) {
           most_recent = increment_array[increment_array.length-1];
           fs.appendFile('backup_catalog.csv', "{0},{1},{2}".format(most_recent, importance.toString(), message.toString()), cb);
      }
    ], callback)
  }

And then you just have to call that function from the webui. Here’s an example of how one would call the stop_and_backup function:

So while the message and importance bits will have to be implemented in the webui with a bunch of HTML changes, on the JS side, you’d be invoking:

server_command('backup_and_tag', 'critical', 'i just built the best normal wood house ever')

All this to create a file /var/games/minecraft/servers/myserver/backup_catalog.csv which constantly adds new lines assigning tags to each important backup. Other backups may be made with the normal functions backup–and they should continue to work entirely–but these specifically just now you’d have to update the server.js to send over the CSV contents (read, parse, then transmit) to the webui (receive, put into a model, and display).

Tying it all together, you’d go here:

Where you would need to add the new code which correlates the increment listing with the ones saved in CSV, with simple substring equality checking–and then place it into the object, such as incr.level and incr.tag

And lastly, go into the HTML template:

And add the new importance and message entries so it’s displayed!

Hooray, that’s all!

… well except for any pruning protections, e.g., search if the level meets a minimum threshold, and whether that level is numeric or full words or whatnot. Same thing with sorting or creating a function that would remove all [some criteria].

And that’s all!


#5

So… a week? :smiley:

I would love to contribute to this, but it goes a little beyond my skills I think, and I understand a lot of your time is being taken up with the Ruby implementation, but it would be a cool feature to see.