Archive and restore point improvements

Serephucus · February 17, 2019, 12:06pm

Something I’ve been waiting for a while with regard to the archive and restore point functionality. Posted about it before, but have a more refined idea now. So:

Grandfather-father-son for archives and snapshots. Something like:

Hourly backups for a day.
Daily backups for a week.
Weekly backups for a month.
Monthly backups for a year.
Yearly backups thereafter.

Archives could be similar, perhaps omitting the first 1-2 lines above.

Ability to comment on specific archives/snapshots.

Right now, the only thing to go by is the timestamp, which isn’t exactly user-friendly, and relies on the user keeping a reference somewhere else to relate in-game process to server timestamps. It would be extremely useful to be able to add “Finished storeroom” or “Started new railway” to a snapshot.

Comments could be added to archives/snapshots from the relevant page after having been created, or from the status page before clicking the “Create now” button. There would also be a checkbox, per-server, of whether to include anything with a comment in the father-son pruning or not. (If something has a comment, it’s possibly an important milestone, and shouldn’t be removed)

I would love to add this functionality myself, but it’s been years since I’ve done any real programming.

Thoughts?

hexparrot · February 18, 2019, 4:18pm

Archives are made with the normal tar binary, standard to pretty much every *nix system, and restore points are made with rdiff-backup which is a well-matured application as well.

However, neither of them have any good built-in support for naming/tagging (that I know of).

One reason why timestamps are implemented and work so well is because they are the native accounting system of rdiff-backup:


Found 2 increments:
    increments.2018-02-21T10:13:59Z.dir   Wed Feb 21 10:13:59 2018
    increments.2018-02-21T10:16:34Z.dir   Wed Feb 21 10:16:34 2018
Current mirror: Wed Feb 21 10:21:56 2018

PROS:

native functionality
intuitively makes sense to work chronologically (and from a technical perspective, transparent to the user)
by design, is a mechanism to determine uniqueness

CONS:

timestamp doesn’t intuitively map to what a user was doing in that timeframe

So, to further your idea of having a comment for archives/restore points, we need to have a few things for proper implementation:

a webui update that can accept comments
a webui update that can detect and display comments
a backend update that can hold the mappings of a set of timestamps along with the comment/other metadata

For simplicity, I engineered the current solution because there is no need for the webui to remember how one piece of information relates to another: we just know the timestamp.

To adding functionality/convenience will likely require adding complexity in the form of some file (likely) or database (not likely) to remember this stuff. I can envision this as a space-separated value file, something like:

increments.2018-02-21T10:13:59Z.dir just built the rainbow bridge
increments.2018-02-21T10:16:34Z.dir right before my main renovations

Then, the webui can be made to load this file (or track this file) and push the remainder of the line to the webui.

For more complex abilities, probably it’ll be better to do CSV (comma separated) if the need for additional stored data:

increments.2018-02-21T10:16:34Z.dir,important,did some stuff i really care about

Seems pretty viable.

Serephucus · February 18, 2019, 4:52pm

Thanks very much for the detailed feedback!

CSV makes more sense to me as well. I can’t imagine it needing a db.

Thoughts on backup pruning? If something that will be implemented, there may need to be a step 4 where if a comment is added an isIncludedInBackupPruning flag is added. (or maybe the prune just checked if the comment string is empty, dunno which is easier)

hexparrot · February 18, 2019, 10:15pm

Here are a few key areas I think would be excellent for this kind of feature addition:

Rather than modify known-working and well-tested functions, it’s better to add new functions. “stop_and_backup” is one such function that … stops, then creates a restore point. There are two functions already existing for this: ‘stop’, and ‘backup’. But now there’s a function that can do both without the fear of introducing unexpected behavior.

stop_and_backup:

github.com

hexparrot/mineos-node/blob/master/mineos.js#L874


      
                  }
                );  
              }
            ], callback);
          }
          
          self.restart = function(callback) {
            async.series([
              async.apply(self.stop),
              async.apply(self.start)
            ], callback)
          }
          
          self.stop_and_backup = function(callback) {
            async.series([
              async.apply(self.stop),
              async.apply(self.backup)
            ], callback)
          }
          
          self.kill = function(callback) {

So in your case, one might create a function like:
[this is completely untested spitballing here!]

  self.backup_and_tag = function(importance, message, callback) {
    async.waterfall([
      async.apply(self.backup),
      function(cb) { cb(self.list_increments); },
      function(increment_array, cb) {
           most_recent = increment_array[increment_array.length-1];
           fs.appendFile('backup_catalog.csv', "{0},{1},{2}".format(most_recent, importance.toString(), message.toString()), cb);
      }
    ], callback)
  }

And then you just have to call that function from the webui. Here’s an example of how one would call the stop_and_backup function:

github.com

hexparrot/mineos-node/blob/master/html/index.html#L874


      
          <header><h4 class="title"><span class="icon icone-home"></span>{{ 'SERVER_ACTIONS' | translate }}</h4></header>
          <section class="body">
              <div class="body-inner">
          
                  <div class="span4">
                      <button type="button" class="btn btn-success span12" ng-disabled="servers[current].heartbeat.up || !valid_server_name(current)" ng-click="server_command('start')" translate="START" ng-show="(servers[current].sc.java.jarfile && servers[current].sc.java.java_xmx)"></button>
                      <button type="button" class="btn btn-success span12" ng-disabled="servers[current].heartbeat.up || !valid_server_name(current)" ng-click="modals.open_new_server()" translate="START" ng-show="!(servers[current].sc.java.jarfile && servers[current].sc.java.java_xmx)"></button>
                  </div>
          
                  <div class="span4">
                      <div class="btn-group btn-block">
                          <button class="btn btn-danger span6" translate="STOP"
                                  ng-disabled="!servers[current].heartbeat.up" 
                                  ng-click="server_command('stop')"
                                  ng-hide="servers[current].sc.minecraft.unconventional"
                                  ></button>
                          <button class="btn btn-danger span6" translate="KILL"
                                  ng-disabled="!servers[current].heartbeat.up" 
                                  ng-click="server_command('kill')" 
                                  ng-hide="!servers[current].sc.minecraft.unconventional"
                                  ></button>

So while the message and importance bits will have to be implemented in the webui with a bunch of HTML changes, on the JS side, you’d be invoking:

server_command('backup_and_tag', 'critical', 'i just built the best normal wood house ever')

All this to create a file /var/games/minecraft/servers/myserver/backup_catalog.csv which constantly adds new lines assigning tags to each important backup. Other backups may be made with the normal functions backup–and they should continue to work entirely–but these specifically just now you’d have to update the server.js to send over the CSV contents (read, parse, then transmit) to the webui (receive, put into a model, and display).

Tying it all together, you’d go here:

github.com

hexparrot/mineos-node/blob/master/mineos.js#L1100


      
          }
          
          self.restore = function(step, callback) {
            var binary = which.sync('rdiff-backup');
            var args = ['--restore-as-of', step, '--force', self.env.bwd, self.env.cwd];
            var params = { cwd: self.env.bwd };
          
            var proc = child_process.spawn(binary, args, params);
            proc.once('exit', function(code) {
              callback(code);
            })
          }
          
          self.list_increments = function(callback) {
            var binary = which.sync('rdiff-backup');
            var args = ['--list-increments', self.env.bwd];
            var params = { cwd: self.env.bwd };
            var regex = /^.+ +(\w{3} \w{3} {1,2}\d{1,2} \d{2}:\d{2}:\d{2} \d{4})/
            var increment_lines = [];
          
            var rdiff = child_process.spawn(binary, args, params);

Where you would need to add the new code which correlates the increment listing with the ones saved in CSV, with simple substring equality checking–and then place it into the object, such as incr.level and incr.tag

And lastly, go into the HTML template:

github.com

hexparrot/mineos-node/blob/master/html/index.html#L1211


      
          <!-- START Row -->
          <div class="row-fluid">
              <!-- START Basic Table -->
              <div class="span12 widget">
                  <header>
                      <h4 class="title" translate="AVAILABLE_RESTORE_POINTS"></h4>
                      <!-- START Toolbar -->
                      <ul class="toolbar pull-right">
                          <li>
                              <a href="#" class="link" data-toggle="dropdown"><span class="icon icone-ellipsis-vertical"></span></a>
                              <ul class="dropdown-menu pull-right">
                                  <li><a href="#" ng-click="server_command('backup')"><span class="icon icone-hdd"></span> {{ 'CREATE_NEW_RESTORE_POINT' | translate }}</a></li>
                              </ul>
                          </li>
                      </ul>
                      <!-- button is put after dropdown, pull-right class makes them float to the right order -->
                      <button type="button" class="btn btn-mini btn-success pull-right" ng-click="servers[current].refresh_increment_sizes()" translate="GET_INCREMENT_SIZES"></button>
                      
                      <!--/ END Toolbar -->
                  </header>
                  <section class="body">

And add the new importance and message entries so it’s displayed!

Hooray, that’s all!

… well except for any pruning protections, e.g., search if the level meets a minimum threshold, and whether that level is numeric or full words or whatnot. Same thing with sorting or creating a function that would remove all [some criteria].

And that’s all!

Serephucus · February 22, 2019, 3:21pm

So… a week?

I would love to contribute to this, but it goes a little beyond my skills I think, and I understand a lot of your time is being taken up with the Ruby implementation, but it would be a cool feature to see.