WebUI Slow to Respond and Making it Quicker

I am having a problem where MineOS is very slow to load server details. When I select a server on the dashboard it sometimes takes several minutes to load.

I think the problem is partially because I setup my server with symlinks to save backups and archives on a separate hard drive as my boot SSD doesn’t have enough space for them. Because the HDD is much slower I think gathering the list of increments takes more time than expected.

I looked at the code, specifically the function get_page_data at line 1265 of server.js. My theory is that because this function waits for all async functions to evaluate before responding, it is limited by the speed of the slowest function.

Changing this function to

function get_page_data(page) {
      switch (page) {
        case 'glance':
          logging.debug('[{0}] {1} requesting server at a glance info'.format(server_name, username));

          async.parallel({
            'increments': function(cb) {
                    cb(null,[])
            },
            'archives': function(cb) {
                    cb(null,[])
            },
            'du_awd': async.apply(instance.property, 'du_awd'),
            'du_bwd': async.apply(instance.property, 'du_bwd'),
            'du_cwd': async.apply(instance.property, 'du_cwd'),
            'owner': async.apply(instance.property, 'owner'),
            'server_files': async.apply(instance.property, 'server_files'),
            'ftb_installer': async.apply(instance.property, 'FTBInstall.sh'),
            'eula': async.apply(instance.property, 'eula'),
            'base_dir': function(cb) {
              cb(null, user_config.base_directory)
            }
          }, function(err, results) {
            if (err instanceof Object)
              logging.error('[{0}] Error with get_page_data'.format(server_name), err, results);
            nsp.emit('page_data', {page: page, payload: results});
          })
          break;
        default:
          nsp.emit('page_data', {page: page});
          break;
      }
    }

disables the increments and backups listing but reduces the response time from 3 minutes to 3 seconds. This seems to prove my theory.

To me the solution seems to be to split the list increments and list archives functions into their own socket requests. I am eager to hear your thoughts on this. If you want I can prototype this and submit a pull request but I am not that familiar with this codebase.

1 Like

I think you’re spot on.

get_page_data has been getting overloaded–it was at it’s core already fairly loaded, with I/O bound processes like du, but with the ever-growing amounts of increments and archives that people experience due to nearly limitless storage, this process has gotten far too slow.

Feel free to prototype a request. I think that this update is critical for the life of the webui.

I’ve started work on this issue.

I decided to post this too Development Dockerfile by ZEBaker98 · Pull Request #454 · hexparrot/mineos-node · GitHub. I am a DevOps engineer by training and I threw this together to make testing easier for me. I program on my mac and didn’t want to setup all the MineOS dependancies. Instead I can just build dev images for running in docker. The first run through when all the dependency layers are built takes a few minutes, but for small code changes after that a new image can be built in seconds.

I just finished working on this issue.
Here is the relevant pull request Split get_page_data function to improve server response time by ZEBaker98 · Pull Request #455 · hexparrot/mineos-node · GitHub.

Hey,
I thought i was alone because my Web-UI somehow is now loading for ages and is not very responsive… also sometimes it even crashes so that the connection times out, than i have to reconnect and wait a very long time for the server to show up… stats, settings and so an are not loading fast either…

and when i reload the “IoWait” of the CPU usage which is normaly near to zero… spikes up to around 80% till the server show up on the UI and than returns to normal values… i don’t know if this is normal but its seems pretty sketchy…

Also please notice in this screenshots that the web-UI is using very much RAM by itselfe… and its kinda pulsating but you get what i meen if you see the screenshots… is this normale or is something broken?

How can I get that fix from you into my system?

For now you can run the following commands to download this patch from my fork of mineos and implement them into your system.

Script removed

Edit: I removed this script because it no longer works and it became apparent that even though my code changes were small the created some unexpected bugs

After you do this you will need to restart mineos for the changes to take effect.

If you want to remove these features you can run the reset_webui.sh script included with MineOS.

I added some more optimizations today that improve loading speed on the Dashboard and Server Status pages. The Restore Points portions of UI are still slow but the rest of MineOS should be significantly faster.

1 Like

Managing your restore points should help reduce the delay. No one needs 30 restore points…

That may be the case but having a lot of restore points shouldn’t slow the rest of the webui to a crawl.

The way the webui was setup I think it actually overloads the mineos server with requests when the dashboard loads. The dashboard would request a list of minecraft servers from the mineos server and begin constructing server objects for use within the webui. The constructor for these server objects was setup to request all the data for that server when only a portion of the data is needed to show on the dashboard. My pull request includes changes so that the dashboard constructs the server objects and requests only a minimum amount of data fields be populated. When the user selects a server and the change_page function is called it tells that server to get all of its data. The result of this is that both the Dashboard and the Server Status pages are much more responsive.

The reason getting a list of restore points is so slow is because mineos requests the list with increment sizes. If you run the command to get the list without sizes its almost instant. There might be a way to use this to send a partial list to the webui while the full list is being gathered but for now I think these changes improve mineos enough.

TLDR: This change makes it so that the server now sends requested data to the webui as it becomes available instead of waiting for all of it to be available and the webui only requests data it actually needs.

TLTLDR: MineOS go BRRRRRRRRRRRR

1 Like

@Zachary_Baker I’ll be reviewing and hopefully getting to merge this in, today/very soon. I appreciate what this is addressing.

@hexparrot It might be too soon to celebrate. I don’t think this upgrade is quite done. The changes I made broke the calendar and while I was trying to fix that I discovered that if you use the currently selected server dropdown to change the server and then click the server status page instead of using the link on the dashboard the webui doesn’t load any data at all. Looks like I have more work to do.

My progress has been slow because my dev instance of MineOS has been crashing with no error logged or stack trace. I added error handlers for unhandledRejection and uncaughtExceptionMonitor. I finally managed to track it down to the ping function initiating a socket connection before the error handler had been registered to the socket as well as a promise in one of the profiles that didn’t have error catching. I think I have a plan for finishing up the Restore Points optimizations so I’ll start on that next.

1 Like

Alright, I was a bit optimistic the first time but I think I am really ready for a code review now. I managed to fix the issue of the calendar not loading by using the much faster --list-increments argument for rdiff-backup instead of --list-increment-sizes. This also speeds up the loading of the Restore Points page but the size data will be missing on first load. The user can still request the size data by pressing the get increment sizes button. Let me know what you think of these changes and if you need anything else done before this pull request is accepted.

1 Like

@hexparrot Any updates on this?

@hexparrot I found another bug in my code.