WebUI not fully loading

I’ve been dealing with an issue for quite a while now, and been looking for similar issues on the forums here but haven’t been able to successfully get it working.

I am able to log in with the mc account, but after about 15 seconds or so, it just stops loading any data, and I can’t do anything. Servers are not loaded so I can not stop/restart or update them, get to the console or logs, or even make new servers or download new profiles.

What I’ve done:

Reset WebUI with the commands ./update_webui.sh and ./reset_webui.sh
Update git commit to latest: git commit: 4bb4eb3
Restart server (not running on vm)
Running everything from this post: https://minecraft.codeemo.com/mineoswiki/index.php?title=Updating_the_Webui
Confirmed Permissions match what is shown in this post: Megathread for MineOS webui issues

Recent items from mineos.log file here: http://pastebin.com/W9ni3DEH

My servers run fine and I can get into the server via ssh, it’s just the WebUI that’s borked somehow. Please let me know what else I need to do to fix this.

A few things:

  1. Have you rebooted the host at any point? The logs suggest that the server is actually up and running when you’re hitting start (even if the webui isn’t reporting it).

  2. One of the hardest things for me to pin down is when modded servers respond in an unexpected way to the Minecraft Ping. While Forge has been known to do this pretty regularly, some other server types also modify their response to the official Mojang Minecraft ping and hold up the webui.

So far, though, out of the many times this has happened, nobody has ever been able to provide me with a reproducible case (they gave up on the server, or never followed through with sharing it with me), so it’s actually been a thorn in my side for some time now.

If you can let me know what kind of server you’re using, and the mods you’re using, perhaps I can reproduce the issue and try to implement a fix. Also, sharing an archive of your server (minus the ‘world’ directory) would do the trick. When the ‘world’ directory is removed, the archives should be on the order of a few megabytes (rather than the hundreds or thousands an archive might otherwise be).

But at any rate, restart the host (linux) and see if there’s a point when the webui goes bad. Maybe it works perfectly until you start a forge server and then the webui goes bad. This sort of behavior is pretty much 100% the #2 description I listed above.

I have rebooted the host previously and that didn’t seem to fix anything, but I’ll do it again and post a copy of the mineos.log file, unless something else is better. I don’t have any issues with the servers running, they’re going great, I just can’t do anything with them via the WebUI.

I’m running a Spigot server, currently the latest release 1.11.2, though was having this issue prior to that, it seems to be hit or miss, though since the last update its been completely unusable. The mods I have running are

  • BoilingCauldrons
  • Bookshelves
  • ChopTree3
  • CustomMobDrops
  • dynmap
  • InfernalMobs
  • LapisPortals
  • LastDeathLocation
  • PermissionsEx
  • RedProtect

I’ve had all of those running the whole time the server’s been up, but this issue has only surfaced the past 3 months or so, and been the worst since Dec.

If there’s a way to archive without getting into the WebUI, please let me know and I’ll do that for you. My most recent is back in July.

I’ll update again after I’ve had a chance to reboot the host. It’s just a small family server that doesn’t get a lot of use, but currently has a few people online so it may be a few hours.

Rebooted the host system and attempted to log into WebUI as soon as I saw the server online. Still unable to get it to load for more than 5 seconds. Log file appears the same, but added latest mineos.log here for you to review: http://pastebin.com/sxyEda2s

Screenshot of the WebUI attached.
Edit: Cant attach, should be visible here: http://i228.photobucket.com/albums/ee183/cray-z_bucket/mineos%20webui.png

Is there documentation somewhere that will show me how to run console commands on my server from command line? And also how to do tasks like upgrade the server, download profiles, make backups and archives, etc without the gui?

Hi.
Have you given @hexparrot the archive he requested? He have after all narrowed this down to a bug he’s not been able to reproduce. Since you have a server with the bug active it may be of great help to him, and will most likely end up in a release where you can use the WebUI without it crashing.

@hexparrot discussed console commands for mineos here:
https://discourse.codeemo.com/t/mineos-console-js-documentation/998/3

As for running MineOS in shell only:
I would say that doing this you do not need the MineOS part installed at all, since the WebUI really just is a web based graphical layer for management.

to start a server it runs a command much like :
screen java -Xmx1024M -Xms1024M -jar minecraft_server.jar nogui

Just swap minecraft_server.jar with your jar file since you do not run a vanilla server.
To learn more about screen: https://www.mattcutts.com/blog/a-quick-tutorial-on-screen/

Archives are done by using the “tar” command: http://www.howtogeek.com/248780/how-to-compress-and-extract-files-using-the-tar-command-on-linux/

Backups are done with r-sync: http://everythinglinux.org/rsync/

The only thing left then is really the profiles, which really only provides handy buttons to download server-files to a local file reprository on your MineOS server. These can easily be downloaded manually, and just copied into the server directories you need them in.

Thanks for the information on making archives. I’ve created a current one (minus the world folders and the dynamap/web folder (that was 22GB!)) and posted it here for you to check out:

http://www.mediafire.com/file/93ui2cyfnjtk975/MCHills_Mods-Archive3.tar.gz

Regarding the console commands, what I’m looking for is a way to interact with the live server and players on it via a console/terminal window, like how you could from the line under the log files in WebUI. I don’t have anyone with OP status on our server, so when things come up that need fixed or whatever I would do it through there using the commands from the various plugins or minecraft server commands from here: http://minecraft.gamepedia.com/Commands#Summary_of_commands. Sorry if this is a basic question, I’m not terribly familiar with these things.

There are a couple of ways of passing commands:

  1. Using ./mineos_console.js
    You’d have to supply a servername and command with variables depending on wht you wish to do. @Hexparrot knows more of those commands and how to build them properly.

  2. using the screen command.
    Since MineOS creates a screen (a self contained persistent shell you can create, disconnect and resume, so that you can have running apps even though you log out Se the link I provided in my previous answer). You can then connect to the screen session of the server you wish to interact with. If you only run one screen this is simple, there is only one screen session running. When you enter the screen session, you will basically see the same as in he log page of MineOS console, and you can send commands.

I downloaded this and I have identified a problem. I imported your server as name sample (not important), but what the key issue here is that inside this directory is an entire (unwanted) directory structure that is keeping MineOS from seeing your jarfiles.

root@mineos servers/sample# pwd
/var/games/minecraft/servers/sample
root@mineos servers/sample# ls
cron.config  server.config  server.properties  var
root@mineos servers/sample# ls /var/games/minecraft/servers/sample/var/games/minecraft/servers/MCHills_Mods/
BuildTools.jar       craftbukkit-1.10.jar    eula.txt                   plugins            spigot-1.11.jar
BuildTools.log.txt   craftbukkit-1.11.1.jar  help.yml                   server.config      spigot.yml
banned-ips.json      craftbukkit-1.11.2.jar  logs                       server.properties  usercache.json
banned-players.json  craftbukkit-1.11.jar    minecraft_server.1.10.jar  spigot-1.10.jar    wepif.yml
bukkit.yml           crash-reports           ops.json                   spigot-1.11.1.jar  whitelist.json
commands.yml         cron.config             permissions.yml            spigot-1.11.2.jar  whitelist.json.backup
root@mineos servers/sample#

Question 1: is this file you’ve uploaded to me an accurate representation of what is on your MineOS system? If so, a fix is easy by moving all these files from their current location to their intended location.

Question 2: What’s the name of your dynmap folder (that exists on your system, but not in this archive)? It’s good you didn’t include the contents, but the folder name might be important. Basically, what could be happening here is that it could be an issue with one of the nodejs modules that tracks filechanges in server directories.

If, as an example (and as dynmap is known to do), it is creating hundreds of thousands of files that nodejs has to track, then this is what is crippling the webui. Known dynmap directories I have (or at least tried to) have removed from tracking, to remove this memory/cpu bloat.

Also, this could be (but is unlikely) caused by too many files in your world directories. Typically world is the default and it’s kept. But sometimes additional worlds make additional directories. Regionfiles typically address this, but it might be time to address this in a different way in code to fix this once and for all. Can you provide to me the names of all the directories in /var/games/minecraft/servers/MCHills_Mods? (not the contents, just the directory names should do).

Yeah, thats my /var/games/minecraft/servers/MCHills_Mods folder. I didn’t put anything there on purpose, that’s just where it’s always gone.

It was dynmap/web that I excluded. In addition to a handful of files, it contains the sub-directories ‘css’, ‘images’, ‘js’, ‘standalone’, and ‘tiles’. It looks like ‘tiles’ is the big one with all the server map images in it.

crash-reports, logs, plugins, world, world_nether, world_the_end. Inside ‘world’ is data, playerdata, region, and stats. Inside world_nether and world_the_end is DIM1, data, and playerdata.

This might be the offending directory. If you feel comfortable editing one of the scripts, line 580:

var fw = fireworm(instance.env.cwd, {skipDirEntryPatterns: ['world', 'dynmap', 'plugins', 'web', 'logs', 'region', 'playerdata', 'stats', 'data']});

to say

var fw = fireworm(instance.env.cwd, {skipDirEntryPatterns: ['world', 'dynmap', 'plugins', 'web', 'logs', 'region', 'playerdata', 'stats', 'data', 'tiles']});

And then reboot the host machine entirely.

This could very well solve the issue you’re having. If it does, I think I’ll have a good idea of what changes I could make to the public code that could identify this sort of issue, and prevent the memory bloat.

I updated the server.js file with the modified code and restarted, but it didn’t seem to change anything.

Hrm, perhaps just to make sure, we should add the other directory names as well. What were worried here is about directories with high file counts (not file sizes).

Can you add images, js, standalone, and css and see if it works any differently?

That seems to have helped a little bit, one of my servers (MCHill) loads before it locks up now. Looking deeper into the dynmap/web/tiles folder, it looks like things are broken down into hundreds of folders, each containing 1-1000’s of files.

Under ‘tiles’, there are three folders, ‘ct’, ‘flat’, and ‘t’. Each of those folders contain a number of sub folders named 0_0, 0_1, 0_-1, etc. Folder ‘t’ has over 1500 of these. Each of these folders can hold 1000’s of png files (some only have 1, others have 1500+). Is it possible to exclude file extensions instead of just names? .hash and .png are the two file types I see here. I’m not sure how you would make a filter to skip all the #_# folders.

This is the folder structure of the smaller of the three ‘tiles’ subfolders.
/dynmap/web/tiles/world/ct
-1_-1/ -1_2/ -2_1/ 0_-3/ 0_3/ 1_-4/ 2_-3/ 3_-3/ 4_-3/ 5_-4/
-1_-2/ -1_3/ -2_2/ 0_0/ 1_-1/ 1_0/ 2_-4/ 3_-4/ 4_-4/ 5_-5/
-1_0/ -2_-1/ 0_-1/ 0_1/ 1_-2/ 2_-1/ 2_-5/ 3_-5/ 4_-5/
-1_1/ -2_0/ 0_-2/ 0_2/ 1_-3/ 2_-2/ 3_-2/ 4_-2/ 5_-3/

I think I have found a better fix, and I made the commit in a new branch. This will eliminate the need for admins to ever have to be aware of high directory-count paths (at least if the fireworm documentation is to be believed).

Log into your terminal as root:

cd /usr/games/minecraft
git fetch
git checkout fireworm_tracking

Then, reboot your host machine. Rebooting your host machine is not necessary for most webui adjustments like these, but the issue you’re having is likely having an unwanted side-effect of leaking a lot of memory, which is what the reboot should reclaim.

If you’re able to get right into the webui and it feels responsive, I’ll push this into the main branch.

When running this I get the error “error: pathspec ‘fireworm_tracking’ did not match any file(s) known to git.”

root@mineos games/minecraft# pwd
/usr/games/minecraft
root@mineos games/minecraft# git fetch
root@mineos games/minecraft# git checkout fireworm_tracking
error: pathspec 'fireworm_tracking' did not match any file(s) known to git.
root@mineos games/minecraft#

Hrm, this is a weird error. Definitely hasn’t come up for me, even when now attempting with a fresh install.

That said, this SO post says we might be able to fix it with these command:

/usr/games/minecraft
git remote update
git fetch 
git checkout --track origin/fireworm_tracking

This one gives me the following error:

root@mineos games/minecraft# git checkout --track origin/fireworm_tracking
fatal: Cannot update paths and switch to branch 'fireworm_tracking' at the same time.
Did you intend to checkout 'origin/fireworm_tracking' which can not be resolved as commit?

I’ll poke around at the stack overflow comments and try some stuff to see if I can get it to work. Is there a way to manually pull it and upload the files?

------Edit 1

Using the below command I got something (not sure what a success should look like), but after rebooting the host machine, it didn’t seem to make any difference with the loading.

root@mineos games/minecraft# git checkout -b origin/fireworm_tracking
M       server.js
Switched to a new branch 'origin/fireworm_tracking'

Rolled back with /bin/bash reset_webui.sh

-------Edit 2
Ran the below, received the message shown:

root@mineos games/minecraft# git checkout -b fireworm_tracking
Switched to a new branch 'fireworm_tracking'

Rebooted, tested webUI, no noticeable change. Rolled back with /bin/bash reset_webui.sh

git checkout -b — the b part create a new branch, which means that even after the host reset, you weren’t using the commit that exists on github–you inadvertently switched to a newly created branch with the same name (but not the same contents) as the hosted one.

Alternatively, it’s also possible to:

a) supervisorctl stop mineos
b) delete the entire /usr/games/minecraft directory, reinstall the webui (just the github part of the instructions) and then switch?

Was this based off a MineOS Turnkey installation or a different installation?

Yes, this was installed with the Turnkey installer, done back about a year ago.

To confirm, do you mean doing this after deleting the folder to reinstall:
git fetch
git merge origin/master

and then trying:
git fetch
git checkout fireworm_tracking

or do I need to follow these steps: MineOS | mineos-node ?

Thanks!