I have noticed over the last couple of weeks that the web UI in my mineOS install takes slightly longer to populate servers and their statuses, even after I reboot my VM / VM host. Today, now the UI seems to lock up / time out entirely, so I cannot control my servers from it at all. I’m thinking one of the plugins in my server is probably the root cause of this, but I’m just curious if there is any debugging from mineOS itself that might help me pin this down. Right now I’m in the process of removing mods one at a time and rebooting my VM (since I can’t stop and restart the server in the web UI) to see if this makes any difference, so far no dice.
What I mean by locking up or timing out- after I log in, the statuses of all three of my servers never loads, the up time timer stops counting and the player / online server counts don’t populate. Sometimes the server list drop down does not populate at all, but if it does, even if I go to the server that is online (I only use one of the three installed right now), its status does not seem to be live, as shown by a white “down” indicator instead of red (one thing to note, the server is actually up and players are online).
I have reset and updated the web UI, made no change. My VM host has 8 cores and 30gb ram total, so I do not feel it is a resource issue. I’m assuming one of the plugins is causing an error somewhere but I do not know where to dig to find this conflict. This behavior occurs in both Chrome and Edge. Any suggestions or pointers / a direction to look in would help a lot. Thank you as always.
so i read your post and right away i thought ‘he has boot on start up checked’.
sure enough i looked at your screens and saw the box checked.
i can not solve the issue however by unchecking the box i think you will find that by not starting any servers you will be allowed to utilize the webui without it locking up.
then you can start the troubleshooting by starting one server at a time, allow it enough time to fully start. if this server did not lock up the webui then stop it and move on to the next one.
if all three start properly one by one without locking up the webui, try two at a time in different combinations, then all three at once to see how far you can push things. this may help point to what server has an issue, if any.
let us know how you do. something to try while you wait for better advice.
cheers!
tNt
EDIT: to add there was an issue regarding allocating too many cores on multiple core machines. i am not certain this was ever resolved, it may have been but i have always only allocated one or two cores per server. please search the googler for more current info re multi threading and minecraft.
Thanks for the info - interesting. So you think having the server boot automatically on startup could cause it?
I know which server is the problem child, I only use one of the three I have setup right now, the other two are always offline (but I don’t quite feel like backing them up to take them down yet lol)
what mine does is alot like yours so what i choose to do to help mitigate is log into my webui, choose my server i want to start, choose the jar and hit start. refresh the server list and look for the utilization to ramp up to confirm it is starting, then log off.
then wait long enough for it to complete and log in. if i did not wait long enough, log out.
if i stay logged in the webui locks up. upon inspection, while booting it seems there is too many lines generated by the logs overwhelming the webui. then it locks up. it says to just refresh it but i found only logging out works.
once it has started and stable the webui behaves as per usual with no further issues.
if i could figure out what was causing it i would fix it, lol but no big deal since once done starting everything is ok.
so that should mean starting the server on start up is not the issue, it is logging into the webui before it is done starting, lol.
test it by checking the box, letting it start by itself but wait a while, log after it’s done to see if the webui is ok. avoid logging in too soon, i mean.
good luck!
tNt
edit to add: but on mine, even if it locks up i can still log out normally for some reason and i can just log right back in. if the server is done starting all is well.
I actually tried that just now - via putty, edited server.config to have [onreboot] start = false, then rebooted my VM. My third server doesn’t populate. I know it’s the third one because the other two are not in use, and when it’s trying to populate the third one, the heartbeat / usage diagram in the center stops updating. It’s good that the game server itself begins, but not being able to control it is a bit of a drag from an administrative perspective. Really any logging from the web UI would be really helpful, and I could figure it out after that if such logging / debugging exists.
Not that I can see, although this is useful to look at. Only things here are that the servers either spin up or don’t, depending on the “boot on startup” property.
However after I dug through the logs from /var/games/servers/minecraft/rBeachCity (the problem child), I found the source - a plugin, Dynmap, had problems trying to generate tiles and was spitting out a ton of I/O errors. However, I removed that plugin and its entire folder, as well as that log file (because it was ~90mb), but the UI is still struggling to load.
Even right now, after doing some file restoration, I re-enabled autostart on this server. The list of all three servers seems to populate consistently now, but the status of this server never comes up, and if I drill into it none of the controls work and the console / log does not appear so I can’t really control it. However, it is still booting up in the background, now I simply can’t control it.
Any suggestions hexparrot? I know it’s an issue with this server and not the UI / mineOS itself, but short of tearing it down and rebuilding, I’m just stumped from a troubleshooting perspective. Alternatively, without being able to access the console, is there a way to issue an rdiff command to restore a backup? I’ve been making incremental restore points ever 4 hours so I can push it back ~48 hours if I need to.
edit: I doubt there’s anything useful in here, but here’s /var/log/mineOS.log from the time that I updated and reset the web UI using the scripts. https://pastebin.com/qXZUKUuJ
edit edit: using du * -sh I found that the ‘region’ folder under the world folder is 2.5gb. Could that be the problem / is that too large?
In the meantime, I pulled out everything from /servers/rBeachCity, rebooted, and it’s still seizing up trying to populate. My next target is /backup because I’m wondering if it’s locking up while trying to analyze all the garbage data that dynmap created because my backups and archives don’t populate either - I have a cron job create a backup every 4 hours and an archive once a week. Pruning this now, gonna take a while.
Let me know if I can provide any specific details that may help.
Got it figured out, my suspicion was correct - I cleared out my /backups/rBeachCity folder (followed by sync sync sync reboot, just to be safe lol) and now the UI responds again perfectly. So having incremental backups of so many tiny files I guess is what caused it to choke up. Makes sense from an IO standpoint though; 10.2gb of backup in the form of 1,980,766 files will make any disk / OS start to sweat if it has to analyze it on a regular basis.
From now on I’ll figure out a way around that with dynmap / and or make more archives and less incremental backups perhaps, however if you cook up something to counteract this for everyone else, I’m definitely all ears.
Thanks for the feedback, and as always, thank you for making mineOS!
You are a gentleman and a scholar. Updated, will report back.
edit: just curious, I assume var default_skips is a list of directories a backup will ignore? ie, if I needed to ignore another mod / etc down the road, or if I removed something such as playerdata to include that directory.