I cant even access the page it self. The last time the settings were on was 4gb of ram. and still has 12gb left over this doesn’t fit my case.
I’m also having a similar problem, although for now I am still able to connect, each passing day it takes more and more time for the web-ui to load anything.
The server size is at arround 40go for now and growing as my players go about their minecraft lives.
I’m on a fresh install of MineOS Turnkey, running an ESXi server
I have tried throwing more ressource at the vm but it doesn’t seem to help.
At some point the VM had 128go of ram and 40 vcores, and it just was not using the cpu, so I don’t think it’s a compute problem, the ram usage did not reach the 96go I allocated to the minecraft server in over 12h of active play from my users.
I also don’t think it could be from the throughtput of my ssds, as it’s running on a ssd raid array capable of well over 4GB/s.
I could also be wrong and grossly misunderstood and/or overlooked something, but I’m quite at a loss here.
For more details :
From looking at top during the web-ui long loading times is that the node process uses over 300% cpu and just keeps using more and more ram up to about 4.3-4.5go where it will sit for a good 5-10min before loading and going down to 3.0go.
After that it will hover between 3.5 and 4.5go until I am done and closes the browser tab.
1 Like
That I can confirm I have similair problems the longer the VM is on the longer the load times are and the more the connection times out. After a day already the web interface is slow enough to not be working as intended. I recently also updated the Web_UI but that didn’t really fix anything. My specs are a bit slower then @Mathox but it should still suffice when running only one server and nobody is online. It’s currently running a I5-6600 and 40GB of RAM (of which 32GB is able to run at 3600MHz) clocked down to 2133MHz, I believe this shouldn’t really be a big issue but it’s often too slow to be bearable. I do want to mention that I only started to notice these problems I believe, 6 to 8 months ago.
Edit: the VM has about 24 to 28GB of allocated RAM and all the processor cores
I do not think this i s a MineOS problem, but a minecraft, chunk, and java problem affecting MineOS. There are several posts on the internet talking about similar issues (usually ingame lag of different types):
- https://www.reddit.com/r/admincraft/comments/ar6ddj/does_a_larger_world_increase_server_lag/
- https://www.spigotmc.org/threads/server-world-size.230290/
- https://www.akliz.net/manage/knowledgebase/13/How-to-Resolve-Minecraft-Server-Lag.html
The short version of it is the number of chunks and entities the server has to handle simultaneously. There are workarounds: Smaller worlds, staying closer toghether, fewer players online simultaneously. These only deals with avoiding triggering it thoug, not fixing anything.
The number of chunks and files in a minecraft world grows large quite quick. The more of these files your server needs to keep in active memory and track (due to user activity in the close area), the more resources your computer needs. As you say: your VM had plenty of resources, but I still think that its a matter of numbers of files your server needs to juggle using java.
Java is unfortunately rubbish at handling oads of active files, and even worse at doing cleanup. From my work we know that the systems using java web-UI’s need a client reset occatinally to keep functional. If we fail to do this, our WebUI freezes within a week.
MineOS is opensource, and available through github, so if anyone has any ideas of how to improve MineOS’s handling of java and java handling huge servers and loads of files please take a look: https://github.com/hexparrot/mineos-node
If you in addition run a map-creating utility or addon, it will bog down your server severely when it tries to check every world chunk for changes. When I create maps for my minecraft servers I usually do this from a separate computer, reading from a shared drive.
1 Like
Thank you for your detailed response @iMelsom.
So basically java would be choking itself to death on the shear number of files nescessary to run a minecraft server and taking MineOS down with it ?
I’m ok with this explanation but for my case the minecraft server side of thing runs butter smooth even with over 50 players.
Only MineOS web-ui is slowed to a crawl, the rest of the system is also fine. And is doesn’t seem to make a difference as to how many players are online.
On another note I will try to restart the whole system tonight to see if it help the web-ui.
I also noticed that the drive activity spikes when loading the web-ui. Could MineOS be trying to read every single file in the server directory ?
@Hexparrot has put in limiters on spesific folders to make sure MineOS ignores the folders having the highest number of files, but yes, MineOS may bog down due to number of tiles in a server directory:
var default_skips = ['world', 'world_the_end', 'world_nether', 'dynmap', 'plugins', 'web', 'region', 'playerdata', 'stats', 'data'];
(from: https://github.com/hexparrot/mineos-node/blob/master/server.js )
So as far as I know, MineOS should ignore any dir known to contain huge amounts of files. Of course, using a custom world name, may force MineOS to look inside a folder anyway, but even inside the “world” directories there are dir that are skipped by default (region and player).
HexParrot can probably better answer what mecanisms used to prevet MIneOS from hanging on reading through loads of files.
As for the minecraft server itself (the java part) it is handled throug the linux service “Screen”, so each server is run an contained separately as distinct proscesses…
This may be a priority issue. The webserver serving the WebUI is lower priority for the host OS than the minecraft server jars, and thus handed less resources.
You could also try the tips in this link : https://scoutapm.com/blog/slow_server_flow_chart
I did take a look at top before, that is what led me to believe this issue was not ressource bind directly. If there is a limitation it might be in the ressources node is able to actively use or something of the sort. For context my Cpu is >80idle at all times and my IO doesn’t go over 0.2-3.
Today I’ve tried to both update mineos and restarting either mineos with supervisor or the whole system. And it doesn’t help, I can’t even get to the webui as the page itself times out.
When restarted the webui semi-works for a few seconds after the login. I can see realtime console output but some ui element are not yet loaded completely.
And that stop here, somewhere after that something goes wrong and node starts using ram like crazy and pretty much dies.
The site quickly becomes unresponsive and then straight up times out until a restart, while the system is still fully responsive and the minecraft server runs fine.
This is quite cumbersome as I have to resort to query manualy mine-os with mineos_console.js to get the screen pid and get the console.
Edit : I can also just list my screens that’s just me not thinking after my long day of work.
1 Like
That’s exactly how I am experiencing it.
And I think I can clearly see that the web-ui’s process is probably called node which is the one with like a very high usage all of the sudden while idle time is above 50%.
Is it possible to change this? So that when I want to get the consoles and web-page that it maybe prioritizes that above the jars. Because a friend of mine wants to host a server but I am gonna give her a account on the server to use. But it wouldn’t be possible if the web-ui is timing out all the time.
I know that this doesn’t seem to be a hardware issue, but from the posts, it sounds like those contributing are using VMs. Has anyone tested/built a bare metal system as a dedicated server? Would someone be willing to share their large file world? I’m interested in testing on my server.
@Mathox I spotted something weird in my logs and was wondering if that might be in yours too. What I saw was whenever my timeouts started/the web page slowed down. Is that it almost always stops at a specific point. In my case it is transmitting logs/latest.log from one of my servers. But it hasn’t send all logs yet. It gets stuck at this one server currently running. But it might all be coïncidence.
So I am currently trying what happens when I throw away (RM) all the servers that I am not currently using.
Edit: Furthermore I have discovered that the program named or “command” node (which is probably the web-ui considering nodejs) sometimes has it’s what I call “Rage” moments this is the moment it doesn’t react and the %CPU is 10x as much compared to what it is normally. (viewed in top) even though the CPU is still idling 58% the web-ui are in these moments completely unreachable until it cools down when it is for a bit of time really easily reached.
How are you currently running it if I may ask? (sorry if this question sounds sarcastic it because it’s not meant to be)
MineOS should only ask for tha latest entries in the “latest.log”, and it should only read the last few lines of that log. I tried to take a look into the code, but cant see any place that should do anything different.
It could be interessting to see the size of the log files it tries to read. I guess it might be a problem with the log file length, but since we ask for the last lines of the file, (not anything specific or anything in the middle) it shouldn’t be to resource heavy gett ing the tail. Then again. if I try to load a really large log file into nano, it takes ages, and the log file size may have an impact. Log files are created new every day, but for large worlds (with many users) the file may grown large quickly due to the amount of items logged.
This is purely speculation though.
I’ve taken a look at my logs and I don’t even get as far as you.
It just times out after the “joined server namespace”.
I’ve made a chronology of what event I could gather.
1- Log in the webui
2- Web-ui is responding but not updating (cached most likely)
3- node goes into ‘rage’ mode and at this point the mineos web site can’t be accessed
4- node calm down from ‘rage’ and the web site is now available
5- go back to step one and repeat…
1-2-3 are pretty much instantaneous, while 4 seems to take a while.
@iMelsom My latest.log seem to be getting out of hand at times indeed but at this moment it’s 305k so nothing unmanagable
Then we have two possible scenarios:
- Huge log files makes for very slow tailing
- adding large server to namespace makes the webui outgrow it bounds but trying to load too much .
I somewhat doubt the first one, since there are spesific functions and variables in most development languages to define what end of a file to read from.
The only place “Joined namesapce” turns up in server.js is int this “else” in a if-else:
else {
logging.info('[{0}] {1} ({2}) joined server namespace'.format(server_name, username, ip_address));
socket.on('command', produce_receipt);
socket.on('get_file_contents', get_file_contents);
socket.on('get_available_tails', get_available_tails);
socket.on('property', get_prop);
socket.on('page_data', get_page_data);
socket.on('cron', manage_cron);
socket.on('server.properties', broadcast_sp);
socket.on('server.config', broadcast_sc);
socket.on('cron.config', broadcast_cc);
socket.on('server-icon.png', broadcast_icon);
socket.on('config.yml', broadcast_cy);
socket.on('req_server_activity', broadcast_notices);
}
So if it hangs on server join to namespace, it has entered this part of the script…
I’m not sure if I fully understand your question, but I am running an HP Proliant DL380 G5 system. The specs are: dual quad core Xeon 2.33 GHz with 16 GB ram and 3 SAS 146GB had setup in RAID-5. It’s local so it isn’t impacted by the bandwidth as some SaaS solutions may be. And being a server, it is designed for data throughput.
What if mine-os fails to get a tail does it just go leave it be or keep trying?
Also where would we see server-icon.png I have actually never seen any server icon png’s.
I think the 2nd scenario might be really likely although I am not very advanced with nodeJS/this area. But I removed my log files and even then it’s still timing out.
I am just gonna post a few forum posts that might be comparable to this topic/problem:
gonna try to run a possible solution but I am waiting for the server to finish up zipping and transferring a backup in case it goes wrong.
Edit:
So I just completed the backup and deleted my biggest server from mineOS. And, it fixed it somehow so it’s probably the file size considering that the tar.bz2 file was already 15GB (my whole server was 30GB in tar.bz2 yesterday) the web-ui runs a lot more fluently now. But at the time of writing it just timed out again, telling me that it still isn’t optimally performing. I might just try installing the version I saw coming by of @JayMontana36 while looking around on the forums.
So yea whatever @hexparrot has coded does not work.
I can confirm if you run on an SSD this still occurs but after like 4 mins which is enough time to start or fix something on the control panel.
So HDD has 0 support essentially. This is kinda sad not being to host a big survival server 
Ok so a bit late but it seems fixed on my side of things.
I had to move my dynmap another server so I switched to a mysql storage instead of the default filetree, I then deleted the files in /plugin/dynmap/web/tiles for my 3 worlds.
MineOS is now more resonsive than ever, so my best guess to the issue is that the code @hexparrot wrote to skip some directories must not work as intended.