So I have a big server and the world is 100GB the problem is that once the world got biggger the slower it got. Once it has actually reach 100gb of data the Web-UI just freezes with a white screen or auto log me out when reloading the browser. How can I fix this limit?
I tried restarting the VM. Force Stopping all servers and still cant login. (Only the survivals server loads on boot and killing it with linux seems to no effect.)
This sounds like it is a capacity issue. How much space and RAM have you allocated for the VM in total? Are you able to spin up a server on a bare metal box and import that same world? While the 100gb world is running, can you join other worlds on the same server, and is it playable?
Storage on the VM is 700GB so only 12% is used. I am allocating 18GB of ram to the VM. The server runs fine 100% no tps issue and can save/load fine. I can load worlds fine. Its just once a world size has reach 100GB the WebUI becomes really impossible to login.
The Host computer has 48GB of ram so no issues here. (Windows 10 Pro)
I cannot test this on a metal box due to world problems at the moment.
Also Logging as another user doesnt auto log me out or white screen and loads 100% but then after 5 seconds it stops loading the cpu load graph and this indicates you been logged out.
What do you have set the Java maximum heap size to? Try increasing that, but leave the minimum empty. That’s the only other thing that I can think of, other than an issue with the node-js.
Either @iMelsom or @hexparrot may better dissect the code.
What? How am I suppose to increase the MineOS WebUI ram? doesn’t it do that automatically?
It seems that this kinda happens on Harddrives. SSD works just fine over 100GB but seems that I need to actually get a bigger SSD for a survival world. Not sure it’s because of the speed of the SSD or just can’t handle reading the data (even tho it’s just 2 servers) So my guess is instead of waiting for the HDD to get done reading the big world it simply times the user out early which not sure if that’s configurable.
Adjusting the Java heap size only allocates a portion of RAM for it to use, and setting a minimum, I’ve seen, causes problems. You should not max out the heap to the system’s max RAM as you need some for the OS and other non-MineOS apps.
Unless the SSDs are of shoddy quality, they should out-perform HDDs. If this issue is due to through put, then I’d be interested to see what happens on a “enterprise class” bare metal server.
The SSD are out preforming the HHDs? The Survival is on an HDD right now because HDDs are cheap and high storage for a 7200rpm which loads fine.
I was saying If its the HDD problem then I was gonna look for a SSD with 1-2 TB storage.
I am not even 100% Sure how to even adjust the Java heap for the WebUI. A document or something could help me.
When you access the world server you will have a section that for Java settings, -Xmx and -Xms. If the Xmx is too high, then the OS could become unstable.
I cant even access the page it self. The last time the settings were on was 4gb of ram. and still has 12gb left over this doesn’t fit my case.
I’m also having a similar problem, although for now I am still able to connect, each passing day it takes more and more time for the web-ui to load anything.
The server size is at arround 40go for now and growing as my players go about their minecraft lives.
I’m on a fresh install of MineOS Turnkey, running an ESXi server
I have tried throwing more ressource at the vm but it doesn’t seem to help.
At some point the VM had 128go of ram and 40 vcores, and it just was not using the cpu, so I don’t think it’s a compute problem, the ram usage did not reach the 96go I allocated to the minecraft server in over 12h of active play from my users.
I also don’t think it could be from the throughtput of my ssds, as it’s running on a ssd raid array capable of well over 4GB/s.
I could also be wrong and grossly misunderstood and/or overlooked something, but I’m quite at a loss here.
For more details :
From looking at top during the web-ui long loading times is that the node process uses over 300% cpu and just keeps using more and more ram up to about 4.3-4.5go where it will sit for a good 5-10min before loading and going down to 3.0go.
After that it will hover between 3.5 and 4.5go until I am done and closes the browser tab.
That I can confirm I have similair problems the longer the VM is on the longer the load times are and the more the connection times out. After a day already the web interface is slow enough to not be working as intended. I recently also updated the Web_UI but that didn’t really fix anything. My specs are a bit slower then @Mathox but it should still suffice when running only one server and nobody is online. It’s currently running a I5-6600 and 40GB of RAM (of which 32GB is able to run at 3600MHz) clocked down to 2133MHz, I believe this shouldn’t really be a big issue but it’s often too slow to be bearable. I do want to mention that I only started to notice these problems I believe, 6 to 8 months ago.
Edit: the VM has about 24 to 28GB of allocated RAM and all the processor cores
I do not think this i s a MineOS problem, but a minecraft, chunk, and java problem affecting MineOS. There are several posts on the internet talking about similar issues (usually ingame lag of different types):
The short version of it is the number of chunks and entities the server has to handle simultaneously. There are workarounds: Smaller worlds, staying closer toghether, fewer players online simultaneously. These only deals with avoiding triggering it thoug, not fixing anything.
The number of chunks and files in a minecraft world grows large quite quick. The more of these files your server needs to keep in active memory and track (due to user activity in the close area), the more resources your computer needs. As you say: your VM had plenty of resources, but I still think that its a matter of numbers of files your server needs to juggle using java.
Java is unfortunately rubbish at handling oads of active files, and even worse at doing cleanup. From my work we know that the systems using java web-UI’s need a client reset occatinally to keep functional. If we fail to do this, our WebUI freezes within a week.
MineOS is opensource, and available through github, so if anyone has any ideas of how to improve MineOS’s handling of java and java handling huge servers and loads of files please take a look: https://github.com/hexparrot/mineos-node
If you in addition run a map-creating utility or addon, it will bog down your server severely when it tries to check every world chunk for changes. When I create maps for my minecraft servers I usually do this from a separate computer, reading from a shared drive.
Thank you for your detailed response @iMelsom.
So basically java would be choking itself to death on the shear number of files nescessary to run a minecraft server and taking MineOS down with it ?
I’m ok with this explanation but for my case the minecraft server side of thing runs butter smooth even with over 50 players.
Only MineOS web-ui is slowed to a crawl, the rest of the system is also fine. And is doesn’t seem to make a difference as to how many players are online.
On another note I will try to restart the whole system tonight to see if it help the web-ui.
I also noticed that the drive activity spikes when loading the web-ui. Could MineOS be trying to read every single file in the server directory ?
@Hexparrot has put in limiters on spesific folders to make sure MineOS ignores the folders having the highest number of files, but yes, MineOS may bog down due to number of tiles in a server directory:
var default_skips = ['world', 'world_the_end', 'world_nether', 'dynmap', 'plugins', 'web', 'region', 'playerdata', 'stats', 'data'];
(from: https://github.com/hexparrot/mineos-node/blob/master/server.js )
So as far as I know, MineOS should ignore any dir known to contain huge amounts of files. Of course, using a custom world name, may force MineOS to look inside a folder anyway, but even inside the “world” directories there are dir that are skipped by default (region and player).
HexParrot can probably better answer what mecanisms used to prevet MIneOS from hanging on reading through loads of files.
As for the minecraft server itself (the java part) it is handled throug the linux service “Screen”, so each server is run an contained separately as distinct proscesses…
This may be a priority issue. The webserver serving the WebUI is lower priority for the host OS than the minecraft server jars, and thus handed less resources.
You could also try the tips in this link : https://scoutapm.com/blog/slow_server_flow_chart
I did take a look at top before, that is what led me to believe this issue was not ressource bind directly. If there is a limitation it might be in the ressources node is able to actively use or something of the sort. For context my Cpu is >80idle at all times and my IO doesn’t go over 0.2-3.
Today I’ve tried to both update mineos and restarting either mineos with supervisor or the whole system. And it doesn’t help, I can’t even get to the webui as the page itself times out.
When restarted the webui semi-works for a few seconds after the login. I can see realtime console output but some ui element are not yet loaded completely.
And that stop here, somewhere after that something goes wrong and node starts using ram like crazy and pretty much dies.
The site quickly becomes unresponsive and then straight up times out until a restart, while the system is still fully responsive and the minecraft server runs fine.
This is quite cumbersome as I have to resort to query manualy mine-os with mineos_console.js to get the screen pid and get the console.
Edit : I can also just list my screens that’s just me not thinking after my long day of work.
That’s exactly how I am experiencing it.
And I think I can clearly see that the web-ui’s process is probably called node which is the one with like a very high usage all of the sudden while idle time is above 50%.
Is it possible to change this? So that when I want to get the consoles and web-page that it maybe prioritizes that above the jars. Because a friend of mine wants to host a server but I am gonna give her a account on the server to use. But it wouldn’t be possible if the web-ui is timing out all the time.
I know that this doesn’t seem to be a hardware issue, but from the posts, it sounds like those contributing are using VMs. Has anyone tested/built a bare metal system as a dedicated server? Would someone be willing to share their large file world? I’m interested in testing on my server.
@Mathox I spotted something weird in my logs and was wondering if that might be in yours too. What I saw was whenever my timeouts started/the web page slowed down. Is that it almost always stops at a specific point. In my case it is transmitting logs/latest.log from one of my servers. But it hasn’t send all logs yet. It gets stuck at this one server currently running. But it might all be coïncidence.
So I am currently trying what happens when I throw away (RM) all the servers that I am not currently using.
Edit: Furthermore I have discovered that the program named or “command” node (which is probably the web-ui considering nodejs) sometimes has it’s what I call “Rage” moments this is the moment it doesn’t react and the %CPU is 10x as much compared to what it is normally. (viewed in top) even though the CPU is still idling 58% the web-ui are in these moments completely unreachable until it cools down when it is for a bit of time really easily reached.
