Cant Keep up! Is the Server Overloaded?

Server-Software Forge
1

Hi there,
First off, want to say I have been confidently using MineOS for quite some time now, and have been a fan of it for ages. Started using it back on a samsung laptop, then moved it to its own PC, and now a Dell T710 running Proxmox. Gotta say, its a brilliant piece of software. Today however, I think minecraft was not so brilliant, and its lack of brilliance is being shown through harder by a triggered command by MineOS. Lets start at the beginning here.

I spent a fair bit of time over the course of a week trying to reduce lag on my Forge 1.18.1 (39.0) Server, and had no luck. So I started ripping out mods thinking that maybe some mod is just not jiving with the others. While this is sort of the case, its not entirely the case. I monitored my server closely and found that every 5 minutes nearly on the dot, it was more closer to 5 minutes and 10 or 15 seconds, but it was very well regulated, the server would lag for 40 - 200 ticks, then come back. Suffice it to say, this is not what I would consider playable.

So this morning, as of two hours before posting this, I figured I would spend a lot more face time with the server and continued ripping mods out trying to find the source. It was my surprise that the first mod I ripped out solved the problem while I was running the server from the command line. So I launched the server, joined it and waited a few minutes, then serious lag, regularly timed every 5 minutes.

I thought it might be my chron settings, but no, I dont have many commands, and only stuff that starts at 2330, and midnight to announce a server reboot and a then a restart. So I went back to ripping out mods, and while doing it from the Web GUI, none of the mods I ripped out of the pack seemed to solve the issue, the lag length got smaller, but the regularly timed rubber banding on the server never changed.

I figured I would go back to the command line, and well, after running it for over 30 minutes monitoring my lag alerter (a cow farm with a bunch of cows just stuffed enough to have them bump against each other, but not cause lag), theyve been happily shifting around the entire time without lagging.

So I propose a question here, what is it that is running every 5 minutes and a few seconds that would cause this lag when run through the Web interface instead of the command line?

MineOS Version: git commit: 11fa164
Forge Server Version: 39.1.2
Minecraft Version: 1.18.1
Number of mods: 153
Server Hardware: Dell T710 w/ Proxmox
MineOS VM RAM: 32GB, 16GB dedicated to forge server

Any assistance would be appreciated.

2

Every 5 minutes the Minecraft server does saving by default unless you tell it not to by using save-off which i don’t recommend doing btw.

Here is a snapshot of a stock standard vanilla server that is not lagging you can see little mini peaks in the graph where the server is losing 3% tps each time this will be the server saving as each peak starts 5 minutes after the last one so these are definitely caused by the saving.

We can also see that World saving is the next thing in the tree after full server tick (which full server tick is just the main game loop) this also tells us one the server is not lagging at all if full server tick is the highest priority for “Lag” and two it corroborates the peaks on the graph as world saving is the next highest process causing lag after the game loop and we only see orange peaks on the graph every 5 minutes no other times than that.
So this server is not a good example because its not lagging but we will just act like it is in theory.

On a modded server your peaks on the 5 minute intervals are going to be massive compared to these ones. i have a modded server but cannot be bothered booting it atm to show you because it takes a while to start up and get into etc but i think i have relied enough info to you so far that you can work it out.

You will want to do the same and profile your server the longer the better 1 hour is a good time because you can read the graph easier. What you want to do is look at what is causing lag around your saving time as i would say that you have many mods saving all sorts of things at this time.

3

I cant disagree with your assessment, however, when I run it outside of MineOS’s procedure it does not lag. So one would deduce that there is something inside of MineOS’s Server Operations that is causing the lag to be worsened.

4

Well you asked what is running every 5 minutes this is something that runs every 5 minutes and is also something to look at especially on modded servers because it can cause the exact issues you are describing.
I have seen modded servers dip down into the single digit TPS range which becomes unplayable for that moment while the world is saving.

20 TPS over 1 second is considered not lagging and 1 tick is 0.05s long. i have seen at saving time modded servers dip down to around ~10 TPS range which is obviously experienced as severe lag to the player/s. Which again happens every 5 minutes because its hard coded into Minecraft.

I was just going off of what you relayed in your opening post as you seemed to think it was mods initially. Now that you state that it could be something external I doubt MineOS is doing anything silly i have never seen it to be the cause of this sort of problem.

Mods add all sorts of complexity and they can also be poorly codded or doing things that are not very performant Forge especially for years although how many good mods it has was poorly optimized. Minecraft itself sucks severely when you start adding mods to it. Minecraft also attracts many mod devs that have never coded prior and Minecraft was how they learned. These are all factors

It has gotten better over time but even to this day it sucks and is not a very performant game when we start modding it is evident especially post 200 mods range but i have seen problems with less mods.

It will be something in the game itself i mean maybe not i could be wrong computers can be strange sometimes and also easier to diagnose in person but i would look at what the game is doing foremost and the best place to start is with a solid baseline.Tried a vanilla world and see what it is doing on your server?

I just seriously doubt that MineOS is doing anything out the box that is affecting the game every 5 minutes like clockwork. We would see more people complaining about similar occurrences probably if this was the case. Although this is subjective to those who install MineOS on top of another distro and those who use a VM provisioning layer because we are starting to change the scope significantly as there is more room for issues to arise and is not out of question.

5

To answer your point about running a vanilla server, I did infact run a vanilla server prior with less hardware allocated to it, and it ran smoother than I have ever played the game ever. Something that I think you are ignoring is that when run using the command line, same mods, it does not lag, but when run with mineOS it lags like clockwork. That’s the point I am trying to drive home here.

6

You are relaying conflicting information tbh in your opening post you say

and then now you say twice that it runs fine from the command line. Nvm i just realized that you meant you started from the webui afterwards.

7

I suppose I should have finished that sentence, I launched the server from the webUI

8

No my bad i should read properly i tend to skim read a lot and look for key points in someone posts but obviously skim reading is still well skim reading lol and i totally skimmed over the part where you said about it working no problem outside of MineOS.

9

Hi.
MineOS starts minecraft servers mosty in the same way you do manually in command line, the only difference beeing that MineOS starts them in a screen session.

Googling screen session lag and screen session priority gave me some ideas as to what is happening, ut there seems to be no real solution available:

  1. linux - Why when I run a program in a detached screen section it became very slow? - Super User
  2. nice - How to change the "niceness" (CPU priority) of a detached tmux session? - Unix & Linux Stack Exchange
10

I used to use a Dell T410 with a pair of X5670s, 64GB of 1333MHz ram and had a pair of SAS SSDs in raid 0. This is my experience on the platform and what you could do.

Honestly, drop the platform. Sell it and get something Ryzen 3rd gen or Intel 10th gen. Those processors are just not good enough to run modern modpacks. You might have fun if its a 1.12.2 pack. But if you want a decent experience with friends, upgrade. I had my packs run on bare hardware using Ubuntu Server as my OS choice and we had issues when we were exploring, setting up a vast computer storage system with auto crafting and doing Minecolonies.

Now I personally upgraded to a Ryzen 5 3500X, 32GB of 3600MHz and I was still using the SAS SSDs in raid 0 with the HBA card from the Dell. I just had to get new cables to connect the drives. But I splurged on myself and got a 1TB Gen 4 drive for my main rig so I had 2 500GB NVMe drives to use and that’s what I’m using now with “/” being one drive and “/var” being the entire second drive (You can set that up when installing Ubuntu Server).

The performance difference was literally night and day. A friend and I had tested two things. First we spawn a huge amount of tnt and light it and we loaded our playthrough of GreedyCraft. With the tnt, the old server kicked us out with network closed. It let us on after 3:24. With the new server, it didn’t kick us out and we saw it almost real time with it destroying the blocks.