Based on the error, I’d don’t think that the issue is neither your system, nor the webui, but instead the actual build–or the Java flags. You should try a newer build if possible and you should try fewer Java options.
XMX and XMS differences
I really encourage you to never set XMS lower than XMX. Or at least never so extremely far.
XMS is the starting memory, which is far less than what Paper ultimately requests. If I’m reading your logs correctly–and that these logs came from a server started within 55 seconds and crashed–then one particular (though not fully explaining issue) is how many garbage collections you’re forcing with a small XMS.
Garbage collections are expensive, cpu-wise, I/O wise and memory-constrained.
GC Heap History (10 events):
Event: 41.715 GC heap before
{Heap before GC invocations=43 (full 2):
PSYoungGen total 741376K, used 226783K [0x0000000791000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 719872K, 28% used [0x0000000791000000,0x000000079d9f7d38,0x00000007bcf00000)
from space 21504K, 92% used [0x00000007beb00000,0x00000007bfe80000,0x00000007c0000000)
to space 25088K, 0% used [0x00000007bcf00000,0x00000007bcf00000,0x00000007be780000)
ParOldGen total 276992K, used 116806K [0x0000000733000000, 0x0000000743e80000, 0x0000000791000000)
object space 276992K, 42% used [0x0000000733000000,0x000000073a211908,0x0000000743e80000)
Metaspace used 54187K, capacity 58969K, committed 59028K, reserved 1099776K
class space used 7708K, capacity 8917K, committed 8920K, reserved 1048576K
Event: 41.916 GC heap after
Heap after GC invocations=43 (full 2):
}
Event: 41.916 GC heap before
{Heap before GC invocations=44 (full 3):
Event: 43.560 GC heap after
...
Event: 50.260 GC heap before
Event: 50.260 GC heap before
...
Event: 50.545 GC heap after
...
Event: 51.930 GC heap before
...
Event: 51.996 GC heap after
...
Event: 54.873 GC heap before
...
Event: 54.944 GC heap after
In this heap history you can see that there was a collection at 41.715, 41.916, 43.560, 50.260, 51.930, 51.996, 54.873, 54.944…
Then crash.
Basically, in the timeframe of like 60 seconds, you’re practically doing nothing but garbage collecting, which I’m sure you can guess is a bad sign. Remember, GC goes like this:
1: heap full (or one of the sub-allocations, younggen/oldgen)
2: scan memory to see what is old, and being often reused
3: decide to promote or not promote
4: detect memory which is old and not being reused
5: release lowly-reused memory, compact memory (compacting not relevant in G1GC)
6: find contiguous memory to allocate to heap
Java tweaks are bad
That said, I vehemently disagree with the tweaks here, and quite frankly, I’d blame it for all this unexpected java behavior.
java -Xms6G -Xmx6G -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions -XX:MaxGCPauseMillis=100 -XX:+DisableExplicitGC -XX:TargetSurvivorRatio=90 -XX:G1NewSizePercent=50 -XX:G1MaxNewSizePercent=80 -XX:G1MixedGCLiveThresholdPercent=35 -XX:+AlwaysPreTouch -XX:+ParallelRefProcEnabled -Dusing.aikars.flags=mcflags.emc.gs -jar paperclip.jar
It is, in my opinion, one of the most self-destructive tweaks I’ve ever seen:
Let’s examine this more closely.
If the application being fine-tuned has a relatively consistent object allocation rate, it is acceptable to raise the target survivor occupancy to something as high as -XX:TargetSurvivorRatio=80 or -XX:TargetSurvivorRatio=90. The advantage of being able to do so helps reduce the amount of survivor space needed to age objects. The challenge with setting -XX:TargetSurvivorRatio= higher is the HotSpot VM not being able to better adapt object aging in the presence of spikes in object allocation rates, which can lead to tenuring objects sooner than you would like.
Spikes would be exactly what you’re experiencing at any initial load time.
Here’s what it means to tenure early:
41.960 ParOldGen total 276992K, used 174838K
43.560 ParOldGen total 453120K, used 180836K
In the course of less than two seconds, the old gen was raised in size from 276MB to 453MB all to accommodate … 180MB in actual memory. Just like the young generation, the old generation is getting sized up hugely but it’s mostly not even getting used!
Old gen is good for reused memory, but we don’t know that that is applicable because we’re seeing things get tenured into oldgen sooner than we would like.
So this tweak has set Survivor ratio at 90%, which means that the young generation survive the memory reaping (deciding if it will live or die) more often. More survival successes means quicker promotion to the old gen (which is exactly what you’re seeing here). If things are unduly promoted to the oldgen, they don’t get cleared out/checked for relevance as often–and oldgen GCs are far more expensive, even!
So let’s follow your error output. Look closely at the eden space, where new fledgling memory allocations are grown: 28%, but the survivor space is already 92% full. This means that Java is running out of survivor space, because it’s saving more survivors, but the survivor space is tiny compared to what is available. Here’s an ascii visual of EDEN vs. survivor areas.
OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOFFTT
Each O is available memory to newly created objects. Each F and T indicate where memory designated as “important enough to keep” is kept (“from” and “to”). Exacerbated by a low XMS which forces judgment on survivors, F and T will fill up at an accelerated pace becaue of TargetSurvivorRatio.
So you have huge numbers in the young gen (741376K or 741MB “eden”) but basically you’re only able to use about 21504K (21MB “from”) before it is worried that it is running low on space. So rather than run out of survivor space, it promotes them to Oldgen space.
Notice how the From space continuously fills up, forcing garbage collections every few seconds? This is made worse by the target MaxGCPauseMillis=100 which says “try to keep your GC’s at 100ms please”. So now you have GC collections sometimes every 200ms, sometimes every other second. In short, ridiculously more often than there should be.
GC Heap History (10 events):
Event: 41.715 GC heap before
{Heap before GC invocations=43 (full 2):
PSYoungGen total 741376K, used 226783K [0x0000000791000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 719872K, 28% used [0x0000000791000000,0x000000079d9f7d38,0x00000007bcf00000)
from space 21504K, 92% used [0x00000007beb00000,0x00000007bfe80000,0x00000007c0000000)
to space 25088K, 0% used [0x00000007bcf00000,0x00000007bcf00000,0x00000007be780000)
ParOldGen total 276992K, used 116806K [0x0000000733000000, 0x0000000743e80000, 0x0000000791000000)
Event: 41.916 GC heap after
Heap after GC invocations=43 (full 2):
PSYoungGen total 744960K, used 25064K [0x0000000791000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 719872K, 0% used [0x0000000791000000,0x0000000791000000,0x00000007bcf00000)
from space 25088K, 99% used [0x00000007bcf00000,0x00000007be77a160,0x00000007be780000)
to space 25088K, 0% used [0x00000007be780000,0x00000007be780000,0x00000007c0000000)
ParOldGen total 276992K, used 174838K [0x0000000733000000, 0x0000000743e80000,
Event: 41.916 GC heap before
{Heap before GC invocations=44 (full 3):
PSYoungGen total 744960K, used 25064K [0x0000000791000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 719872K, 0% used [0x0000000791000000,0x0000000791000000,0x00000007bcf00000)
from space 25088K, 99% used [0x00000007bcf00000,0x00000007be77a160,0x00000007be780000)
to space 25088K, 0% used [0x00000007be780000,0x00000007be780000,0x00000007c0000000)
ParOldGen total 276992K, used 174838K [0x0000000733000000, 0x0000000743e80000, 0x0000000791000000)
Event: 43.560 GC heap after
Heap after GC invocations=44 (full 3):
PSYoungGen total 744960K, used 0K [0x0000000791000000, 0x00000007c0000000, 0x00000007c0000000)
eden space 719872K, 0% used [0x0000000791000000,0x0000000791000000,0x00000007bcf00000)
from space 25088K, 0% used [0x00000007bcf00000,0x00000007bcf00000,0x00000007be780000)
to space 25088K, 0% used [0x00000007be780000,0x00000007be780000,0x00000007c0000000)
ParOldGen total 453120K, used 180836K [0x0000000733000000, 0x000000074ea80000, 0x0000000791000000)
If the GC spends enough time doing constant GCs and pausing and not doing real work, other threads and operations could be blocked. Important operations that are blocked may be too important for paper to continue working without, and you can get a crash.
I have always long held that unless people actually use Java profilers, you should stay away from heavily tweaking the startup flags, especially when using things like UnlockExperimentalVMOptions, which by its name already seems like it isn’t production-quality.
I think it’s very possible for you to use your existing hardware for a good experience. I think that you’re overdoing it with the tweaks and you should keep to simple and tweak if you have evidence it works–I don’t think that the linked tweaks you read have any substance to them that would translate from their commercial-grade Minecraft offering to your older, aging computer.
Arguments against his justifications
Alright, here’s my ranting. Don’t worry if this doesn’t make much sense, this is as much for my edification as anything.
It appears the site you linked also says this at the end:
If you are running with 10GB or less memory for MC, you should not adjust these parameters.
I can’t tell whether that means you shouldn’t change parameters from what they offer, of if they mean you shouldn’t apply them at all. At any rate, with your hardware, it seems inappropriate to use these Java flags.
TargetSurvivorRatio : I’m sure your all use to seeing this one suggested. Good news! It’s actually a good flag to use. This setting controls how much of the Survivor space is ABLE to be used before promotion. If survivor gets too full, stuff starts promoting to Old Gen.
Here, I believe he is getting a favorable result, but not for the reason he is thinking. The survivor space can be used beyond the default 50%, but each time a new allocation occurs that pushes the space > 50%, a small collection occurs, moving memory from “FROM” to “TO”. This induces an additional check that makes it possible for stuff to get promoted, but it is not about fullness, it is about surpassing the MaxTenuringThreshold=31.
G1NewSize Percent: […] With these settings, we tell G1 to not use its default 5% for new gen, and instead give it 50% at least! Minecraft has an extremely high a memory allocation rate, ranging to at least 800 Megabytes a second on a 30 player server! And this is mostly short lived objects (Block Position)
#1 this contradicts what I linked above from an actual Java whitepaper: not being able to better adapt object aging in the presence of spikes in object allocation rates . If this guy is claiming Minecraft ha sa high memory allocation rate — AND OF SHORT LIVED OBJECTS – then high survivor ratios are bad.
#2 SHORT LIVED OBJECTS. SHORT LIVED OBJECTS! They belong in eden space where it is inexpensive to garbage collect them… they should not be promoted to Old Gen where it is expensive. They will soon and eventually be collected because they are short lived objects, i.e., things that belong in young generation.
Now, this means MC REALLY needs more focus on New Generation to be able to even support this allocation rate. If your new gen is too small, you will be running new gen collections 1-2+ times per second, which is really bad.You will have so many pauses that TPS has risk of suffering, and the server will not be able to keep up with the cost of GC’s.
The pauses are because the survivor rate is bad and he should instead be modifying the NewRatio, and other ratios which help make sure there is good use of both the survival spaces and the eden spaces.
Of course, he’s also pushing the G1 Garbage collector, where these ratios are more malleable because they aren’t contiguous, but unused allocations are bad, no matter which collector.
G1MixedGCLiveThresholdPercent : Controls when to include Mixed GC’s in the Young GC collection, keeping Old Gen tidy without doing a normal Old Gen GC collection. When your memory is less than this percent, old gen won’t even be included in ‘mixed’ collections. Mixed are not as heavy as a full old collection, so having small incremental cleanups of old keeps memory usage light.
WHAT!? The whole point he was trying to make about tenuring objects earlier from the survivor space (on short lived objects, no less) makes no sense if he lowers the threshold in which old gen should be counted in for expensive Oldgen GC!
The default is 65 and he lowered it to 35, which means at 35% capacity, start checking ALL stuff in the old gen to see if it should be collected. If he’s going to fast-promote things to the old gen, he shouldn’t have all that stuff doubly-checked when only 35% capacity! This is completely backwards!
Memory usage should not be kept “light”. Memory usage should be exactly what is asked by the calling program, in this case, PaperSpigot. Incremental cleanups at 35% of old gen is more expensive than incremental cleanups of new gen, so why not let them live in new gen longer and die there, so they never need to be copied to old gen in the first place?
AlwaysPreTouch : AlwaysPreTouch gets the memory setup and reserved at process start ensuring it is contiguous, improving the efficiency of it more. This improves the operating systems memory access speed.
Sounds solid. Definitely you should do this only when XMX and XMS are near, though, which he does recommend.
MaxGCPauseMillis =100: This setting controls how much memory is used in between the Minimum and Maximum ranges specified for your New Generation. This is a “goal” for how long you want your server to pause for collections. 100 is equal to 2 ticks, aiming for an at most loss of 2 ticks. This will result in a short TPS drop, however Spigot and Paper both can make up for this drop instantly, meaning it will have no meaningful impact to your TPS. 100ms is lower than players can recognize.
This is a tweak that “sounds” like it makes sense. Why not have garbage collection limited in time to an amount a user won’t notice, right?
Well, the thing is when GC needs to happen, you want it to finish, too. And if it doesn’t finish–like what you are experiencing with this multiple collection–then all it does is add overhead and require requeueing of the GC and ends up costing you more than if you left it alone.
I’m really annoyed by this guys recommendations, even if all people using it had 10GB of RAM for Minecraft.