Servers automatically restarting and nonresponsive UI


#23

if MineOS is stopped or crashes could or does it kill running minecraft servers?

Edit: Now that you have access have you seen first hand have you see the issues I’ve described?


#24

I haven’t yet been able to find out the root cause of the problem, but I did seem to deduce that it is definitely Forge-only. Running the vanilla server runs without issue for infinity, but once the Forge server is selected, it seems to crash without any reason noted in the Forge logs.

At first I thought it was related to recent commits involving either query or save-all, but turning back to a much later commit has shown that the issue persists. Then I thought it was because your Gliby voice mod ran on UDP on 25565, which is the exact same port you’re running query (as in query-enable) on. Turns out this isn’t much of a concern either.

MineOS stopping or crashing should not affect any running servers; servers are run in their own process, not as children, but as siblings–meaning MineOS can be killed, stopped, and restarted without any impact on Minecraft.

However, this is a unique situation where it appears Forge is crashing, but the way it’s crashing (or why) is perhaps locking a file that MineOS cares about…and suddenly can’t access and then dies. The second half of that is on my end: MineOS shouldn’t die under any circumstances of file locking, editing, or deletion, but alas it is.

The forge issue crashing is probably a bigger deal though. For example, I’ve started your ServerLive instance numerous times where it consistently making MineOS the webui fail, but through repeated restarts, the server remains up.

If you run the Forge universal jar and type in help, you’ll notice an internal Java error in the console. You might look into “helpfixer” to see if it is simply masking a larger problem–or if it’s a red herring. At any rate, 100% of crashes were on the Forge jar, so it’s something that requires more examination.


#25

I feel bad I wish I could’ve saved you some time if I would’ve thought to the test mc instances independently.

You’ve given me a direction to pursue.

  1. Pairing down mods and seeing if any of them affect the overall issue.
  2. Trying to forge without any mods.
  3. Updating forage
  4. Uninstalling and reinstalling Java
  5. Checking certificates are valid, which can affect Java/forge

I’m guessing you have not have been able to reproduce this in any of your testing environments. So I should leave everything intact and spin up a third MC instance for testing purposes. Outside of that how should I proceed? As you’ve previously said that this is kind of a unique constellation of issues to cause problems with MineOS side of things. I don’t want to unintentionally deny you the ability to improve upon MineSO. I wouldn’t want to destroy the testing environment. I could make an image of the server now just in case.

Did you by chance look into users and groups showing up on the web UI?


#26

Just to be crystal clear you’re saying ‘ServerLive’ might run continuously for period of time without stopping but the web UI is crashing and restarting? So you observe that the server stayed stable through multiple crashes of the MineOS web UI? Eventually the server failed but it’s not a one-to-one relationship between the server and the web UI.


#28
  1. A copy of ServerLive runs fine on Windows
  2. "[Server console handler/ERROR]
    [net.minecraft.server.dedicated.DedicatedServer]: Exception handling
    console input java.io.IOException: The handle is invalid"is from the server not getting enough RAM. It’s been commonly reported. Although finding a solution by adding more RAM took a while to find. We can safely say this is not the symptom or the cause of our problems.

Each # represents the forge server being started through the web UI the addition of a vanilla server running.

  1. Everything runs as expected
  2. Everything runs as expected with Thaumcraft-1.8-5.0.2.jar added.
  3. Everything is the same from 2 except I deleted the logs and worlds folders. When starting the server instance the web UI crashes and the user must re-login. The folders log and worlds are not regenerated.
  4. After relogging in from the crash from 3 but not changing a thing everything works expected when starting the server. Both server seem to be running fine over a large span of time without stopping or without causing the web UI to crash.

Edit 1: I was able to replicate this using a vanilla server TestingGrounds. With TestingGrounds stopped. I deleted the logs and worlds folders. When starting the server instance the web UI crashes and the user must re-login. The folders log and worlds are not regenerated. The same results as described above with 3. No other server instances need to be running.

Edit 2: After some more testing deleting latest.log (maybe any *.log in logs folder?) seems to crash the web UI when starting server.


#29

Just you know my public IP address changed. In the text file I sent you there is a URL to a spreadsheet that contains the latest IP address.


#30

Is there anything else I can do to help you?

Have you been able to replicate the issue outside of my server environment?


#31

OK, FIRST ISSUE: webui crashes the minecraft instances (resolved)

Turns out that the reason I was unable to reproduce the issue consistently on a different server is because on all other servers, I wasn’t using systemd! Looks like systemd has a default setting that when the main forked process dies, it seeks out and kills all it’s child processes so as not to orphan them. This isn’t the default behavior for any of the other methods.

I was able to update the mineos.service file so that no screen/java instances close when the webui crashes. No matter what happens to the webui now, the instances will remain untouched.

SECOND ISSUE: forge touching/modifying files in a way that crashes npm fireworm (not resolved)

As best as I can tell, this might be a fireworm bug. It’s watching a file that is maybe(?) getting locked/edited/repopulated in a way that fireworm hates, and it’s crashing the module. Due to the asynchronous of the design of nodejs, I’m pretty sure I can’t trap this error in MineOS code, but I’d have to dig into fireworm.

So, overall this second issue is not resolved, but at the least, the crashing webui a) does not take down the server and b) has an open issue with the fireworm dev.

(to zone only).

Right now your /usr/games/minecraft is set to the branch fs_tweaks, which is due to get pulled into the main master branch soon. You’ll want to switch to master soon (but not yet, I’ll let you know) and then execute these lines:

cd /usr/games/minecraft
git checkout master
git reset --hard origin/master

Also, you need to validate your cron jobs. Right now you have a cronjob that runs most certainly not what you’re expecting.


#32

I’m glad you’re able to make headway and that this issue resolved will further MineOS’s code base.

I’m quite new to cronjob. I created that originally and wasn’t able deleted between the issues I was having with the web UI. Thank you for sparing me. That website to validate cron jobs would be helpful resource in the mineOS wiki.

As a future note most MC server admins that have heavily modded servers restart their servers every 24 or 12 hours.
0 0 * * * will execute cron jobs every 24 hours every day of the week .
To the best of my understanding that 24 hour period starts upon executing cronjob. Which means an administrator has to be up at the time that he wants to be 24 to start.


#33

Just checking in to make sure this didn’t fall under the cracks! Happy Thanksgiving!


#34

It has. I have no idea how to solve this.

The most likely way this is happening is that forge–instead of adjusting files in-place, is creating temporary files with the new, adjusted content, un-linking the original file, then renaming the new file to the old file’s name.

If that’s the case, this is a fireworm issue, which doesn’t seem any more close to being identified or fixed.


#35

What are my options out side of switching to the MineOS python? Should I try a different distribution of Linux?


#36

Different Linux won’t solve an issue with an npm module, which will be used on that distro too. Another option is helping find an npm module that doesn’t crash from temporary-file/unlink/renames or to find a MC mod that does the same functionality by editing files in-place.


#37

There are two avenues for fixing this problem one is to find an alternative to npm fireworm or addressing the forge side of the equation for writing files.

Second is there a way to keep fireworm from crawling or watching certain files or folders? Because I would like to figure out the specific files that are crashing fireworm. If I were able to find the files then I might be able to speak to those involved with forge to see if there’s any information they might be able to contribute.

Meantime I’ll see if I can find another file watcher module. I will continue to edit this portion as I find alternatives. Let me know your thoughts.
Edit: possible candidates?
chokidar A fs.watch wrapper. seems to have good documentation and was designed in mind to keep from reporting events twice.
sane seems to have good documentation, utilizes fs.watch.
watchr, last touched 10 months ago, has decent documentation, considered a replacement for fs.watch.


#38

spoiler, I’ve tried literally all the watchers you can find on npm.


#39

Worst spoiler ever, so what’s the number one issue that makes them unusable?


#40

What chokidar doesn’t do that we need, detect recreated files and folders.

What chokidar does that we do need is have the option of directory depth.

What fireworm does that nothing else does is detect changes to files when the files are replaced by temporary file/unlink/renames (where the other ones work by inodes)

It’s not about the specific files that get changed, its how they get changed. Forge, unlike all other servers, doesn’t edit files “in-place”. In-place file change detection works on all modules, Forge’s method is detectable only by fireworm.


#41

Great that means we can’t simply request functionality an existing project that’s well-maintained due to the design.

You might strike up a conversation with the author through ‘airportyh at gmail dot com’ about the issue if you feel inclined. Some people that utilize github don’t check it very often and turned off/blocked email notifications.


Parts of the Web UI do not respond
#42

All right well I think I’ll have to switch back to MineOS python until this is resolved as forge is the modern API for minecraft. Since you can reproduce the issue I’m going to reimage the server.


#43

@hexparrot
Assuming this issue it is not resolved could you create open Github ticket summarizing the issue. That way its progress can be tracked and referenced on Github.