I feel bad I wish I could’ve saved you some time if I would’ve thought to the test mc instances independently.
You’ve given me a direction to pursue.
Pairing down mods and seeing if any of them affect the overall issue.
Trying to forge without any mods.
Updating forage
Uninstalling and reinstalling Java
Checking certificates are valid, which can affect Java/forge
I’m guessing you have not have been able to reproduce this in any of your testing environments. So I should leave everything intact and spin up a third MC instance for testing purposes. Outside of that how should I proceed? As you’ve previously said that this is kind of a unique constellation of issues to cause problems with MineOS side of things. I don’t want to unintentionally deny you the ability to improve upon MineSO. I wouldn’t want to destroy the testing environment. I could make an image of the server now just in case.
Did you by chance look into users and groups showing up on the web UI?
Just to be crystal clear you’re saying ‘ServerLive’ might run continuously for period of time without stopping but the web UI is crashing and restarting? So you observe that the server stayed stable through multiple crashes of the MineOS web UI? Eventually the server failed but it’s not a one-to-one relationship between the server and the web UI.
"[Server console handler/ERROR]
[net.minecraft.server.dedicated.DedicatedServer]: Exception handling
console input java.io.IOException: The handle is invalid"is from the server not getting enough RAM. It’s been commonly reported. Although finding a solution by adding more RAM took a while to find. We can safely say this is not the symptom or the cause of our problems.
Each # represents the forge server being started through the web UI the addition of a vanilla server running.
Everything runs as expected with Thaumcraft-1.8-5.0.2.jar added.
Everything is the same from 2 except I deleted the logs and worlds folders. When starting the server instance the web UI crashes and the user must re-login. The folders log and worlds are not regenerated.
After relogging in from the crash from 3 but not changing a thing everything works expected when starting the server. Both server seem to be running fine over a large span of time without stopping or without causing the web UI to crash.
Edit 1: I was able to replicate this using a vanilla server TestingGrounds. With TestingGrounds stopped. I deleted the logs and worlds folders. When starting the server instance the web UI crashes and the user must re-login. The folders log and worlds are not regenerated. The same results as described above with 3. No other server instances need to be running.
Edit 2: After some more testing deleting latest.log (maybe any *.log in logs folder?) seems to crash the web UI when starting server.
OK, FIRST ISSUE: webui crashes the minecraft instances (resolved)
Turns out that the reason I was unable to reproduce the issue consistently on a different server is because on all other servers, I wasn’t using systemd! Looks like systemd has a default setting that when the main forked process dies, it seeks out and kills all it’s child processes so as not to orphan them. This isn’t the default behavior for any of the other methods.
I was able to update the mineos.service file so that no screen/java instances close when the webui crashes. No matter what happens to the webui now, the instances will remain untouched.
SECOND ISSUE: forge touching/modifying files in a way that crashes npm fireworm (not resolved)
As best as I can tell, this might be a fireworm bug. It’s watching a file that is maybe(?) getting locked/edited/repopulated in a way that fireworm hates, and it’s crashing the module. Due to the asynchronous of the design of nodejs, I’m pretty sure I can’t trap this error in MineOS code, but I’d have to dig into fireworm.
So, overall this second issue is not resolved, but at the least, the crashing webui a) does not take down the server and b) has an open issue with the fireworm dev.
(to zone only).
Right now your /usr/games/minecraft is set to the branch fs_tweaks, which is due to get pulled into the main master branch soon. You’ll want to switch to master soon (but not yet, I’ll let you know) and then execute these lines:
cd /usr/games/minecraft
git checkout master
git reset --hard origin/master
Also, you need to validate your cron jobs. Right now you have a cronjob that runs most certainly not what you’re expecting.
I’m glad you’re able to make headway and that this issue resolved will further MineOS’s code base.
I’m quite new to cronjob. I created that originally and wasn’t able deleted between the issues I was having with the web UI. Thank you for sparing me. That website to validate cron jobs would be helpful resource in the mineOS wiki.
As a future note most MC server admins that have heavily modded servers restart their servers every 24 or 12 hours. 0 0 * * * will execute cron jobs every 24 hours every day of the week .
To the best of my understanding that 24 hour period starts upon executing cronjob. Which means an administrator has to be up at the time that he wants to be 24 to start.
The most likely way this is happening is that forge–instead of adjusting files in-place, is creating temporary files with the new, adjusted content, un-linking the original file, then renaming the new file to the old file’s name.
If that’s the case, this is a fireworm issue, which doesn’t seem any more close to being identified or fixed.
Different Linux won’t solve an issue with an npm module, which will be used on that distro too. Another option is helping find an npm module that doesn’t crash from temporary-file/unlink/renames or to find a MC mod that does the same functionality by editing files in-place.
There are two avenues for fixing this problem one is to find an alternative to npm fireworm or addressing the forge side of the equation for writing files.
Second is there a way to keep fireworm from crawling or watching certain files or folders? Because I would like to figure out the specific files that are crashing fireworm. If I were able to find the files then I might be able to speak to those involved with forge to see if there’s any information they might be able to contribute.
Meantime I’ll see if I can find another file watcher module. I will continue to edit this portion as I find alternatives. Let me know your thoughts.
Edit: possible candidates? chokidar A fs.watch wrapper. seems to have good documentation and was designed in mind to keep from reporting events twice. sane seems to have good documentation, utilizes fs.watch. watchr, last touched 10 months ago, has decent documentation, considered a replacement for fs.watch.
What chokidar doesn’t do that we need, detect recreated files and folders.
What chokidar does that we do need is have the option of directory depth.
What fireworm does that nothing else does is detect changes to files when the files are replaced by temporary file/unlink/renames (where the other ones work by inodes)
It’s not about the specific files that get changed, its how they get changed. Forge, unlike all other servers, doesn’t edit files “in-place”. In-place file change detection works on all modules, Forge’s method is detectable only by fireworm.
Great that means we can’t simply request functionality an existing project that’s well-maintained due to the design.
You might strike up a conversation with the author through ‘airportyh at gmail dot com’ about the issue if you feel inclined. Some people that utilize github don’t check it very often and turned off/blocked email notifications.
All right well I think I’ll have to switch back to MineOS python until this is resolved as forge is the modern API for minecraft. Since you can reproduce the issue I’m going to reimage the server.
@hexparrot
Assuming this issue it is not resolved could you create open Github ticket summarizing the issue. That way its progress can be tracked and referenced on Github.
one of the issues i had now resolved, was identified by iMelsom as the ‘fireworm’ issue documented here:
@zone
clearly i can only say the fireworm issue was resolved for me and may not apply to your good self.
your post has shed some light on what was happening to me and related to what i can and can not do going forward. i never saw this until now, so thanks for your inadvertent help to me.
@Sir hexparrot
the best recap my limited command of things that may or may not apply to others that did end up applying to me had to do with permissions my SFTP client was changing as it handled files and most importantly had to do with how fireworm changed filenames as it handled files for example a file named yada_Yada could be handled and written back as Yadayada (an exaggerated example) simply because fireworm was unable to have any command of syntax by default.
AND i do think there was a commit upgrade also around the same time that put to bed issues as well.
to sum up i had to:
be very carefull how SFTP was used.
important, use very simple server names when creating servers. no underlines, no caps, no spaces and not too many characters.
keep MineOS commit updated.
my proven results have been multiple instances of very heavily modified forge servers (FTBInfinity for example) all running together so stable for hundreds of hours, weeks at a time, no more re-booting servers every 12-24 hours like before.
