Having no problems with the 7z file you provided. All servers appear in the webui and operate as expected; the ServerLive starts upon boot. I created a mineos_admins group and have been able to reassign servers to have mineos_admins group ownership.
One thing to note, however, is that 7z actually deletes permissions. Upon extraction on my Ubuntu box, the servers the directories looked like this:
drwx------ 5 will will 4096 Nov 9 14:44 archive
drwx------ 5 will will 4096 Nov 9 14:44 backup
drwx------ 2 will will 4096 Nov 9 12:27 import
drwx------ 4 will will 4096 Nov 9 14:44 profiles
drwx------ 5 will will 4096 Nov 9 14:44 servers
This means that the permissions and ownership information was lost when it showed up on my machine. It was auto-chowned to will because that was my user, but as you can see its chmod 700, which I had to change manually–and recursively.
That also means I have no idea whether the problem you are having is caused by inconsistent permissions, such as if some files inside servers/ServerLive/* were mis-owned. I blanket chmodded 664 to everything, and also to the user will, which again could possibly mask an actual issue you have if your files are not perm’ed right.
will@willonubuntu:/var/games/minecraft/servers$ groups will
will : will adm cdrom sudo dip plugdev lpadmin sambashare mineos_admins
Have you looked inside the contents of each of these log files to see if they are showing any errors or issues?
I’m actually thinking the reverse causality. Perhaps Minecraft is crashing at whatever frequency of your logs…and MineOS isn’t able to handle polling the server or other actions (e.g., commit world via saveall) because the server is down/crashed/restarting.
I’ve opened up most of logs and this is pretty representative. Typically when minecraft crashes a separate log is generated. I don’t believe this is the case with the situation as I’m not seen many crash logs generated nor did you see it during your testing. I’ve had some crashes but I should be having a crash every time a log is generated and it doesn’t explain why the vanilla server’s being stopped.
Random sample log 1
[16:28:16] [Server thread/INFO]: Starting minecraft server version 1.8
[16:28:20] [Server thread/INFO]: Loading properties
[16:28:20] [Server thread/INFO]: Default game type: SURVIVAL
[16:28:20] [Server thread/INFO]: Generating keypair
[16:28:20] [Server thread/INFO]: Starting Minecraft server on 0.0.0.0:25565
[16:28:21] [Server thread/INFO]: Preparing level "world"
[16:28:21] [Server thread/INFO]: Preparing start region for level 0
[16:28:22] [Server thread/INFO]: Done (0.749s)! For help, type "help" or "?"
[16:28:22] [Server thread/INFO]: Starting GS4 status listener
[16:28:22] [Query Listener #1/INFO]: Query running on 0.0.0.0:25565
[16:29:11] [Server Shutdown Thread/INFO]: Stopping server
Random sample log 2
[16:15:13] [Server thread/INFO]: Starting minecraft server version 1.8
[16:15:17] [Server thread/INFO]: Loading properties
[16:15:17] [Server thread/INFO]: Default game type: SURVIVAL
[16:15:17] [Server thread/INFO]: Generating keypair
[16:15:17] [Server thread/INFO]: Starting Minecraft server on 0.0.0.0:25565
[16:15:19] [Server thread/INFO]: Preparing level "world"
[16:15:19] [Server thread/INFO]: Preparing start region for level 0
[16:15:19] [Server thread/INFO]: Done (0.845s)! For help, type "help" or "?"
[16:15:19] [Server thread/INFO]: Starting GS4 status listener
[16:15:19] [Query Listener #1/INFO]: Query running on 0.0.0.0:25565
[16:25:54] [Server Shutdown Thread/INFO]: Stopping server
[16:25:54] [Server Shutdown Thread/INFO]: Saving players
[16:25:54] [Server Shutdown Thread/INFO]: Saving worlds
As for the reboot intervals the op has a picture that gives a pretty good description of intervals. Which is approximately 60 seconds.
As for Misconfigured permissions, I would think that would be unlikely. I should see when I login to SFTP any discrepancies and everything is transferred as the MC user. But there has to be something that’s going wrong in its most likely user error I just can’t imagine what.
This also affects a vanilla server. Everything at that point is handled by the web interface.
Well I think I found the source of our problem as seen with the vanilla server.
[14:30:25] [Server thread/INFO]: Starting minecraft server version 1.8.8
[14:30:25] [Server thread/WARN]: To start the server with more ram, launch it as "java -Xmx1024M -Xms1024M -jar minecraft_server.jar"
[14:30:25] [Server thread/INFO]: Loading properties
[14:30:25] [Server thread/INFO]: Default game type: SURVIVAL
[14:30:25] [Server thread/INFO]: Generating keypair
[14:30:25] [Server thread/INFO]: Starting Minecraft server on 0.0.0.0:25564
[14:30:26] [Server thread/INFO]: Using epoll channel type
[14:30:26] [Server thread/INFO]: Preparing level "world"
[14:30:26] [Server thread/INFO]: Preparing start region for level 0
[14:30:26] [Server thread/INFO]: Done (0.809s)! For help, type "help" or "?"
[14:41:03] [Server console handler/ERROR]: Exception handling console input
java.io.IOException: Input/output error
at java.io.FileInputStream.readBytes(Native Method) ~[?:1.8.0_66-internal]
at java.io.FileInputStream.read(FileInputStream.java:255) ~[?:1.8.0_66-internal]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:1.8.0_66-internal]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_66-internal]
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_66-internal]
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_66-internal]
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_66-internal]
at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_66-internal]
at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:1.8.0_66-internal]
at java.io.BufferedReader.readLine(BufferedReader.java:324) ~[?:1.8.0_66-internal]
at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[?:1.8.0_66-internal]
at ko$2.run(SourceFile:81) [minecraft_server.1.8.8.jar:?]
This might be the real problem and it could be massed by Forge. This does not generate a crash report.
Try updating to the latest commit c576847c5841b706d1c7054cc42f55025bd0a2af and see if this issue persists.
This was a suspicion I had–because I introduced a bug recently (that I didn’t think would have this effect)…
I added the “save-all” frequency to the webui, which sent the command saveall to the console. Of course, the correct command is save-all, so it was a typo. Despite that, the game … shouldn’t crash because it receives saveall–it would be literally like if you typed it into the console and then everything died (it should handle it more gracefully).
however, if you went back to the commit I told you earlier:
this previous commit I had you roll back to–didn’t have this functionality at all (and thus didn’t have the bug). Rolling back to the previous commit should have solved this issue (if going to the most recent commit, in fact, fixes anything).
But at least we have something to work with. Update to the most recent commit and try setting the value to (which disables this) and see if your servers stay up.
I updated to the latest commit. The results are both of the servers end up stopping. The modded server doesn’t reboot. I made sure the values were blank for Minutes between committing world to disk via 'save-all' (empty to disable). Although in the box it still says in a light gray color 60.
The logs are identical for Modded when compared vanilla.
[17:38:34] [Server thread/INFO]: Starting minecraft server version 1.8
[17:38:38] [Server thread/INFO]: Loading properties
[17:38:38] [Server thread/INFO]: Default game type: SURVIVAL
[17:38:38] [Server thread/INFO]: Generating keypair
[17:38:38] [Server thread/INFO]: Starting Minecraft server on 0.0.0.0:25565
[17:38:40] [Server thread/INFO]: Preparing level “world”
[17:38:40] [Server thread/INFO]: Preparing start region for level 0
[17:38:41] [Server thread/INFO]: Done (1.145s)! For help, type “help” or “?”
[17:38:41] [Server thread/INFO]: Starting GS4 status listener
[17:38:41] [Query Listener #1/INFO]: Query running on 0.0.0.0:25565
[17:39:30] [Server console handler/ERROR]: Exception handling console input
java.io.IOException: Input/output error
at java.io.FileInputStream.readBytes(Native Method) ~[?:1.8.0_66-internal]
at java.io.FileInputStream.read(FileInputStream.java:255) ~[?:1.8.0_66-internal]
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:1.8.0_66-internal]
at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_66-internal]
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) ~[?:1.8.0_66-internal]
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ~[?:1.8.0_66-internal]
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) ~[?:1.8.0_66-internal]
at java.io.InputStreamReader.read(InputStreamReader.java:184) ~[?:1.8.0_66-internal]
at java.io.BufferedReader.fill(BufferedReader.java:161) ~[?:1.8.0_66-internal]
at java.io.BufferedReader.readLine(BufferedReader.java:324) ~[?:1.8.0_66-internal]
at java.io.BufferedReader.readLine(BufferedReader.java:389) ~[?:1.8.0_66-internal]
at net.minecraft.server.dedicated.DedicatedServer$2.run(DedicatedServer.java:97) [pq.class:?]
[17:39:30] [Server Shutdown Thread/INFO]: Stopping server
[17:39:30] [Server Shutdown Thread/INFO]: Saving players
[17:39:30] [Server Shutdown Thread/INFO]: Saving worlds
[17:39:30] [Server Shutdown Thread/INFO]: Saving chunks for level ‘world’/Overworld
If the commands you gave were complete I believe I did rollback the server as requested.
Edit: it seems all this is revolving around the act of writing to the disk file’s which causes the server to shut down. Maybe it’s not the command itself but the act of writing to disk. So that leads us back to permissions but also the possibility of hhd integrity? I’ll try running badblocks -nvs /dev/sdb1
Can you check to make sure that all the files in the server are properly owned by the user it should be? (presumably ben:mineos_admins)?
Especially the log files–if any are owned by a different user (such as root), these problems would crop up regularly. However, files don’t typically get chowned to root either, so, it’s worth looking at.
Actually its mc:mc
Although I would like to when this is figured out ben:mineos_admins
As of right now neither ben or mineos_admins show up in the web UI.
Connect to your server with your mc user and connect specifically to your console with screen -r. Try typing in commands like saveall or save-all and see if you can replicate the crash.
The only substantial change in code I’ve made to MineOS recently that would be related to the input error you’re getting relates to this save-all, so also have you verified that the crashes continue even with save-all disabled (emptied out? – the gray placeholder text does not have any effect).
save-all works as expected on both servers without causing them to stop.
I noticed something very interesting! While testing using screen -r/saveall I had the shell and browser open side by side. As I’ve previously mentioned I’ve had problems with the web UI becoming unresponsive forcing me to re-login.
During this round of testing I was refreshing the browser every few seconds. At the exact time shell reported [screen is terminating](which means the server shutdown) the web UI forced me to re-login. It’s consistent every time when relogging in both are shut down. So being forced to re-log in to the web UI in correlates with the servers
terminating.
That would explain the error message and why the mc servers are not being shut down gracefully.
I haven’t yet been able to find out the root cause of the problem, but I did seem to deduce that it is definitely Forge-only. Running the vanilla server runs without issue for infinity, but once the Forge server is selected, it seems to crash without any reason noted in the Forge logs.
At first I thought it was related to recent commits involving either query or save-all, but turning back to a much later commit has shown that the issue persists. Then I thought it was because your Gliby voice mod ran on UDP on 25565, which is the exact same port you’re running query (as in query-enable) on. Turns out this isn’t much of a concern either.
MineOS stopping or crashing should not affect any running servers; servers are run in their own process, not as children, but as siblings–meaning MineOS can be killed, stopped, and restarted without any impact on Minecraft.
However, this is a unique situation where it appears Forge is crashing, but the way it’s crashing (or why) is perhaps locking a file that MineOS cares about…and suddenly can’t access and then dies. The second half of that is on my end: MineOS shouldn’t die under any circumstances of file locking, editing, or deletion, but alas it is.
The forge issue crashing is probably a bigger deal though. For example, I’ve started your ServerLive instance numerous times where it consistently making MineOS the webui fail, but through repeated restarts, the server remains up.
If you run the Forge universal jar and type in help, you’ll notice an internal Java error in the console. You might look into “helpfixer” to see if it is simply masking a larger problem–or if it’s a red herring. At any rate, 100% of crashes were on the Forge jar, so it’s something that requires more examination.
I feel bad I wish I could’ve saved you some time if I would’ve thought to the test mc instances independently.
You’ve given me a direction to pursue.
Pairing down mods and seeing if any of them affect the overall issue.
Trying to forge without any mods.
Updating forage
Uninstalling and reinstalling Java
Checking certificates are valid, which can affect Java/forge
I’m guessing you have not have been able to reproduce this in any of your testing environments. So I should leave everything intact and spin up a third MC instance for testing purposes. Outside of that how should I proceed? As you’ve previously said that this is kind of a unique constellation of issues to cause problems with MineOS side of things. I don’t want to unintentionally deny you the ability to improve upon MineSO. I wouldn’t want to destroy the testing environment. I could make an image of the server now just in case.
Did you by chance look into users and groups showing up on the web UI?
Just to be crystal clear you’re saying ‘ServerLive’ might run continuously for period of time without stopping but the web UI is crashing and restarting? So you observe that the server stayed stable through multiple crashes of the MineOS web UI? Eventually the server failed but it’s not a one-to-one relationship between the server and the web UI.
"[Server console handler/ERROR]
[net.minecraft.server.dedicated.DedicatedServer]: Exception handling
console input java.io.IOException: The handle is invalid"is from the server not getting enough RAM. It’s been commonly reported. Although finding a solution by adding more RAM took a while to find. We can safely say this is not the symptom or the cause of our problems.
Each # represents the forge server being started through the web UI the addition of a vanilla server running.
Everything runs as expected with Thaumcraft-1.8-5.0.2.jar added.
Everything is the same from 2 except I deleted the logs and worlds folders. When starting the server instance the web UI crashes and the user must re-login. The folders log and worlds are not regenerated.
After relogging in from the crash from 3 but not changing a thing everything works expected when starting the server. Both server seem to be running fine over a large span of time without stopping or without causing the web UI to crash.
Edit 1: I was able to replicate this using a vanilla server TestingGrounds. With TestingGrounds stopped. I deleted the logs and worlds folders. When starting the server instance the web UI crashes and the user must re-login. The folders log and worlds are not regenerated. The same results as described above with 3. No other server instances need to be running.
Edit 2: After some more testing deleting latest.log (maybe any *.log in logs folder?) seems to crash the web UI when starting server.
OK, FIRST ISSUE: webui crashes the minecraft instances (resolved)
Turns out that the reason I was unable to reproduce the issue consistently on a different server is because on all other servers, I wasn’t using systemd! Looks like systemd has a default setting that when the main forked process dies, it seeks out and kills all it’s child processes so as not to orphan them. This isn’t the default behavior for any of the other methods.
I was able to update the mineos.service file so that no screen/java instances close when the webui crashes. No matter what happens to the webui now, the instances will remain untouched.
SECOND ISSUE: forge touching/modifying files in a way that crashes npm fireworm (not resolved)
As best as I can tell, this might be a fireworm bug. It’s watching a file that is maybe(?) getting locked/edited/repopulated in a way that fireworm hates, and it’s crashing the module. Due to the asynchronous of the design of nodejs, I’m pretty sure I can’t trap this error in MineOS code, but I’d have to dig into fireworm.
So, overall this second issue is not resolved, but at the least, the crashing webui a) does not take down the server and b) has an open issue with the fireworm dev.
(to zone only).
Right now your /usr/games/minecraft is set to the branch fs_tweaks, which is due to get pulled into the main master branch soon. You’ll want to switch to master soon (but not yet, I’ll let you know) and then execute these lines:
cd /usr/games/minecraft
git checkout master
git reset --hard origin/master
Also, you need to validate your cron jobs. Right now you have a cronjob that runs most certainly not what you’re expecting.
I’m glad you’re able to make headway and that this issue resolved will further MineOS’s code base.
I’m quite new to cronjob. I created that originally and wasn’t able deleted between the issues I was having with the web UI. Thank you for sparing me. That website to validate cron jobs would be helpful resource in the mineOS wiki.
As a future note most MC server admins that have heavily modded servers restart their servers every 24 or 12 hours. 0 0 * * * will execute cron jobs every 24 hours every day of the week .
To the best of my understanding that 24 hour period starts upon executing cronjob. Which means an administrator has to be up at the time that he wants to be 24 to start.
