I just did a fresh install of MineOS on my Arch server and cant seem to get systemd to start it up. I can start it manually though the service.js script, but the systemd call fails. As far as I can tell though inserting a few console.log statements, webui.js is up and failing before it can even get started when daemonize2 tries to start it. I’m not sure what could be different between starting service.js through systemd and starting it as the root user, but something obviously is. I am using the supplied systemd service script.
I do not know if this is a bug with daemonize2 or if something is mis-configured in my system. Any ideas?
Thanks!
Here is my systemd status mineos output including a few of my extra debug log statements.
● mineos.service - Start MineOS minecraft front-end
Loaded: loaded (/etc/systemd/system/mineos.service; disabled; vendor preset: disabled)
Active: failed (Result: resources) since Thu 2015-09-10 17:42:33 EDT; 4min 38s ago
Process: 1100 ExecStart=/usr/bin/node service.js start (code=exited, status=0/SUCCESS)
Sep 10 17:42:33 sherlock systemd[1]: Starting Start MineOS minecraft front-end...
Sep 10 17:42:33 sherlock node[1100]: SERVICE START
Sep 10 17:42:33 sherlock node[1100]: daemon START
Sep 10 17:42:33 sherlock node[1100]: Starting mineos daemon...
Sep 10 17:42:33 sherlock node[1100]: daemon STARTed
Sep 10 17:42:33 sherlock node[1100]: mineos daemon failed to start: Module '/usr/games/minecraft/webui.js' stopped unexpected
Sep 10 17:42:33 sherlock systemd[1]: mineos.service: PID 1105 read from file /var/run/mineos.pid does not exist or is a zombie.
Sep 10 17:42:33 sherlock systemd[1]: Failed to start Start MineOS minecraft front-end.
Sep 10 17:42:33 sherlock systemd[1]: mineos.service: Unit entered failed state.
Sep 10 17:42:33 sherlock systemd[1]: mineos.service: Failed with result 'resources'.
Did you run the SSL generator, generate-ssl.sh?
I’m pretty sure I did. I’ll double check and make sure it ran successfully when I get a chance.
If that’s not the fix, just try running the ./webui.js as root and any errors should be produced to the screen (or /var/log/mineos.log) so we can know definitively.
So I reran generate-ssl.sh as root which ran fine. Then I started webui.js as root and got the following log:
cat /var/log/mineos.log
{"level":"info","message":"[WEBUI] ryan connected from 10.0.0.136","timestamp":"2015-09-10T23:11:02.575Z"}
{"level":"info","message":"[WEBUI] Downloading official profiles.","timestamp":"2015-09-10T23:11:02.747Z"}
{"redownload":true,"command":"refresh_profile_list","level":"info","message":"[WEBUI] Received emit command from 10.0.0.136:ryan","timestamp":"2015-09-10T23:11:41.687Z"}
{"level":"info","message":"[WEBUI] Downloading official profiles.","timestamp":"2015-09-10T23:11:41.688Z"}
{"command":"refresh_server_list","level":"info","message":"[WEBUI] Received emit command from 10.0.0.136:ryan","timestamp":"2015-09-10T23:11:45.039Z"}
So it seems like webui.js is running just fine when I run from root. It also runs fine when I start it as a daemon through service.js as root. Its only when running from systemd that has any issue.
It seems like the issue is webui.js never gets run at all. I had experimented by putting a console.log("HERE") as the first line in webui.js and that was never echoed out at all. If daemonize2 hides logs somewhere else then maybe I would not have seen it.
This is completely breaking my concept of systemd though… it should run fine if it runs okay as root.
For what its worth. If I start it up though ./service.js start as root it does work fine, but then if I try to stop it with systemd (which I’m not sure if it would even work anyway, but it should just run node service.js stop anyway…) it does not stop the server.
I’m really at a loss. I can start up the server manually by running ./service.js so its not the end of the world… but I can’t have it start on boot, right now.
I did a lot of investigating and I think it has something to do with systemd and maybe it screws with daemonize2 or something.
I have officially ruled out that it has nothing to do with my server config (since I locked it down with the grsec kernel and paxd) I booted using the standard kernel and stopped paxd, same problem.
At the point where daemonize2 fails it is getting a return code of 1 from the process it forks. I’m not sure if that is code 1 from its wrapper.js file which didnt seem to run at all or from webui.js. Or if child_process.spawn is just failing on return code of 1.
Also I noticed that daemonize2 says it supports node 0.8.x and above. That was about two years ago since it has been touched and node is now on v0.12.7. Could there just be a problem with the package? I just cant see that since it does work when I start it outside of systemd, but just a thought.
Thats about as far as I was able to get. Has anyone else experienced weird problems with systemd? Maybe this is an Arch specific problem?
Hrm, still trying to reproduce this issue.
[root@mineos-arch ~]# systemctl status mineos
- mineos.service - Start MineOS minecraft front-end
Loaded: loaded (/etc/systemd/system/mineos.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2015-09-11 11:22:14 MST; 1min 39s ago
Process: 233 ExecStart=/usr/bin/node service.js start (code=exited, status=0/SUCCESS)
Main PID: 241 (mineos)
CGroup: /system.slice/mineos.service
`-241 mineos
Sep 11 11:22:13 mineos-arch systemd1: Starting Start MineOS minecraft front-end…
Sep 11 11:22:13 mineos-arch node[233]: Starting mineos daemon…
Sep 11 11:22:14 mineos-arch node[233]: mineos daemon started. PID: 241
Sep 11 11:22:14 mineos-arch systemd1: Started Start MineOS minecraft front-end.
Freshly installed, then did the pacman -Syu to update all the packages. Rebooted and then followed the instructions on the wiki.
The installed version of node at this point was 0.12.7:
nodejs-0.12.7-1-x86_64 2.7 MiB 393K/s 00:07 [#######################################] 100%
npm-2.14.3-1-any 1950.6 KiB 411K/s 00:05 [#######################################] 100%
I’ll continue looking into this. What are your versions?
Hmm I just did a fresh install of Arch yesterday. I just completely uninstalled MineOS and cleaned a few things up. Its as close to fresh as I could get it. The only main differences is I am using the grsec kernel and paxd is running for security. Both configured with defaults, I don’t see them causing any issues. If we determine that is the only difference between our systems I can uninstall them and try again.
I just updated and am rebooting now. Ill do a fresh install of MineOS and see what happens.
My system has the following packages (sorry if its a bit much, just want to make sure). Also I have Oracle JDK installed. So you won’t see OpenJDK in the list.
autoconf 2.69-2
automake 1.15-1
bash 4.3.042-1
bash-completion 2.1-5
binutils 2.25.1-3
bison 3.0.4-1
bzip2 1.0.6-5
checksec 1.5-4
coreutils 8.24-1
cpupower 4.2-1
cryptsetup 1.6.7-1
device-mapper 2.02.125-1
dhcpcd 6.9.3-1
diffutils 3.3-2
e2fsprogs 1.42.12-2
fakeroot 1.20.2-1
file 5.24-1
filesystem 2015.02-1
findutils 4.4.2-6
flex 2.5.39-1
gawk 4.1.3-1
gcc 5.2.0-2
gcc-libs 5.2.0-2
gettext 0.19.5.1-1
git 2.5.1-1
glibc 2.22-3
grep 2.21-2
grub 1:2.02.beta2-5
gzip 1.6-1
hdparm 9.48-1
hidepid 1-1
htop 1.0.3-2
inetutils 1.9.4-1
iproute2 4.0.0-2
iputils 20140519.fad11dc-1
jfsutils 1.1.15-4
less 479-1
libtool 2.4.6-3
licenses 20140629-1
linux 4.1.6-1
linux-grsec 4.1.6.201508230818-1
logrotate 3.9.1-1
lvm2 2.02.125-1
m4 1.4.17-1
make 4.1-1
man-db 2.7.2-1
man-pages 4.02-1
mdadm 3.3.4-1
nano 2.4.2-1
netctl 1.11-1
nodejs 0.12.7-1
npm 2.14.3-1
openssh 7.1p1-1
os-prober 1.66-1
pacman 4.2.1-2
patch 2.7.5-1
pax-utils 1.0.5-1
paxd 30-1
paxtest 0.9.13-1
pciutils 3.3.1-1
pcmciautils 018-7
perl 5.22.0-1
pkg-config 0.28-2
procps-ng 3.3.11-1
psmisc 22.21-2
rdiff-backup 1.2.8-7
reiserfsprogs 3.6.24-1
rsync 3.1.1-4
s-nail 14.8.5-1
screen 4.3.1-1
sed 4.2.2-3
shadow 4.2.1-3
sudo 1.8.14.p3-2
sysfsutils 2.1.0-9
systemd-sysvcompat 225-1
tar 1.28-1
texinfo 5.2-3
usbutils 008-1
util-linux 2.26.2-1
vi 1:070224-1
vim 7.4.854-1
wget 1.16.3-1
which 2.21-1
xfsprogs 3.2.4-1
I just reinstalled MineOS following the wiki to a T. Still not working. I dont have physical access to the server right now so I can’t do a full reinstall but I have a few more things I can try.
Update: I uninstalled the grsec kernel, paxd, and hidepid (another security measure that I had installed). Rebooted, and tried again. Same problem. But I can still run it normally through service.js.
Also my arch version is 4.1.6-1-ARCH #1 SMP PREEMPT if that helps.
Update 2 Welp I just borked my server, and I don’t have physical access to it this weekend. Ill completely re-install Arch from scratch late sunday and get MineOS installed as early as possible to see if systemd magically wants to work.
mineos.service - Start MineOS minecraft front-end
Loaded: loaded (/etc/systemd/system/mineos.service; enabled; vendor preset: disabled)
Active: failed (Result: start-limit) since Fri 2015-09-11 16:08:48 EDT; 5s ago
Process: 1243 ExecStart=/usr/bin/node service.js start (code=exited, status=0/SUCCESS)
Sep 11 16:08:48 sherlock systemd[1]: Failed to start Start MineOS minecraft front-end.
Sep 11 16:08:48 sherlock systemd[1]: mineos.service: Unit entered failed state.
Sep 11 16:08:48 sherlock systemd[1]: mineos.service: Failed with result 'resources'.
Sep 11 16:08:48 sherlock systemd[1]: mineos.service: Service hold-off time over, scheduling restart.
Sep 11 16:08:48 sherlock systemd[1]: Stopped Start MineOS minecraft front-end.
Sep 11 16:08:48 sherlock systemd[1]: mineos.service: Start request repeated too quickly.
Sep 11 16:08:48 sherlock systemd[1]: Failed to start Start MineOS minecraft front-end.
Sep 11 16:08:48 sherlock systemd[1]: mineos.service: Unit entered failed state.
Sep 11 16:08:48 sherlock systemd[1]: mineos.service: Failed with result 'start-limit'.
So in an effort to see if this is reproduce-able I set up a Virtual Box VM with a fresh install of Arch 64 bit and did nothing but update install MineOS. Same problem. So I can reproduce it. I used the latest arch iso newly downloaded just to make sure.
Not sure what to make of it… I’m going to give it another go just to make sure and Ill post my exact set up when done.
I’ll also download the most recent Arch ISO (although I’m not sure I already didn’t have it) and try it again and perhaps we’ll see if anything surfaces on my end.
Alright, give that a shot… but after reformatting my server yet again… it freaking works now… the ONLY difference is it seems to fail when I format my drive using lvm2. I swear that was the only thing that changed. I formatted just a straight partition in ext4 rather than going though the lvm2 setup.
Either way, this is fixed for now, but would be interesting to figure out what was going on.
Marking this as solved since it is not an issue any more.
For future readers: If this problem seems like an issue you are having the only change I made was to reformat my server and NOT use lvm2 for my drive partitions. I have no clue if that is what was causing an issue (I highly doubt it) but its the only thing I can think of that was different. Give it a shot and if you have any extra info to add please do! I’m still interested in figuring the problem out if anyone else runs int oit.
1 Like