Spigot Autobuild

Development
1

Hi there, I am new to the project and have a proposal for a new feature (not a request, Ill go off and make this). I was going to build this into the project for myself, but I figured I would propose it here before I got started just to get feedback and see if this is something that would be accepted as a pull request. If it might be than I will do a much better job of coding this up as a full quality feature rather than just hacking it together for my own uses.

The What
I would love to build in a way to allow spigot BuildTools to be downloaded and used from the WebUI.

The Why
Currently to create/update a spigot server a user needs to ssh into the server, run BuildTools and copy the spigot jar into the new server directory. This is something I believe should be accessible from the web. I’m going to give some of my friends accounts to log into the WebUI and I would like for them to be able to update spigot without needing ssh access.

The How
I have two thoughts on how to implement this. (Keep in mind I am completely unfamiliar with the codebase and my thoughts may change once I get in there and take a look around).

My first thought is to create a Spigot section in the Profiles page drop down. Instead of a “Download” button, there can be a “Build” button. Since spigot BuildTools allows you to target a specific version of minecraft, each version can be built off of the same BuildTools download. If no BuildTools jar exists it will first be downloaded, then the targeted verison of minecraft will be built.

Me second thought is to create a spigot page that can be linked to from the username’s drop down in the top right. The page will have the appropriate buttons and options to download BuildTools and build a specific version of spigot. A spigot profile will become available to use on a server if a build of spigot has been performed on the server.

I personally like the my first thought better, but im not sure if the current codebase will really allow for a such different action to be taken without a ton of work. Ideally I wouldn’t want to change too much of the existing code just to support this, small feature.

Thoughts? Comments?

3 Likes
2

I’ve been tooling around with this for the last day or so to see about which way seemed easiest to get Spigot into the webui. I’d been putting this off for so long because multi-step profiles I knew would take a lot of work–and now I feel suitably ready that I have the time for it.

I took your second approach, which was to create a second page. It’s still in heavy dev, but I’ve actually gotten to build the spigot and craftbukkit jars entirely from the webui. This isn’t to say the workflow is finalized or that it is the best possible approach, but it is the approach that made the most sense to me, while trying to keep the number of code changes minimal–and keep it to mostly code addition.

Feel free to look at it here: https://github.com/hexparrot/mineos-node/tree/spigot_profiles

The general idea of what changes I made are these:

  • profiles continues to be an automatically generated and grouped list of software
  • the buildtools jar, which updates often, operates like the rest of the profiles; you download it and it puts it in a /var/games/minecraft/profiles/Buildtools-45 directory
  • The new page (currently called buildtools, but wouldn’t stay that way) partitions off all the new functionality. In this way, profiles have no immediate changes either to the webui design or the backend.
  • The buildtools page has a dropdown to select the buildtools jar you wish to use. In almost 100% of cases, the user should be expected to use the most recent one, but I still leave it as a step for the user currently
  • The buildtools page actually taps into the onscreen output of BuildTools.jar. There’s so many places that the building can go wrong, that this was unavoidable. There’s an upside (25 lines of actual information) and a downside (this uses just an unbelievable amount of bandwidth).
  • In its current state, it’s still lacking. The files themselves are created in the webui but aren’t moved to the server’s live directory, which is what I’ll be working on next. I’m still trying to come up with a clean way to do this.

Note, I created this branch to help move in the direction of your stated goal, but not to discourage you from your own designs. You are more than welcome to use any of the code I’ve created if you feel that there is something fundamentally improve-able over my choices here, or you may simply wait for me to develop this in entirety.

I have visualized in my mind the appropriate way that these files will be moved over to the right place strictly from the webui and I think I’ll have a working model by the end of the day. I encourage you to give the branch a shot at some point to see if this workflow is intuitive for you and all users who you will be sharing your webui with.

3

It’s now all on there. Able to build spigot/cb entirely from the webui on this branch.

I’ll keep it unmerged for some time to come while I try to get additional eyes on it to make sure it works as one would expect, but it seems like the approach put in place in this branch will require the least amount of codechange possible.

One thing I noticed is that buildtools.jar will fail to build anything that doesn’t have a corresponding json file here: https://hub.spigotmc.org/versions/

So that limits the available options thus far to 1.8, 1.8.3+ (nightlies aren’t included. should they be?)

4

Wow, haha totally didn’t expect that. Thanks so much for doing that so quickly!

After poking around I really like the way it works, but I can’t build at the moment. Im getting an error

fatal: $HOME not set

What should $Home be set to? I don’t remember that being something that BuildTools needed?

Once I can get it all working I will totally let you know my full thoughts and seeing if it needs any improvement and if so where.

Thanks again! I totally didn’t expect you to just go do it.

5

Where are you getting that error? I’m completely lost where such an error might even appeark ,as $HOME isn’t any variable I’m familiar with coding anywhere in the entire project.

Is that at the command line–is your console environment asking you about echo $HOME not being set?

6

Downloaded than ran BuildTools through the new web pages. The BuildTools Output is:

fatal: $HOME not set
Git name not set, setting it to default value.
fatal: $HOME not set
Exception in thread "main" java.lang.RuntimeException: Error running command, return status !=0: [git, config, --global, user.name, BuildTools]
    at org.spigotmc.builder.Builder.runProcess(Builder.java:535)
    at org.spigotmc.builder.Builder.main(Builder.java:154)

Are you kicking off the build as user root? I just tried that manually and I did not get the error.

7

it looks like this is an error from git which needs a $HOME set. My root user (which MineOS runs as) has $HOME set, but it may not be getting set if the terminal is being run in non-interactive mode (complete speculation on my part).

I need to look into if services running as root have access to the normal bash variables that get set for users.

Update: Just confirmed $HOME is not set when running a service through systemd. I just added User=root to the mineos.service file and everything works just fine :slight_smile: Ill play with it a bit more just to make sure that doesn’t screw with anything else. If you are okay with it I can issue a pull request later tonight to update the systemd service file. I don’t know enough about systemd to know if setting the User to root really changes much or has any caveats.

8

Very interesting find! I wouldn’t have even thought to look there–almost all of my testing occurs with me either manually invoking the webui or through supervisord. I’d be interested to hearing your results on the testing; for the most part I highly doubt that there will be any unexpected, unwanted behaviors from user=root.

Thanks for your follow up!

9

No problem! After thinking about it I had a feeling. Already filed the pull request.

As for some feedback on the feature in general. I really like it! You did such a good job with it. I really only have one main problem. I understand why you used the profile page to download the BuildTools jar, but I think that does make things a little confusing and complicates the work-flow.

Firstly, you can now pick spigot buildtools as a profile to run the server with on the Status page. Which may be misleading… and obviously just wont work.

Secondly its a long workflow to go:

Profiles > Click on Select Box > Select Spigot Build Tools > Click Download > go to Spigot / CraftBukkit > Click Select Box > Select Build Tools Version > Click Build.

Now, that may seem like some nit picking… but I personally like to keep interfaces simple with minimal steps to get things done. After thinking about it… would it be possible to just have the select box on the Spigot page just list all the build tools versions that are listed on the Build Tools download page? Then you just select what build tool version (which can just default to the latest which I think most people would want most of the time) and when you click build it will automatically download the build tools first.

I suggest that only because it seems like the BuildTools.jar is so small the download is almost instant. And since the build process is already going to take a bit… we can just tack on a second to download the jar.

Update
I’m fairly new to spigot, but my understanding is that you only need the spigot.jar file to run the server. I don’t think you need to copy all the jars over? Was that intentional?
End Update

What do you think? I’m very interested to hear feedback on my comments. There may be good reasons you did the build tools download through the profiles page.

Also, I understand that profiles recently changed… if profiles isn’t a good name for that page any more and it will become more of a download manager… then it makes sense. I just think calling the BuildTools a profile is a bit misleading. Not that I don’t think users can figure it out, people aren’t stupid, but its just not an optimal work-flow imho.

1 Like
10

No, you’re definitely right on the money with the clunky workflow. For a good amount of time today, I attempted (and succeeded) at moving BuildTools entirely off profiles and onto the actual page where it belonged. By coincidence, I also made it just download the latest–knowing that nobody would want the older ones (and the older ones don’t seem to work anyway).

I went ahead and copy/pasted much of the code behind the profiles, tuned it to work with only one URL, relocated all the web controls to the new page and then finally settled on a hard-coded path for the BuildTools.jar to live in.

Unfortunately, I had to discard that commit because it was just an unmaintainable mess. Looking at the git diff told me that if I ever needed to update that code for whatever reason, I’d also have to remember to do it in two places. And after all that, I even lost functionality of the % downloaded label.

I’m not surrendering just yet–I suspect I’m going to try it again and hopefully it’ll end up differently and I won’t hate what it turns out to be, but in the meantime, the explanation for why it was able to get punched up so quickly was precisely because of the fact I could trust the code that drove profiles.

Even as I type this out, I suspect I have a good solution for this; one that absolutely is way better than the one I produced today. And had I kept what I did today, I’d kick myself for having to undo it again.

That said, BuildTools.jar produces spigot and craftbukkit jars–if BuildTools is going to go through the work to do that, I wouldn’t want to waste it by not copying it–especially if the craftbukkit jar still is useful; I’m guessing it is, or else it shouldn’t even get compiled.

But the BuildTools.jar shouldn’t be moved over. I’ve been battling with rsync to try to make it not copy it over, ‘–exclude’ and all that…but just haven’t found the right recipe (and deleting it after the fact isn’t an option).

So we’ll see tomorrow if my idea for getting BuildTools off profiles works!

11

Alright, it worked out.

With the commit 48abe67d, there’s only one button now, on the Spigot/Craftbukkit page and it will download the Latest successful build. No more appearance in profiles, latest is all that is used now.

12

Hi,

Please forgive my general lack of knowledge, which is the service file that you modified? I am also experiencing the same $HOME issue.

EDIT: Found the file - service.js? Just trying to find out where in the file to place ‘user=root’ where I still do not get the following:

Loading BuildTools version: git-BuildTools-1847da7-45 (#45)
Git name not set, setting it to default value.
fatal: $HOME not set
Exception in thread “main” java.lang.RuntimeException: Error running
command, return status !=0: [git, config, --global, user.name,
BuildTools]
at org.spigotmc.builder.Builder.runProcess(Builder.java:535)
at org.spigotmc.builder.Builder.main(Builder.java:154)
fatal: $HOME not set

END Edit

Thanks!

13

If you take a look at the updated systemd service file in init/systemd_conf it will show you where to add it. The actual file your server is using is most likely located at /etc/systemd/system/mineos.service

In short you just need to add it in the [Service] section in the file.

All of this is assuming you are using systemd. If you are using another service manager then look in the documentation for that manager for how to run a program as a user. Even though you are still running it as root, telling the manager to use the “user” root will set up the needed environment variables.

Let me know if you have any other issues and I’ll see if I can look into it further for your specific situation.

Hope this helps :smile:!

1 Like
14

Thanks KayoticSully!

It does help somewhat, I am running Ubuntu 14.04 and therefore Upstart instead of Systemd. From what I have read of the Upstart docs at http://upstart.ubuntu.com/cookbook/#run-a-job-as-a-different-user, I need to modify the exec stanza in /etc/init/mineos.conf to something like the following:
exec start-stop-daemon --start -c root --exec /usr/bin/node service.js start

This just means I can no longer get into the GUI. I tried wrapping the command in single quotes (’/usr/bin/node service.js start’) but that also did not work with the same “Unable to connect” error in firefox.

Any ideas on what I am doing wrong here?

Thanks again for your help!.

15

Thanks for letting us know your working environment. There’s such a great variety of distros and startup management scripts it’s hard to keep them all in check by myself; I depend on your useful input to ensure my Dev time is focused and properly prioritized. I’ll hopefully have a full answer for your upstart answer committed tomorrow.

16

Thanks hexparrot,

If there is anything else I can tell you about my setup, just ask. I am looking forward to getting this running! This has been a great and very useful tool that I am definitely getting some use out of.

17

Alright, I updated the upstart script and it works as expected now:

exec /usr/bin/node service.js start
to
exec sudo -u root /usr/bin/node service.js start

Fetch the most recent commit then copy it over:

git fetch
cp init/upstart_conf /etc/init/mineos.conf
stop mineos
start mineos
18

Awesome! Looks way better than what I was attempting.

Just followed your instructions, saw that there was an update pulled down, and the change was not present in upstart_conf. Manually edited /etc/init/mineos.conf and it works fine.

Thank you!