Logging: Enhance log capabilities by adding elastisearch plugin to Winston

Hi @hexparrot.

As a part of work related tasks I am currently learning the ELK stack. As I am still very much in the learning and trying and failing stage, I am playing around on my own servers rather than risk anything on work servers.

So far I have managed to make GROK-queries for two of the loglines in mineos.log (request sent / request recieved when using start / stop button), and I could probably keep parsing the logs and making queries as a traning session. I’ll have to do that to parse the minecraft server logs later on, so I thought I’d hurry up and advance to new and steeper mountains.

After doing some diving into the mineos node script I found that you use Winston to generate logs. Winston has a Elastisearch plugin that would allow me to send log contents straight from mineos to elastisearch, in stead of first to log file, then parsing the logfile.

Before I go ahead running around in you scripts causing havoc, I thought I’d ask for some hints on what changes (and where) would have to be made to add the winston-elastisearch plugin, activate it, and get the logs using it?

I am aiming at doing this myself, so hint, tips only please :slight_smile:

the plugin:

Admittedly, I don’t have experience with the ELK stack, but my suspicion is that 100% of you adapting MineOS to use winston-elasticsearch is 100% within the server.js 10-14 lines.

That is to say, you might simply be changing from a file-based transport to an elastic-based transport, and you’re good to go.

I’ll leave it at that to stay in the hint realm. I’d be happy to help further when requested.

1 Like

Soo… Before even starting to playing around in the scripts I broke my MineOS install…

npm install --save winston winston-elasticsearch and npm install winston winston-kafka-connect --save both claimed to not be able to find a winston installation. even after running npm install winston.

The MineOS isntallation did not like this however, so I tried removing all the winstons, and resetting all the MineOS script (including npm install). No go.

Decided that my MineOS installation probably irrevokably broke, So I removed the entire installation starting from srcratch (I thought), which gave me an even more destroyed installation where the MineOS service crashes almost instantly.

Luckily I’m running this in Virtualbox, and have a snapsot available, so now I’m just backing up the Minecraft servers, then resetting the VM to an earlier state, and trying again.

Augh.
If there is one thing I hate, it’s systems suddenly fixing thems selves.
After backing up all servers, MineOS suddenly works normally again…

Anyway… That means back to experimenting.

So far I have managend to find new and interesting ways to crash MineOS, most related to node.js, and modules.

After cheking around, and thinking I got a stable node.js version, I tried to check out winston. It seems the winstons plugins do not recognise the winston version MineOS uses.

Cheking out version info seems to tell me MineOS uses Winston@1.1.2, while the latest and current one is Winston@3.3.3 : https://www.npmjs.com/package/winston.

I tried upgrading winston, but MineOS really did not like that. I flat out refused to start.

The modules I want is:
winston-kafka-connect (needs winston v3 and up, and winston transport 4.2.0 and up)
winston-elasticsearch (needs winston v3.3.3 and up, and winston transport 4.4.o and up)

It seems I need to get Winston updated, and update MineOS to handle the newer version.

@hexparrot:
Since my focus is the elasticsearch and kibana part, I think I can allow myself to ask for some help with what needs upgrading to handle a newer winston version. Help please?

It’s not you, it’s me.

Also, it seems MineOS-node is reaching a critical mass. Right now, having been built for one of the earliest releases of node, we’re finding the modules I’ve included and depended on have moved on in the direction of Node v12 API and beyond.

And that’s meaning that more and more developers are pushing their npm modules to work with later versions, breaking earlier ones such as v8.

I did a very moderate npm audit fix and it blasted the webui to kingdom come.

There’s going to need to be a fairly serious amount of updating effort put forward if MineOS-node is to survive, and it looks like I’m going to need to spend some time modernizing it.

It’s really quite possible (though with a non-trivial amount of effort) that I can get this working on modern node versions. It’s also quite possible that some behaviors, like new-file-creation-detection might suffer and there could be losses of features. However, this also might mean the end of confusing dependency chains and an easy fix for winston-elasticsearch.

TLDR; alright, I’ll start working on fixing this stale codebase…

Oh dear.
Tell me if I can help, and I’ll do my best. I’m not to versed in node.js, but have some experience in Jquery and webdev. I’m also looking into other technologies due to work, and it seems learnign node-js is a good thing, since it lets me use MineOS as a part of a test and learning platform for things I need to learn now (the above mentioned ELK stack -> Elaticsearch, Logstash, Kibana stack).

(I could off course use syslog and apachelogs, but they have premade modules made, so MineOS and the different logs produced for Mineos and game servers is perfect, since it forces me to learn to do things outside of premade addons and modules. )

For now I can fall back to use filebeat to read and parse the log files, and have other things I can look into as well, so not being able to add modules to winston and MineOS is not stopping me learning :smiley:

For those of you who want, I have a hacked toghether a logstash-pipeline with a GROK that parses MineOS and minecraft logs to logstash. I’ll work on beautifying it, and removing hard coded text, but it works rigth now for the logs currently produced.

Best of luck, I’m looking forward to see what you have in store. If there’s anything I can do to help as well, let me know.

And thank you for the support and hard work!

Would this be a complete overhaul of the interface if you take on this project? would you plan on adding any more functionality in the WebUI?

Currently, MineOS uses AngularJS. That’s Angular versions lower than 2.x.

2.x–and quite frankly, even 1.5 are quite hugely changed.

It looks like it’s a complete overhaul, and quite frankly, it’s looking way too daunting to want to rewrite the webui to get feature parity, much less feature improvements.

If I’m the sole developer of this upgrade (as I expect I’ll be), it is really undetermined how long this can take, or whether it’ll ever get done. mineos-ruby would be out if I didn’t have to do front end (html/js/css) work.

Now, mineos-node is going to be requiring backend js and frontend js work to stay modern.

oof.

1 Like

Thank goodness. I’ve been working to get Angular versions bumped up as much as possible without further code rewrite. I think I’m able to get it up to a bit more modern “right now” without hitting the next major 2.x. (1.5.11) I was able to get socket.io bumped as well, which is a big deal for security related reasons.

This is good news, because I didn’t have to touch the auth stack at all…and if I did, I would expect this to be a huge, long-drawn-out affair.

I’ll look into Winston + elasticsearch now, but for the time being, updating the webui to the latest commit should help on many, many fronts.

I’ve been able to get these modules installed on the current mineos-node commit:

+  "winston": {
+    "version": "3.3.3",

+  "winston-transport": {
+    "version": "4.4.0",

...

     "winston": "1.1.x",
+    "winston-elasticsearch": "^0.10.0",
+    "winston-kafka-connect": "^2.0.12",

Here are the steps I took to get to this point:

  1. reset the webui to the most current commit
  2. npm install winston-elasticsearch winston-kafka-connect

What’s odd is that I kept winston version locked at 1.1.x and despite this, everything still seems to come up.

Upgrading winston to 3.3.3 directly breaks everything. It appears that each specifically-listed module in package.json is allowed to have its own dependency chain. To that end, I can continue to use winston/1.1.2 and you can use winston/3.3.3 for your own needs.

There might be some namespacing issues you have if you put winston-* code into any module currently require('winston)` but if you keep them in separate files or separate JS objects, I imagine this should cause no trouble.

Let me know where you end up with this!

Hey!
nice work. I¨ll try it out, and see what happends :slight_smile:

Thanks for all your hard work :slight_smile:

gave me this result:

@hexparrot: Do this match your result?

The Err! you’re getting is from missing --unsafe-perm like the normal instructions have.

I left it out in my previous post as well (but also because I was working with a unique install). At any rate,

npm install --unsafe-perm winston-elasticsearch winston-kafka-connect