Logging: Enhance log capabilities by adding elastisearch plugin to Winston

[iMelsom]

Hi @hexparrot.

As a part of work related tasks I am currently learning the ELK stack. As I am still very much in the learning and trying and failing stage, I am playing around on my own servers rather than risk anything on work servers.

So far I have managed to make GROK-queries for two of the loglines in mineos.log (request sent / request recieved when using start / stop button), and I could probably keep parsing the logs and making queries as a traning session. I’ll have to do that to parse the minecraft server logs later on, so I thought I’d hurry up and advance to new and steeper mountains.

After doing some diving into the mineos node script I found that you use Winston to generate logs. Winston has a Elastisearch plugin that would allow me to send log contents straight from mineos to elastisearch, in stead of first to log file, then parsing the logfile.

Before I go ahead running around in you scripts causing havoc, I thought I’d ask for some hints on what changes (and where) would have to be made to add the winston-elastisearch plugin, activate it, and get the logs using it?

I am aiming at doing this myself, so hint, tips only please

the plugin:

winston-elasticsearch

[hexparrot]

Admittedly, I don’t have experience with the ELK stack, but my suspicion is that 100% of you adapting MineOS to use winston-elasticsearch is 100% within the server.js 10-14 lines.

That is to say, you might simply be changing from a file-based transport to an elastic-based transport, and you’re good to go.

I’ll leave it at that to stay in the hint realm. I’d be happy to help further when requested.

[iMelsom]

Soo… Before even starting to playing around in the scripts I broke my MineOS install…

npm install --save winston winston-elasticsearch and npm install winston winston-kafka-connect --save both claimed to not be able to find a winston installation. even after running npm install winston.

The MineOS isntallation did not like this however, so I tried removing all the winstons, and resetting all the MineOS script (including npm install). No go.

Decided that my MineOS installation probably irrevokably broke, So I removed the entire installation starting from srcratch (I thought), which gave me an even more destroyed installation where the MineOS service crashes almost instantly.

Luckily I’m running this in Virtualbox, and have a snapsot available, so now I’m just backing up the Minecraft servers, then resetting the VM to an earlier state, and trying again.

[iMelsom]

Augh.
If there is one thing I hate, it’s systems suddenly fixing thems selves.
After backing up all servers, MineOS suddenly works normally again…

Anyway… That means back to experimenting.

[iMelsom]

So far I have managend to find new and interesting ways to crash MineOS, most related to node.js, and modules.

After cheking around, and thinking I got a stable node.js version, I tried to check out winston. It seems the winstons plugins do not recognise the winston version MineOS uses.

Cheking out version info seems to tell me MineOS uses Winston@1.1.2, while the latest and current one is Winston@3.3.3 : https://www.npmjs.com/package/winston.

I tried upgrading winston, but MineOS really did not like that. I flat out refused to start.

The modules I want is:
winston-kafka-connect (needs winston v3 and up, and winston transport 4.2.0 and up)
winston-elasticsearch (needs winston v3.3.3 and up, and winston transport 4.4.o and up)

It seems I need to get Winston updated, and update MineOS to handle the newer version.

@hexparrot:
Since my focus is the elasticsearch and kibana part, I think I can allow myself to ask for some help with what needs upgrading to handle a newer winston version. Help please?

[hexparrot]

It’s not you, it’s me.

Also, it seems MineOS-node is reaching a critical mass. Right now, having been built for one of the earliest releases of node, we’re finding the modules I’ve included and depended on have moved on in the direction of Node v12 API and beyond.

And that’s meaning that more and more developers are pushing their npm modules to work with later versions, breaking earlier ones such as v8.

I did a very moderate npm audit fix and it blasted the webui to kingdom come.

There’s going to need to be a fairly serious amount of updating effort put forward if MineOS-node is to survive, and it looks like I’m going to need to spend some time modernizing it.

It’s really quite possible (though with a non-trivial amount of effort) that I can get this working on modern node versions. It’s also quite possible that some behaviors, like new-file-creation-detection might suffer and there could be losses of features. However, this also might mean the end of confusing dependency chains and an easy fix for winston-elasticsearch.

TLDR; alright, I’ll start working on fixing this stale codebase…

[iMelsom]

Oh dear.
Tell me if I can help, and I’ll do my best. I’m not to versed in node.js, but have some experience in Jquery and webdev. I’m also looking into other technologies due to work, and it seems learnign node-js is a good thing, since it lets me use MineOS as a part of a test and learning platform for things I need to learn now (the above mentioned ELK stack -> Elaticsearch, Logstash, Kibana stack).

(I could off course use syslog and apachelogs, but they have premade modules made, so MineOS and the different logs produced for Mineos and game servers is perfect, since it forces me to learn to do things outside of premade addons and modules. )

For now I can fall back to use filebeat to read and parse the log files, and have other things I can look into as well, so not being able to add modules to winston and MineOS is not stopping me learning

For those of you who want, I have a hacked toghether a logstash-pipeline with a GROK that parses MineOS and minecraft logs to logstash. I’ll work on beautifying it, and removing hard coded text, but it works rigth now for the logs currently produced.

[fornaxbeowulf]

Best of luck, I’m looking forward to see what you have in store. If there’s anything I can do to help as well, let me know.

And thank you for the support and hard work!

[fornaxbeowulf]

Would this be a complete overhaul of the interface if you take on this project? would you plan on adding any more functionality in the WebUI?

[hexparrot]

Currently, MineOS uses AngularJS. That’s Angular versions lower than 2.x.

2.x–and quite frankly, even 1.5 are quite hugely changed.

@fornaxbeowulf

Would this be a complete overhaul of the interface if you take on this project?

It looks like it’s a complete overhaul, and quite frankly, it’s looking way too daunting to want to rewrite the webui to get feature parity, much less feature improvements.

If I’m the sole developer of this upgrade (as I expect I’ll be), it is really undetermined how long this can take, or whether it’ll ever get done. mineos-ruby would be out if I didn’t have to do front end (html/js/css) work.

Now, mineos-node is going to be requiring backend js and frontend js work to stay modern.

oof.

[hexparrot]

Thank goodness. I’ve been working to get Angular versions bumped up as much as possible without further code rewrite. I think I’m able to get it up to a bit more modern “right now” without hitting the next major 2.x. (1.5.11) I was able to get socket.io bumped as well, which is a big deal for security related reasons.

This is good news, because I didn’t have to touch the auth stack at all…and if I did, I would expect this to be a huge, long-drawn-out affair.

I’ll look into Winston + elasticsearch now, but for the time being, updating the webui to the latest commit should help on many, many fronts.

[hexparrot]

@iMelsom

The modules I want is:
winston-kafka-connect (needs winston v3 and up, and winston transport 4.2.0 and up)
winston-elasticsearch (needs winston v3.3.3 and up, and winston transport 4.4.o and up)

I’ve been able to get these modules installed on the current mineos-node commit:

+  "winston": {
+    "version": "3.3.3",

+  "winston-transport": {
+    "version": "4.4.0",

...

     "winston": "1.1.x",
+    "winston-elasticsearch": "^0.10.0",
+    "winston-kafka-connect": "^2.0.12",

Here are the steps I took to get to this point:

1. reset the webui to the most current commit
2. npm install winston-elasticsearch winston-kafka-connect

What’s odd is that I kept winston version locked at 1.1.x and despite this, everything still seems to come up.

Upgrading winston to 3.3.3 directly breaks everything. It appears that each specifically-listed module in package.json is allowed to have its own dependency chain. To that end, I can continue to use winston/1.1.2 and you can use winston/3.3.3 for your own needs.

There might be some namespacing issues you have if you put winston-* code into any module currently require('winston)` but if you keep them in separate files or separate JS objects, I imagine this should cause no trouble.

Let me know where you end up with this!

[iMelsom]

Hey!
nice work. I¨ll try it out, and see what happends

Thanks for all your hard work

[iMelsom]

@hexparrot

npm install winston-elasticsearch winston-kafka-connect

gave me this result:
https://nik.melsom.be/statusnpmadd.png

@hexparrot: Do this match your result?

[hexparrot]

The Err! you’re getting is from missing --unsafe-perm like the normal instructions have.

I left it out in my previous post as well (but also because I was working with a unique install). At any rate,

npm install --unsafe-perm winston-elasticsearch winston-kafka-connect

[iMelsom]

So an update from my endeavours of trying to use the elkstack for logging…

I am back to square one after a rather serious hardwarecrash.

A powerunit to my NAS shorted out, taking out my fuses which brought my server crashing down powerless. it seems this was one to many powrouts for my RAID-controller, so my 5 disk RAID6 got filled with bad superblocks and corrupted files. I had backups of all the important parts, but most of my virtual machines got viped out. Important data on them where backed up, so I am safe in that way, but I had to try to recue as much as possible anyway,so I bought licenses for a couple of heavy duty file recovery softwares. getting things out took the most fo 4 days.

After that, to eliminate the bad superblocks I had to deep format the enirte RAID volume. 12 Terrabyte of disk takes days to deep format!

So sinceI needed to dismantel and take everything down anyway I decided to do some much needed hardware upgrades as well as swapping base server OS.

My new setup is running with a new porcessor, new motherboad, and DDR4 memory. It also runs on Ubuntu server (previous iteration of my home server was windows 2012R2) with KVM instead of virtualbox. My RAID also got a filesystemswap from NTFS to EXT4 sine I had to reformat deep anyway.

So I am just now getting to catch up with MineOS, and are reinstalling it from the bottom up in a new virtual KVM instance. new name: MineServVI. My sixt iteration of a MineOSServer:D

I’ll kkeep you posted when I get to the point where I can start wokring on logging again

[iMelsom]

Update:
Fresh install on ubuntu server 20.04 with latest commit.

added
npm install --unsafe-perm winston-elasticsearch winston-kafka-connect

added
npm install --unsafe-perm winston@^3

changes to server.js:
commented out :
/*
logging.add(new logging.transports.File(), {
filename: ‘/var/log/mineos.log’,
handleExceptions: true
});
*/

modified:
var logging= require(‘winston’); to const winston = require(‘winston’);

Added:
const logging =
winston.createLogger({
level: ‘info’,
// format: winston.format.json(),
transports: [new winston.transports.File({ filename: ‘/var/log/mineos.log’, handleExceptions: true})]
});
I now have winstons 3 running with addons, and it logs to mineos.log as before.

Unintended effect:
I can add cron jobs, but not activate or deactivate them. This errors out.
Other than that everything seems to work.

Other peculiarities:
Due to some symlinking, fstabbing, and sambafying there may be some rights errors since I have some problems:

1. creating a new server works, but adding server files takes a while, and not always shows up in sever jar selector.
2. Starting a new server worked, but it refused to stop. I had to kill it
3. MineOS seems to “randomly” restart.

Next step:
Try to send logs to Kafka as well as log file.

[iMelsom]

Update:
I finally did it! My MineOS is now repporting over Kafka to Logstach (in an active ELK-stack).

To do it I had to install “winston-kafka-connect” version 1.2.3 (older version, but supports the version of winston MineOS uses.

After that I only had to add the following in server.js beneath the include for winston

logging.transports.Kafka = require('winston-kafka-connect');
.

Then add the following little snippet:

logging.add(logging.transports.Kafka, {
  //defaults
    name: 'WinstonKafkaLogger',
    timestamp: function() {return Date.now()},
    formatter: JSON.stringify,
    meta: {hostname:"MineOS", dataset:"mineos.log"},
    kafkaHost: "192.168.2.9:9092",
    clientId: "winston-kafka-logger",
    topic: "MineOs"
});

I kept the original logging function as well, so now it logs both to kafka and to file.

I had to do some config on the recieving end in Logstash, and configure a grok-filter to parse the incomming logs.

Please note:
If you want to do this you need to have at least a server running kafka. MineOS produces a lot of “cant connect to kafka server” and occationally crashes if the server you try to send to do not exist or respond.

You also have to have a log handler on the other side of karka to parse and handle the logs. I chose the ELK-stack, and configured the apropriate plugins and addosn, and a grok-filter to parse my logs. This is definately for advanced users willing to expreriment (just getting my ELK-stack running took me quite a few tries and resets)

Do not try to add a newer version of winston-kafka-connect. It needs a newer verson of winston-transports, and tries to find a newer version of winston. I managed to get both old and new version of winston to run in parallel, with transport and kafka available, but it either crashed my MineOS server, or did nothing. There is simply to much dependencies between winston and modules. rewriting MineOS to support both also proved a bit of a big task.

[iMelsom]

Planned updates for my modification:

• Move configuration variables out to a config file from the script file. Right now al the configuration for the Kafka connector are hard coded in server.js. this means that everybody that want to use this needs to modify server.js, rather than filling out a config file

• check to see if the kafka service is available and answering on the given port, skip log option if not available

• See if I can modify the WebUI to include an “advanced” tab that gives added options for things like this.

This is now over from work related learning, to private scripting, so I’ll have to look at this when I have itme.

[iMelsom]

I’m also looking into adding the RCON node.js plugin, so that I can read console data from a minecraft server, and pass them through winston and kafka to logstash.

RCON node.js: GitHub - randunel/node-srcds-rcon: Node.JS high-level wrapper for SRCDS's remote console (RCON) https://developer.valvesoftware.com/wiki/RCON