Selfhosted

I also dislike graphana kabana elastic behemot.

You can use rsyslog to centralize the logs. Then there's tools like this for anomaly detection on those logs.

I use logcheck which should be available with your distro. It's simple but pretty dumb though. It works by scanning your system logs, excludes any pre-configured regexps (it already comes with defaults for many of the most common logs), and sends you an email if there are any unexpected logs. I did have to add a bunch of custom regexps to exclude additional logs specific to my setup. But I just did this by adding new regexps whenever I got a logcheck email that I deemed irrelevant so not terribly difficult.

The end result is that I get an email with logs whenever anything unexpected happens. For example, I get emails whenever any SSH session is established (including my own) which gives me the confidence that if something starts going down, I should be able to see it.

I went for kiss and very very low resources: All my logs go to one central syslog. That takes in the logs, filters out crap, send critical messages to IRC and writes logs to postgres. Then grafana queries the postgresdb

Graylog might be worth a look

In years past, I've used Elasticsearch and Kibana. The learning curve is steep and the system resource requirements warrant a dedicated machine, but once you get it dialed, it's really effective as a centralized logging server.

Prometheus and Grafana are for time-series data (metrics), not logs. If you're already getting that from netdata, don't bother with these, as they'd be redundant with what you have.

syslog is about as idiomatic as it gets for log management in linux, but i don't have enough experience using it effectively to give any pointers there. If you don't really know what you want, yet, and just want to collect logs from all the things and see them in one place so you can begin to try and make sense of them and make refinements from there, then syslog seems like an excellent place to start.

Having used GreyLog and Loki&Vector/Grafana I would recommend the latter.

Pretty sure GoAccess covers all types of logging as far as I know, I mainly use it for access logs from my reverse proxy.

One thing about grafana, though, is that you get logs, metrics and monitoring in the same package. You can use loki as the actual log store and it's easy to integrate it with the likes of journald and docker.

Yes, you will have to spend more time learning LogQL, but it can be very handy where you don’t have metrics (or don’t want to implement them) and still want some useful data from logs.

After all, text logs are just very raw, unstructured events in time. You may think that you only look into them very occasionally when things break and you would be correct. But if you want to alert on them, oftentimes that means you’re going from raw logs to structured data. Loki's LogQL does that, and it's still ten times easier to manage than the elastic stack.

VictoriaMetrics has its own logging product too, now, and while I didn’t try it yet, VM for metrics is probably the best thing ever happened since Prometheus. Especially for resource constrained homelabs.

I use the elastic stack at work and it's more than I want to set up for my homelab. I'm looking at Loki but didn't start anything yet. I'm already running Grafana.

i just stumbled upon this and i like it: https://github.com/control-theory/gonzo