Some stuff for work is running on a Kubernetes 1.3 cluster. It nicely restarted when something went wrong in its environment. That’s good. Our pods were logging – that’s good. But when the pod restarted, its logs were lost: that’s bad. No idea why the pod restarted to help determine if it’s something we should either prevent or add a health check for. So, now I’m evaluating logging solutions appropriate for our Kubernetes environment. And because I’m a geek, I’m blogging about it here in my pjs on my day off: you know you’re in the right field when you’re enjoying researching something for it on your day off.
Scope of problem: not trying to log all of the kubernetes events themselves: that’s the job of the infrastructure. Which also means then I’m not worried about filtering out messages I shouldn’t see from pods that aren’t my own. I want to collect log messages in my namespace, have them tagged (at least) by which application source they came from, and I want to have them be archived _off_ of the cluster. The guys running the infrastructure are great, but if they reconfigure the cluster (as happens occasionally) or have a hiccup with their persistent volumes, still want my logs.
A bit of survey suggests there are two primary solutions out there: Fluentd or Logstash, one of the components of the ELK stack. Both are open-source. I’ve long heard of ELK, though not yet myself futzed with it in a system. I’ve seen many stickers for Fluentd in open-source conferences… 🙂 I’ve also noticed Fluentd as part of the Cloud Native Computing Foundation‘s list of supported projects. Hmm.. that last bit gives me a bit of bias towards Fluentd, sight unseen.
So now to look for other folks comparisons of the two to pick the first one to try out.. My strategy is to start with one and see if it’s ‘good enough’ to meet our needs. But I’ll often rely on what the ecosystem has already discovered to make my choice between.
– Fluentd vs. Logstash: A Comparison of Log Collectors: logz.io uses both. My takeaway – leans towards Fluentd in its recommendation, even though it’s built on ELK
– Fluentd vs. Logstash: Panda Strike’s tried both.. They’ve ended up with EFK (ElasticSearch, Fluentd, Kibana) as their platform for reasons of stability and performance. Side note: describes logs forwarders vs. log shippers. Found another post on their site that describes EFK in action for JMeter load testing result visualization. Another side note: found a good bit of useful content in their blog overall.. adding ’em to the rotation of stuff to keep an eye on…
– Fluentd vs Logstash: Jason Wilder walks through both. Older post (Nov 2013) but assuming all is still relevant, good technical info.
Outcome: looks like I’ll be starting with Fluentd.