Kubernetes Monitoring and Logging

1. Introduction
2. Kubernetes Logging
3. SLOs in Kubernetes
4. ECK Elastic Cloud on Kubernetes
5. Telegraf Operator
6. Monitoring Certificates Expiration
7. kubeshark
8. k8spacket
9. Kubelog
10. Microsoft Retina eBPF
11. Videos

Introduction

• DZone: Kubernetes Monitoring Essentials
• kube-prometheus Use Prometheus to monitor Kubernetes and applications running on Kubernetes. This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
  • prometheus-community/kube-prometheus-stack 🌟🌟 kube-prometheus-stack collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.
  • faun.pub: Becoming DevOps — Observability Observability with Kube-Prometheus
  • levelup.gitconnected.com: Installing & Exploring the Kube-Prometheus Project
• medium: Kubernetes Monitoring: Kube-State-Metrics
• Kubernetes Monitoring 101 — Core pipeline & Services Pipeline
• medium: Utilizing and monitoring kubernetes cluster resources more effectively
• sysdig.com: Seven Kubernetes monitoring best practices every monitoring solution should enable
• magalix.com: Best Practices And Tools For Monitoring Your Kubernetes Cluster
• sysdig.com: Monitoring Kubernetes in Production
• sysdig.com: How to monitor Kubernetes control plane
• thenewstack.io: 12 Critical Kubernetes Health Conditions You Need to Monitor
• devopscurry.com: Best Open-Source Monitoring Tools for Kubernetes in 2021
• circonus.com: 12 Critical Kubernetes Health Conditions You Need to Monitor and Why
• circonus.com: Guide to Kubernetes Monitoring: Part 1
  • circonus.com: Guide to Monitoring Kubernetes, Part 2: Which Metrics and Health Conditions You Should be Monitoring
• infracloud.io: Monitoring Kubernetes cert-manager Certificates with BotKube - botkube.io
• kube-state-metrics 🌟 Add-on agent to generate and expose cluster-level metrics. kube-state-metrics is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. (See examples in the Metrics section below.) It is not focused on the health of the individual Kubernetes components, but rather on the health of the various objects inside, such as deployments, nodes and pods.
• itnext.io: Monitoring Kubernetes Jobs
• cncf.io: Avoiding Kubernetes cluster outages with synthetic monitoring
• medium: Replication Controller & Replica sets in Kubernetes
• kubermatic.com: The Complete Guide to Kubernetes Metrics
• arabitnetwork.com: K8S – Enabling Auditing Logs | Step-by-Step
• youtube.com: Cloud Quick POCs - Kubernetes monitoring metrics using Grafana Cloud on AWS EKS | Observability | Grafana
• loft.sh: Kubernetes Cost Monitoring with Prometheus & Grafana
• anaisurl.com: Full Tutorial: Monitoring and Troubleshooting stack with Prometheus, Grafana, Loki and Komodor 🌟
• blog.flant.com: Collecting system information from a bunch of Kubernetes clusters
• medium.com/is-it-observable: How to collect metrics in a Kubernetes cluster
• itnext.io: How to tackle Kubernetes observability challenges with Pixie
• medium.com/@lucapompei91: Kubernetes observability
• dev.to: Monitoring Kubernetes cluster logs and metrics using Grafana, Prometheus and Loki
• hitesh-pattanayak.medium.com: Observability in Kubernetes
• middlewareinventory.com: Get CPU and Memory Usage of NODES and PODS – Kubectl 🌟
• betterstack.com: 10 Best Kubernetes Monitoring Tools in 2022 🌟
• medium.com/@kylekhunter: Kubernetes Monitoring with Prometheus
• adamtheautomator.com: Utilizing Grafana & Prometheus Kubernetes Cluster Monitoring 🌟 In this guide, you’ll learn how to monitor your Kubernetes cluster, viewing internal state metrics with a Prometheus and Grafana dashboard.
• grafana.com: Introducing Kubernetes Monitoring in Grafana Cloud Kubernetes Monitoring is available to all Grafana Cloud users, including on free tier. Container orchestration to deploy at scale, iterate quickly, and manage a large number of apps and services.
• medium.com/@clymeneallen: Best Practices, Monitoring System for Multi-K8s Cluster Environments Using Open Source
• medium.com/@magstherdev: OpenTelemetry on Kubernetes 🌟
• betterprogramming.pub: 6 Metrics To Watch for on Your K8s Cluster 🌟 The most critical Kubernetes metrics to monitor. In this article, you will cover the 6 most critical metrics based on Kubernetes’s metadata that form a good baseline for monitoring your workloads and ensuring they’re in a healthy state.
• figments.medium.com: Observable Kubernetes Cluster Using Grafana-Loki-Prometheus
• medium.com/@isalapiyarisi: Getting Started on Kubernetes observability with eBPF
• 8 Best Kubernetes monitoring tools; Paid & open-source
• dev.to/mikeyglitz: Proactive Kubernetes Monitoring with Alerting In this tutorial, you’ll learn how to combine Prometheus, Alertmanager, Grafana and Linkerd to deliver timely alerts when a problem occurs in a Kubernetes cluster.
• isovalent.com: What are the 4 Golden Signals for Monitoring Kubernetes?
• grafana.com: How to manage high cardinality metrics in Prometheus and Kubernetes
• containiq.com: Leveraging Kubernetes Metrics | Tutorial & Tools To Use In Kubernetes, resource metrics, cluster state metrics, and control plane metrics are mission-critical. In this article, we explore these three types of metrics, explain how to use them, and introduce a number of tools to collect and monitor them.
• medium.com/@HirenDhaduk1: Top Kubernetes Observability Tools and their Usage
• milindasenaka96.medium.com: Setup Prometheus and Grafana to Monitor the K8s Cluster
• blog.fourninecloud.com: Kubernetes monitoring — How to monitor using prometheus?
• rcarrata.com: Network Observability Deep Dive in Kubernetes with NetObserv Operator How can we analyze our Network Flows in our Kubernetes clusters? How can we enable Network Observability for Kubernetes in a simple, searchable and visual way? How can we leverage cool technologies such as eBPF or IPFIX to enable Network Observability for our K8s Network Traffic?
• kemilad.medium.com: Monitoring-Stack Deployment To A Kubernetes Cluster — Prometheus | Grafana | AlertManager | Loki + Exporters | Dashboards and etc 🌟
• newrelic.com: Pixie
  • xgrid.medium.com: Tackling Kubernetes Observability Challenges with Pixie
  • newrelic.com: What Is eBPF and Why Does It Matter for Observability?
• awstip.com: Monitoring Your EKS Cluster with the Power of Prometheus and Grafana through Helm
• grafana.com: A beginner’s guide to Kubernetes application monitoring
• medium.com/@poseidon.os: Poseidon: A Kubernetes Cluster Visualization & Cost Analysis Tool
• aws.amazon.com: Using Prometheus to Avoid Disasters with Kubernetes CPU Limits 🌟 In this article, you’ll discuss how CPU throttling can affect Kubernetes’ node performance, and how to avoid this by setting the right values for limits. The author also suggests using Prometheus as a tool to help set reasonable limits
• umeey.medium.com: Four Golden Signals Of Monitoring: Site Reliability Engineering (SRE) Metrics Golden Signal Monitoring using FastAPI on k8s
• grafana.com: How to optimize resource utilization with Kubernetes Monitoring for Grafana Cloud 🌟 Overprovisioning or underprovisioning your Kubernetes resources can have significant consequences on both your budget and your app performance.
• medium.com/@lambdaEranga: Monitor Kubernets Services/Endpoints with Prometheus Blackbox Exporter 🌟 In this article, you will discuss monitoring different endpoints/services in a Kubernetes cluster using Prometheus Blackbox Exporter
• samiislam0306.medium.com: Insightful Monitoring of Kubernetes Clusters with Traces Gain valuable insights into the performance of your Kubernetes cluster with trace monitoring. A guide that helps you optimize your operations for maximum efficiency and productivity.
• medium.com/@walissonscd: Monitoring Kubernetes Cluster Resources: Using Top Metrics Commands
• blog.devops.dev: Prometheus metrics within Kubernetes — an aerial view | Joseph Esrig
• grafana.com: How to monitor Kubernetes clusters with the Prometheus Operator In this guide, you’ll learn how to deploy and use the Prometheus Operator to configure and manage Prometheus instances in your Kubernetes cluster. You’ll also discover how to deploy Grafana to help analyze and visualize the health of your clusters.
• blog.palark.com: Service communication monitoring in Kubernetes with NetFlow
• betterprogramming.pub: Improve Cluster Monitoring With Network Mapping in Grafana A deep dive into obtaining network maps and correlating IP with cluster workloads to speed up debugging
• betterprogramming.pub: Kubernetes Observability Part 1: Events, Logs, and Integration With Slack, OpenAI, and Grafana Build a Kubernetes custom controller to watch Kubernetes Events and forward them to Grafana Loki using Promtail
• itnext.io: Kubernetes: monitoring with Prometheus — exporters, a Service Discovery, and its roles
• opentelemetry.io: Creating a Kubernetes Cluster with Runtime Observability
• medium.com/@onai.rotich: Understand container metrics and why they matter In this 2-part article, you will explore the key metrics to scrape in your cluster
• kkamalesh117.medium.com: Setting up Prometheus and Grafana Integration on Kubernetes with Helm
• medium.com/@MetricFire: Monitoring Kubernetes tutorial: Using Grafana and Prometheus
• medium.com/globant: Monitoring a multi-cluster Kubernetes Deployment Deploying a high resilience monitoring and observation platform for Kubernetes multi-cluster solutions
• medium.com/@martin.hodges: Adding observability to a Kubernetes cluster using Prometheus Monitoring your services is vital and should be considered as part of your underlying infrastructure for your services. You should put this in place ahead of creating and deploying your services. In this article I look at how to deploy Prometheus provide the observability you need to run your services.
• addozhang.medium.com: Non-intrusive Inject OpenTelemetry Auto-Instrumentation in Kubernetes
• medium.com/@abhisman.sarkar: Kubernetes Monitoring: Effective Cluster Tracking with Prometheus
• aws.plainenglish.io: Mastering Monitoring: The Complete Guide to Using Prometheus and Grafana with Kubernetes
• medium.com/@muppedaanvesh: A Hands-On Guide to Kubernetes Monitoring Using Prometheus & Grafana Understanding Prometheus & Grafana Setup in Kubernetes: A Comprehensive Guide
• signoz.io: Kubernetes Cluster Monitoring with OpenTelemetry | Complete Tutorial 🌟

Kubernetes Logging

• cncf.io: Logging in Kubernetes: EFK vs PLG Stack
• medium: How to Deploy an EFK stack to Kubernetes
• digitalocean.com: How To Set Up an Elasticsearch, Fluentd and Kibana (EFK) Logging Stack on Kubernetes
• portworx.com: How to backup and restore Elasticsearch on Kubernetes
• elastic.co: Elastic Stack Monitoring with Elastic Cloud on Kubernetes (ECK - official operator) In this blog post, we’ll explore how the official ElasticCloud on Kubernetes operator can be used to easily deploy and manage ElasticStack Monitoring using the new Beat CRD.
• papertrail.com: Quick and Easy Way to Implement Kubernetes Logging The SolarWinds® Papertrail™ team is excited to announce SolarWinds rKubeLog, an open-source project designed to streamline Kubernetes logging. rKubeLog allows you to forward logs to Papertrail from within a Kubernetes cluster without using a daemon or setting up application-level logging or a logging sidecar.
• qlinh.com: Leveraging Kubernetes audit logs for threat detection Kubernetes audit logs can provide great visibility into the operation and inner workings of your cluster. Learn how to leverage Kubernetes audit logs for threat detection
• itnext.io: Kubernetes Logging in Production
• opensource.com: What you need to know about cluster logging in Kubernetes 🌟 Explore how different container logging patterns in Kubernetes work.
• devopscube.com: Kubernetes Logging Tutorial For Beginners 🌟
• aws.plainenglish.io: Kubernetes Deep Dive: Log Management Part 28 of a series of articles about learning k8s!
• medium.com/vmacwrites: Kubernetes Audit Logs: Who created or deleted a namespace? Learn how to set up log forwarding and collect audit logs that are passed through the Kubernetes API server to IBM Log Analysis to check who initiated a request and when they did so.
• shivanshu1333.medium.com: Structured logging in Kubernetes
  • Logs are an essential aspect of observability and a critical tool for debugging. But Kubernetes logs have traditionally been unstructured strings, making any automated parsing difficult and any downstream processing, analysis, or querying challenging to do reliably.
  • In Kubernetes 1.19, we are adding support for structured logs, which natively support (key, value) pairs and object references. We have also updated many logging calls such that over 99% of logging volume in a typical deployment are now migrated to the structured format.
  • To maintain backwards compatibility, structured logs will still be outputted as a string where the string contains representations of those “key”=”value” pairs. Starting in alpha in 1.19, logs can also be outputted in JSON format using the –logging-format=json flag.
• tealfeed.com: Kubernetes Audit Logs: Who created or deleted a namespace? Learn how to set up log forwarding and collect audit logs that are passed through the Kubernetes API server to IBM Log Analysis to check who initiated a request and when they did so.
• containiq.com: Kubernetes Audit Logs | Use Cases & Best Practices Kubernetes audit logs are powerful, but only if enabled and correctly configured. This article will help you get started using audit logs, and show you how to get the most out of them.
• blog.devops.dev: Importance of Logging In Kubernetes, Intro to Grafana Loki & deploying with helm-charts
• faun.pub: Kubernetes Practice — Logging with Logstash and FluentD by Sidecar Container We are going to learn how to use the Sidecar Container pattern to install Logstash and FluentD on Kubernetes for log aggregation.
• dev.to: Kubernetes Practice — Logging with Logstash and FluentD by Sidecar Container
• signoz.io: Kubernetes Audit Logs - Best Practices And Configuration
• blog.amhaish.com: Observing the K8 cluster using ELK stack
• kube-logging/logging-operator The Logging operator automates the deployment and configuration of a Kubernetes logging pipeline. The operator deploys and configures a Fluent Bit daemonset on every node to collect container and application logs from the node file system.
• akyriako.medium.com: Kubernetes Logging with Grafana Loki & Promtail in under 10 minutes 🌟 Consolidate all your Kubernetes logs in a intuitive Grafana dashboard.
• yuminlee2.medium.com: Kubernetes: Container and Pod Logging
• medium.com/kubernetes-tutorials: Cluster-level Logging in Kubernetes with Fluentd
• shivanshu1333.medium.com: Contextual Logging in Kubernetes The Structured Logging Working Group has added new capabilities to the logging infrastructure in Kubernetes. This post explains how developers can use those to make log output more useful.

SLOs in Kubernetes

• thenewstack.io: Service Level Objectives in Kubernetes an SLO is simply a metric, a goal for that metric, and a time period. For instance: “the success rate for service A must be at least 99.7% percent over the past 30 days.” The metric is known as the “service level indicator” (SLI) and the goal is the “objective.”
• thenewstack.io: SLOs in Kubernetes, 1 Year Later

ECK Elastic Cloud on Kubernetes

• elastic.co: How to configure Elastic Cloud on Kubernetes with SAML and hot-warm-cold architecture Elastic Cloud on Kubernetes (ECK) is an easy way to get the Elastic Stack up and running on top of Kubernetes. That’s because ECK automates the deployment, provisioning, management, and setup of Elasticsearch, Kibana, Beats, and more.

Telegraf Operator

• influxdata.com: Expand Kubernetes Monitoring with Telegraf Operator

Monitoring Certificates Expiration

• itnext.io: Monitoring Certificates Expiration in Kubernetes with X.509 Exporter

kubeshark

• kubeshark.co The API Traffic Viewer for kubernetes. Deep visibility and monitoring of all API traffic and payloads going in, out and across containers and pods inside a Kubernetes cluster.
• kubeshark/kubeshark The API traffic viewer for Kubernetes providing deep visibility into all API traffic and payloads going in, out and across containers and pods inside a Kubernetes cluster. Think TCPDump and Wireshark re-invented for Kubernetes
• medium.com/kernel-space: KubeShark: Wireshark for Kubernetes

k8spacket

• medium.com/@bareckidarek: TCP packets traffic visualization for kubernetes by k8spacket and Grafana
• medium.com/geekculture: K8s Networking — k8spacket Use k8spacket and Grafana to visualize K8s cluster Traffic
• pakdailytimes.com: TCP packets traffic visualization for kubernetes by k8spacket and Grafana

Kubelog

• kubelog.de kubelog is a graphical log viewer for Kubernetes, which works with your existing Kubernetes logging infrastructure. Kubelog is a log viewer for kubernetes. Tail multiple pods in one view and use searches to highlight and show results in context.

Microsoft Retina eBPF

• github.com/microsoft/retina - retina.sh eBPF distributed networking observability tool for Kubernetes
  • Retina is a cloud-agnostic, open-source Kubernetes network observability platform that provides a centralized hub for monitoring application health, network health, and security. It provides actionable insights to cluster network administrators, cluster security administrators, and DevOps engineers navigating DevOps, SecOps, and compliance use cases.
  • Retina collects customizable telemetry, which can be exported to multiple storage options (such as Prometheus, Azure Monitor, and other vendors) and visualized in a variety of ways (like Grafana, Azure Log Analytics, and other vendors).
• blog.teknews.cloud/aks: A look at Retina on AKS

Videos

Click to expand!