I built Kubernetes Event Watcher

The Kubernetes Event Watcher serves as an automated monitoring and notification system for Kubernetes clusters. Its primary functions are to monitor Kubernetes clusters for specific critical events as per the users configurations.

Overview

Kubernetes triggers events whenever there is a state change within your cluster. These are stored somewhere in the cluster and have a limited retention time, by default 1 hour. These events provide information that can supplement debugging in the case of deployed resource failing.

You can extend the TTL from 1 hour using the tag referenced here — event-ttl but some cloud providers do not provide a way to natively do this. Hence my desire to create a custom event watcher that is customizable and can be deployed to any cluster.

I then created my own app, kubernetes-watcher, that I can deploy to my cluster and it will inform me of any events that I configure it to watch. The app is available on GitHub if you would like to check it out https://github.com/mamin11/kubernetes-watcher.

Currently, this allows me to configure what events to watch through kubernetes configmaps. These are then injected into the app at start time and will trigger an alert if a matching event if found within the cluster. This config looks something like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filter-config
data:
  config.json: |
    [
      {
        "kind": "Pod",
        "reasons": ["Killing", "Failed", "FailedScheduling", "BackOff", "Unhealthy"]
      }
    ]

The above will add a filter to watch events that happen on Pods and the reason for the event is in the list provided. We can add as many filters as we like just by adding a new filter object to the array.

System Purpose

The Kubernetes Event Watcher serves as an automated monitoring and notification system for Kubernetes clusters. Its primary functions are to:

  • Monitor Kubernetes clusters for specific critical events
  • Filter events based on configurable criteria
  • Format event information into human-readable notifications
  • Deliver notifications via configured channels (Slack and email), more notification providers to come

The system helps operations teams quickly identify and respond to issues in their Kubernetes environments without constant manual monitoring, or have these sent to messaging channels of choice.

System Architecture

We have a kubernetes client api config which allows us to communicate with the kubenetes internal API. This then gives us access to events, pods, services and much more. For our case we are interested in the events.

We use the event watcher service to continually listen for events that occur within the cluster. We also filter these so that we only forward the events that we are interested in. More on this filtering later.

Then we forward these events to the notifications service, which has multiple adapter, slack and emails for now. Based on whichever is enabled or if both, we send a notification through the channel. You can configure which channels you want to receive events through configmaps.

Importantly, for our app to be able to interact with the kubernetes API, we need this service account created. This will allow use to get, and watch events within the cluster.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: k8s-event-watcher-cluster-role
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch"]

Then this configmap to configure the notification channels. The following is an extract but full file can be found here https://github.com/mamin11/kubernetes-watcher/blob/main/manifests/configmaps.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: k8s-watcher-config
data:
  ...
  notification.slack.config.enabled: "true"
  notification.slack.config.channel: "k8s-events"
  notification.slack.config.username: "Kubernetes-Watcher"
  notification.email.config.enabled: "true"
  notification.email.config.host: "host.docker.internal"
  notification.email.config.port: "1025"
  notification.email.config.protocol: "smtp"
  notification.email.config.to: "support@mycompany.com"

Deployment

The Kubernetes Event Watcher is designed to be deployed as a container within a Kubernetes cluster. We have a manifest directory with all the files you need to deploy this to your cluster. The standard deployment includes:

  • Docker container image (mamin11/kubernetes-watcher:v1.0.0)
  • Kubernetes Deployment resource
  • ConfigMap for configuring notifications and event filters
  • Secret for sensitive information (email and slack credentials)
  • ServiceAccount with appropriate permissions

Testing and Development

The system includes test resources to:

  • Simulate Kubernetes events
  • Verify notification delivery
  • Test configuration changes

Test pods can be deployed to trigger specific event types, making it easier to validate the systems functionality. Manifests for these can be found here in the .test-pods directory https://github.com/mamin11/kubernetes-watcher/tree/main/.test-pods.