Kubernetes Monitoring and Alerting

This article describes how to monitor a Kubernetes cluster. In the process of running MogDB cluster, you need to monitor container resources, hosts, Kubernetes components, etc. For the monitoring of these components or resources, you need to deploy monitoring system in the whole Kubernetes cluster dimension to achieve it.

Host Monitoring

Node Exporter is an official node resource collection component provided by Prometheus that can be used to collect data on server nodes, such as CPU frequency information, disk IO statistics, remaining free memory, etc. Node Exporter converts the collected information into Prometheus-aware Metrics data. Prometheus can collect and store these metrics from the Node Exporter and monitor server node resources based on real-time changes in these data.

Container Monitoring

cAdvisor is Google's open source container resource monitoring and performance analysis tools, it is specifically for containers, in Kubernetes, we do not need to go to a separate installation, cAdvisor as part of the kubelet built-in program can be used directly, that is, we can directly use cAdvisor to collect data, you can collect and container operation of all the relevant indicators.

The typical monitoring metrics obtained in cAdvisor are as follows:

Metric Name	Type	Description
container_cpu_load_average_10s	gauge	Average container CPU load over the last 10 seconds
container_cpu_usage_seconds_total	counter	Cumulative time occupied by containers on each CPU core (in seconds)
container_cpu_system_seconds_total	counter	System CPU cumulative occupancy time (in seconds)
container_cpu_user_seconds_total	counter	User CPU cumulative occupancy time (in seconds)
container_fs_usage_bytes	gauge	File system usage in containers (in bytes)
container_fs_limit_bytes	gauge	Total amount of file systems available to the container (in bytes)
container_fs_reads_bytes_total	counter	Total amount of data read by the container cumulatively (in bytes)
container_fs_writes_bytes_total	counter	Total amount of data written to the container cumulatively (in bytes)
container_memory_max_usage_bytes	gauge	Maximum memory usage of the container (in bytes)
container_memory_usage_bytes	gauge	The current memory usage of the container (in bytes)
container_spec_memory_limit_bytes	gauge	Memory usage limits for containers
machine_memory_bytes	gauge	Total memory of the current host
container_network_receive_bytes_total	counter	Total cumulative data received by the container network (in bytes)
container_network_transmit_bytes_total	counter	Total cumulative data transferred by container network (in bytes)

Resource Object Monitoring

kube-state-metrics provides monitoring of the resource objects themselves, such as pod running status, how many jobs are running, etc. It is developed based on client-go, polls the Kubernetes API, and converts structured information from Kubernetes into metrics.

Metrics types include:

• CronJob Metrics

• DaemonSet Metrics

• Deployment Metrics

• Job Metrics

• LimitRange Metrics

• Node Metrics

• PersistentVolume Metrics

• PersistentVolumeClaim Metrics

• Pod Metrics

• Pod Disruption Budget Metrics

• ReplicaSet Metrics

• ReplicationController Metrics

• ResourceQuota Metrics

• Service Metrics

• StatefulSet Metrics

• Namespace Metrics

• Horizontal Pod Autoscaler Metrics

• Endpoint Metrics

• Secret Metrics

• ConfigMap Metrics

MogDB Monitoring

In order to get the monitoring metrics of MogDB itself, you need the accompanying MogDB exporter to collect the corresponding data.

Architecture