[appendix]
[id='metrics-{context}']
== Metrics

This section describes how to monitor Strimzi Kafka and ZooKeeper clusters using Grafana dashboards.
In order to run the example dashboards you must configure Prometheus server and add the appropriate https://github.com/prometheus/jmx_exporter[Prometheus JMX Exporter] rules to your Kafka cluster resource.

WARNING: The resources referenced in this section serve as a good starting point for setting up monitoring, but they are provided as an example only.
If you require further support on configuration and running Prometheus or Grafana in production then please reach out to their respective communities.

ifdef::InstallationAppendix[]
When adding Prometheus and Grafana servers to an Apache Kafka deployment using `minikube` or `minishift`, the memory available to the virtual machine should be increased (to 4 GB of RAM, for example, instead of the default 2 GB). Information on how to increase the default amount of memory can be found in the following section <<installing_kubernetes_and_openshift_cluster>>.
endif::InstallationAppendix[]

=== Kafka Metrics Configuration

Strimzi uses the https://github.com/prometheus/jmx_exporter[Prometheus JMX Exporter] to export JMX metrics from Kafka and ZooKeeper to a Prometheus HTTP metrics endpoint that is scraped by Prometheus server.
The Grafana dashboard relies on the Kafka and ZooKeeper Prometheus JMX Exporter relabeling rules defined in the example `Kafka` resource configuration in https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/kafka/kafka-metrics.yaml[`kafka-metrics.yaml`].
Copy this configuration to your own `Kafka` resource definition, or run this example, in order to use the provided Grafana dashboards.

==== Deploying on {OpenShiftName}

To deploy the example Kafka cluster the following command should be executed:

[source,shell,subs=attributes+]
oc apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/kafka/kafka-metrics.yaml

ifdef::Kubernetes[]
==== Deploying on {KubernetesName}

To deploy the example Kafka cluster the following command should be executed:

[source,shell,subs=attributes+]
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/kafka/kafka-metrics.yaml

endif::Kubernetes[]

=== Prometheus

The provided Prometheus https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/prometheus.yaml[`prometheus.yaml`] YAML file describes all the resources required by Prometheus in order to effectively monitor a Strimzi Kafka & ZooKeeper cluster.
These resources lack important production configuration to run a healthy and highly available Prometheus server.
They should only be used to demonstrate this Grafana dashboard example.

The following resources are defined:

* A `ClusterRole` that grants permissions to read Prometheus health endpoints of the Kubernetes system, including cAdvisor and kubelet for container metrics.  The Prometheus server configuration uses the Kubernetes service discovery feature in order to discover the pods in the cluster from which it gets metrics.  In order to have this feature working, it is necessary for the service account used for running the Prometheus service pod to have access to the API server to get the pod list.
* A `ServiceAccount` for the Prometheus pods to run under.
* A `ClusterRoleBinding` which binds the aforementioned `ClusterRole` to the `ServiceAccount`.
* A `Deployment` to manage the actual Prometheus server pod.
* A `ConfigMap` to manage the configuration of Prometheus Server.
* A `Service` to provide an easy to reference hostname for other services to connect to Prometheus server (such as Grafana).

Prometheus also provides an alerting system through the link:https://prometheus.io/docs/alerting/alertmanager/[alert manager] component.
In order to enable alerting, the provided link:https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alerting-rules.yaml[`alerting-rules.yaml`] file describes a `ConfigMap` resource which defines sample alerting rules on Kafka and Zookeeper metrics.
When an alert condition is evaluated as true on the Prometheus server, it sends the alert data to the alert manager which then uses the configured notification methods to notify the user.

More information about how to set up alerting rules https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/[here].

==== Deploying on {OpenShiftName}

The provided `prometheus.yaml` file, with all the Prometheus related resources, creates a `ClusterRoleBinding` in the `myproject` namespace.
It also discovers an alert manager instance in the same namespace.
If you are using a different namespace, download the resource file and update it as follows:

[source,shell,subs=attributes+]
curl -s https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/prometheus.yaml | sed -e 's/namespace: .*/namespace: _my-namespace_/;s/regex: myproject/regex: _my-namespace_/' > prometheus.yaml

The provided `alerting-rules.yaml` file creates a `ConfigMap` with sample alerting rules; download the resource file as follows:

[source,shell,subs=attributes+]
curl -s https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alerting-rules.yaml > alerting-rules.yaml

To deploy all these resources you can run the following.

[source,shell,subs=attributes+]
oc login -u system:admin
oc apply -f alerting-rules.yaml
oc apply -f prometheus.yaml

ifdef::Kubernetes[]
==== Deploying on {KubernetesName}

The provided `prometheus.yaml` file, with all the Prometheus related resources, creates a `ClusterRoleBinding` in the `myproject` namespace.
If you are using a different namespace, download the resource file and update it as follows:

[source,shell,subs=attributes+]
curl -s https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/prometheus.yaml | sed -e 's/namespace: .*/namespace: _my-namespace_/;s/regex: myproject/regex: _my-namespace_/' > prometheus.yaml

The provided `alerting-rules.yaml` file creates a `ConfigMap` with sample alerting rules; download the resource file as follows:

[source,shell,subs=attributes+]
curl -s https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alerting-rules.yaml > alerting-rules.yaml

To deploy all these resources you can run the following.

[source,shell,subs=attributes+]
kubectl apply -f alerting-rules.yaml
kubectl apply -f prometheus.yaml

endif::Kubernetes[]

=== Grafana

A Grafana server is necessary to get a visualisation of the Prometheus metrics.  The source for the Grafana docker image used can be found in the `./metrics/examples/grafana/grafana-openshift` directory.

==== Deploying on {OpenShiftName}

To deploy Grafana the following commands should be executed:

[source,shell,subs=attributes+]
oc apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/grafana/grafana.yaml

ifdef::Kubernetes[]
==== Deploying on {KubernetesName}

To deploy Grafana the following commands should be executed:

[source,shell,subs=attributes+]
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/grafana/grafana.yaml

endif::Kubernetes[]

=== Grafana dashboard

As an example, and in order to visualize the exported metrics in Grafana, two sample dashboards are provided https://github.com/strimzi/strimzi-kafka-operator/blob/{GithubVersion}/metrics/examples/grafana/strimzi-kafka.json[`strimzi-kafka.json`] and https://github.com/strimzi/strimzi-kafka-operator/blob/{GithubVersion}/metrics/examples/grafana/strimzi-zookeeper.json[`strimzi-zookeeper.json`].
These dashboards represent a good starting point for key metrics to monitor Kafka and ZooKeeper clusters, but depending on your infrastructure you may need to update or add to them.
Please note that they are not representative of all the metrics available.
No alerting rules are defined.

The Grafana Prometheus data source, and the above dashboards, can be set up in Grafana by following these steps.

NOTE: For accessing the dashboard, you can use the `port-forward` command for forwarding traffic from the Grafana pod to the host. For example, you can access the Grafana UI by running `oc port-forward grafana-1-fbl7s 3000:3000` (or using `kubectl` instead of `oc`) and then pointing a browser to `http://localhost:3000`.

. Access to the Grafana UI using `admin/admin` credentials.  On the following view you can choose to skip resetting the admin password, or set it to a password you desire.
+
image::grafana_login.png[Grafana login]

. Click on the "Add data source" button from the Grafana home in order to add Prometheus as data source.
+
image::grafana_home.png[Grafana home]

. Fill in the information about the Prometheus data source, specifying a name and "Prometheus" as type. In the URL field, the connection string to the Prometheus server (that is, `http://prometheus:9090`) should be specified. After "Add" is clicked, Grafana will test the connection to the data source.
+
image::grafana_prometheus_data_source.png[Add Prometheus data source]

. From the top left menu, click on "Dashboards" and then "Import" to open the "Import Dashboard" window where the provided https://github.com/strimzi/strimzi-kafka-operator/blob/{GithubVersion}/metrics/examples/grafana/strimzi-kafka.json[`strimzi-kafka.json`] and https://github.com/strimzi/strimzi-kafka-operator/blob/{GithubVersion}/metrics/examples/grafana/strimzi-zookeeper.json[`strimzi-zookeeper.json`] files can be imported or their content pasted.
+
image::grafana_import_dashboard.png[Add Grafana dashboard]

. After importing the dashboards, the Grafana dashboard homepage will now list two dashboards for you to choose from.  After your Prometheus server has been collecting metrics for a Strimzi cluster for some time you should see a populated dashboard such as the examples list below.

==== Kafka Dashboard

image::grafana_kafka_dashboard.png[Kafka dashboard]

==== ZooKeeper Dashboard

image::grafana_zookeeper_dashboard.png[ZooKeeper dashboard]

==== Metrics References

To learn more about what metrics are available to monitor for Kafka, ZooKeeper, and Kubernetes in general, please review the following resources.

* http://kafka.apache.org/documentation/#monitoring[Apache Kafka Monitoring] - A list of JMX metrics exposed by Apache Kafka.
It includes a description, JMX mbean name, and in some cases a suggestion on what is a normal value returned.
* https://zookeeper.apache.org/doc/current/zookeeperJMX.html[ZooKeeper JMX] - A list of JMX metrics exposed by Apache ZooKeeper.
* https://kubernetes.io/docs/tasks/debug-application-cluster/resource-usage-monitoring/[Prometheus - Monitoring Docker Container Metrics using cAdvisor] - cAdvisor (short for container Advisor) analyzes and exposes resource usage (such as CPU, Memory, and Disk) and performance data from running containers within pods on Kubernetes.
cAdvisor is bundled along with the kubelet binary so that it is automatically available within Kubernetes clusters.
This reference describes how to monitor cAdvisor metrics in various ways using Prometheus.
** https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md[cAdvisor Metrics] - A full list of cAdvisor metrics as exposed through Prometheus.

=== Prometheus alerting

In the monitoring space, one of the useful aspects is to be notified when some metrics conditions are verified.
They allow a human operator to get notifications about problems in the monitored system.

Prometheus allows to write so called "alerting rules" which describe such a conditions using https://prometheus.io/docs/prometheus/latest/querying/basics/[PromQL] expressions that are continuously evaluated.
When an expression becomes true, the described condition is met and the Prometheus server fires an alert.

Prometheus itself is not responsible for sending notifications to the users when an alert is fired.
A different component, the Prometheus alert manager, is in charge to do so, sending emails, chat messages or using different notification methods.
When an alert condition is verified, the alert is fired and the Prometheus server sends it to the alert manager which will send notifications.

=== Prometheus alert manager

Other than a server for scraping metrics, Prometheus provides an alerting system through the alert manager component.
It is possible to declare alerting rules on the Prometheus server in order to be notified about specific conditions in the metrics.
When an alert condition is evaluated as true, Prometheus sends alert data to the alert manager which then sends notifications out.
Notifications can be sent via methods such as email, Slack, PagerDuty and HipChat

The provided Prometheus https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alertmanager.yaml[`alertmanager.yaml`] YAML file describes all the resources required for deploying and configuring the alert manager.

The following resources are defined:

* A `Deployment` to manage the actual alert manager pod.
* A `ConfigMap` to manage the configuration of the alert manager.
* A `Service` to provide an easy to reference hostname for other services to connect to alert manager (such as Prometheus).

The provided sample configuration configures the alert manager to send notification to a Slack channel.
Before deploying the alert manager it is needed to update the following parameters:

* The `slack_api_url` field with the actual value of the Slack API URL related to the application for the Slack workspace.
* The `channel` field with the actual Slack channel on which sending the notifications.

==== Deploying on {OpenShiftName}

To deploy the alert manager the following commands should be executed:

[source,shell,subs=attributes+]
oc apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alertmanager.yaml

ifdef::Kubernetes[]
==== Deploying on {KubernetesName}

To deploy the alert manager the following commands should be executed:

[source,shell,subs=attributes+]
kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alertmanager.yaml

endif::Kubernetes[]

==== Alerts examples

The provided https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/{GithubVersion}/metrics/examples/prometheus/alerting-rules.yaml[`alerting-rules.yaml`] YAML file provides the following sample alerting rules on Kafka and Zookeeper metrics.

Kafka alerts are:

* `UnderReplicatedPartitions`: the under replicated partitions metric gives the number of partitions for which the current broker is the leader replica but the follower replicas are not caught up.
This metric provides insights about offline brokers which hosts the follower replicas.
This alert is raised when this value is greater than zero, providing the information of the under replicated partitions for each broker.

* `AbnormalControllerState`: the active controller metric indicate if the current broker is the controller for the cluster.
It can just be 0 or 1. 
During the life of a cluster, only one broker should be the controller and the cluster needs to have always an active controller.
Having two or more brokers saying that they are controllers indicates a problem.
This alert is raised when the sum of all the values for this metric on all broker is not equals to 1.
It means that there is no active controller (the sum is 0) or more than one controller (the sum is greater than 1).

* `UnderMinIsrPartitionCount`: the Kafka broker `min.insync.replicas` allows to specify the minimum number of replicas that have to acknowledge a write operation for successful in order to be in-sync.
The under min ISR partition count metric defines the number of partitions that this broker leads for which in-sync replicas count is less than the min in-sync.
This alert is raised when this value is greater than zero, providing the information of the under min ISR partition count for each broker.

* `OfflineLogDirectoryCount`: the offline log directory count metric indicate the number of log directories which are offline (due to an hardware failure for example) so that the broker cannot store incoming messages anymore.
This alert is raised when this value is greater than zero, providing the information of the number of offline log directories for each broker.

* `KafkaRunningOutOfSpace`: the running out of space metric indicates the remaining amount of disk space that can be used for writing Kafka's data.
This alert is raised when this value is lower than 5GiB. It provides information on the disk that is running out of space for each persistent volume claim.
NOTE: The availability of this metric and alert is dependent on your version of {ProductPlatformName}.

Zookeeper alerts are:

* `AvgRequestLatency`: the average request latency metric indicates the amount of time it takes for the server to respond to a client request.
This alert is raised when this value is greater than 10 (ticks), providing the actual value of the average request latency for each server.

* `OutstandingRequests`: the outstanding requests metric indicates the number of queued requests in the server.
This value goes up when the server receives more requests than it can process.
This alert is raised when this value is greater than 10 (ticks), providing the actual number of outstanding requests for each server.

* `ZookeeperRunningOutOfSpace`: the running out of space metric indicates the remaining amount of disk space that can be used for writing data to Zookeeper.
This alert is raised when this value is lower than 5GiB. It provides information on the disk that is running out of space for each persistent volume claim.
Note: The availability of this metric and alert is dependent on your version of {ProductPlatformName}.
