Free trial

Kafka

Early Adopter

Apache Kafka is an open-source, distributed publish-subscribe message bus designed to be fast, scalable, and durable. DESK automatically recognizes Kafka processes and instantly gathers Kafka metrics on the process and cluster levels.

Prerequisites

  • DESK SaaS/Managed version 1.155+
  • Apache Kafka or Confluent-supported Kafka 0.9.0.1+
  • If you have more than one Kafka cluster, separate the clusters into individual process groups via an environment variable in DESK settings

Activation

  1. In the navigation menu, select Settings.
  2. Select Monitoring > Monitored technologies.
  3. Find Kafka and set the Global monitoring switch to the On position.
    After you turn Kafka monitoring on, DESK automatically activates Kafka monitoring on all hosts and monitors all Kafka components.

Events

Name Condition DESK event
Under-replicated partitions Partition followers are out-of-sync with the leader Performance (PERFORMANCE_EVENT)
Offline partitions There are no partition leaders Performance (PERFORMANCE_EVENT)
Cluster controller mismatch There are multiple controllers detected by brokers Error (ERROR_EVENT)

To customize problem detection thresholds for Kafka, go to Settings > Anomaly detection > Plugin events and find Kafka in the list.

Metrics

Cluster metrics

Metric Description
Partitions All partition replicas available on this broker. The leader partition counts as a partition replica. This should be even across the cluster.
Under replicated partitions The number of under-replicated partitions in the cluster. Under-replicated partitions indicate that replication is ongoing, consumers aren’t getting data, and latency is growing.
Offline partitions The number of partitions without active leaders and thus not writable.
Active cluster controllers The number of active controllers in the cluster. An alert is raised if the aggregated sum across all brokers in the cluster is anything other than 1, because there should be exactly one controller per cluster.

Broker metrics

Metric Description
Mean time Time taken to flush the partition log to disk either exceeds time to flush or exceeds maximum size.
95th percentile The 95th percentile of log flush time. Even a slight log flush time change can drastically impact Kafka performance.
Incoming byte rate The incoming broker byte rate throughput from clients (consumers, producers, and connectors).
Outgoing byte rate The outgoing broker byte rate throughput from clients (consumers, producers, and connectors).
Partitions All partition replicas available on this broker. The leader partition counts as a partition replica. This should be even across the cluster.
Under replicated partitions The number of under-replicated partitions.
Produce request rate The produce request rate.
Failed produce requests The rate of produce requests that failed.
Produce latency The produce latency.
Fetch request rate The fetch request rate.
Failed fetch requests The number of failed fetch requests.
Leader election rate Election rates go up when there are broker failures.
Unclean election rate Unclean election rate.
Leader count Partition leaders on this broker.
Request queue size Size of the request queue. A congested request queue will not be able to process incoming or outgoing requests.
Messages in rate Messages in rate.
Max follower lag Maximum lag in messages between the follower and leader replicas. This is controlled by the replica.lag.max.messages config. Lag is measured as the difference in offset between follower broker and leader broker. Max lag is the lag of the partition that is the most out of sync.
ZooKeeper disconnects The ZooKeeper client is disconnected from the ensemble: the client has lost its connection to a server and is trying to reconnect. The session is not necessarily expired.
ZooKeeper expires The ZooKeeper session expire rate. When a session expires, we can have leader changes and even a new controller. It is important to keep an eye on the number of such events across a Kafka cluster. If the overall number is high:
  1. Check the health of your network
  2. Check for garbage collection issues and tune it accordingly
  3. If necessary, increase the session time out by setting the value of zookeeper.session.timeout.ms

Request metrics

Metric Description
Requests per second Requests per second.
Total time per request Total time per request.

Kafka producer, consumer, and connect metrics

Metric Description
Requests Number of requests processed per second by client.
Request size Average size of request in a one-minute frame.
Incoming/outgoing byte rate Processed byte rate by client.