Introduction
Prometheus can be used to collect monitoring data from Kafka.
To enable this you need to update your cluster configuration to include a metricsConfig section for both the cluster and zookeeper.
kafka: authorization: type: simple superUsers: - CN=kowl - CN=connect ... metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yml ... zookeeper: replicas: 1 storage: deleteClaim: false size: 10Mi type: persistent-claim metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: zookeeper-metrics-config.yml key: zookeeper-metrics-config.yml
This is the kafka-metrics.yaml for creating the config map: kafka-metrics.yaml
Then add kafka exporter component.
kafkaExporter: groupRegex: ".*" topicRegex: ".*" logging: debug enableSaramaLogging: true readinessProbe: initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: initialDelaySeconds: 15 timeoutSeconds: 5
When you start your cluster you should see an additional pod running for Kafka exporter.
You can create a service for this using this yaml file: kafka-exporter.yaml
Using minikube tunnel you can view the metrics using this link: localhost:9100/metrics
Next edit the scrape configs section in prometheus.yaml to include an entry for these metrics.
scrape_configs: - job_name: prometheus static_configs: - targets: - localhost:9090 - job_name: kafka-exporter scrape_interval: 10s metrics_path: /metrics static_configs: - targets: - kafka-exporter.kafka:9100
You can then setup the grafana dashboard to view your metrics : kafka-exporter-overview_rev5.json
These metrics can also be imported into telegraf by adding the following lines to your inputs:
[[inputs.prometheus]] urls = ["http://kafka-exporter.kafka:9100/metrics"] response_timeout = "10s"
Note: It is also possible to obtain JMX metrics using jolokia (GitHub - paksu/kafka-jolokia-telegraf-collector: Simple Kafka broker JMX metric collection with Telegraf) but this requires creating a custom a custom kafka cluster image if you are using Strimzi.
Links
Grafana Dashboard for Kafka Exporter