Prometheus can be used to collect monitoring data from Kafka.
To enable this you need to update your cluster configuration to include a metricsConfig section for both the cluster and zookeeper.
metricsConfig
kafka: authorization: type: simple superUsers: - CN=kowl - CN=connect ... metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: kafka-metrics-config.yml ... zookeeper: replicas: 1 storage: deleteClaim: false size: 10Mi type: persistent-claim metricsConfig: type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: kafka-metrics key: zookeeper-metrics-config.yml key: zookeeper-metrics-config.yml
This is the kafka-metrics.yaml for creating the config map: kafka-metrics.yaml
Then add kafka exporter component.
Kafka Exporter
kafkaExporter: groupRegex: ".*" topicRegex: ".*" logging: debug enableSaramaLogging: true readinessProbe: initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: initialDelaySeconds: 15 timeoutSeconds: 5