利用Prometheus监控Kafka Jmx

Kafka可以用CMAK(Kafka Manager)监控集群基本信息，但是更加详细的Jmx信息无法展现。可以通过Prometheus + Grafana监控完整的Kafka Jmx信息

Environment

Software	Version
Kafka	1.0.0
Grafana	6.2.5
Prometheus	2.10.0
jmx_prometheus_javaagent	0.9

Prometheus + Grafana就不多说了，部署也很简单

Kafka Jmx

Exposed Jmx

• kafka-server-start.sh

......
exec $base_dir/kafka-run-class.sh $EXTRA_ARGS kafka.Kafka "$@"
......

• kafka-run-class.sh

......
# JMX settings
if [ -z "$KAFKA_JMX_OPTS" ]; then
  KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false "
fi

# JMX port to use
if [  $JMX_PORT ]; then
  KAFKA_JMX_OPTS="$KAFKA_JMX_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT "
fi
......

• Start kafka Server

// 引入JMX_PORT
JMX_PORT=9999 bin/kafka-server-start.sh config/server.properties &

Get Jmx Info

jdk/bin/jconsole.exe，远程连接 IP:JMX_PORT

• Metrics详细参数参见官网

Kafka Jmx Metrics

Prometheus Jmx Agent

jmx_prometheus_javaagent 下载

• kafka-1_0_0.yml

lowercaseOutputName: true
rules:
- pattern : .*

• jmx_prometheus_javaagent 配置

// 启动的时候引入
// 7999为prometheus采集端口
// /app/prometheus_javaagent/kafka/ 为自定义路径
export KAFKA_OPTS="-javaagent:/app/prometheus_javaagent/kafka/jmx_prometheus_javaagent-0.9.jar=7999:/app/prometheus_javaagent/kafka/kafka-1_0_0.yml"

• 依次重启Kafka 每个broker节点

Prometheus Scrape Data

• prometheus.yml

// 添加配置
- job_name: kafka
  scrape_interval: 30s
  static_configs:
- targets:
    - ip1:7999
    - ip2:7999
    - ip3:7999
    - .....

• 动态刷新配置

curl -X POST http://IP:9090/-/reload

Grafana

Manage -> Import -> Json

• Kafka Dashboard Json 下载

• Dashboard

---

转载请注明出处：https://github.com/imperio-wxm

---