Kafka 브로커의 JMX 메트릭을 exporter 로 노출하고 Prometheus 가 긁어 Grafana 로 본다. 브로커에서 JMX 를 켜는 방법은 Kafka JMX enable, exporter 종류는 Prometheus Exporter 를 볼 것.
Kubernetes 환경에서 운영하는 경우는 Prometheus 로 간다.
현행 버전은 3.14.0, LTS 는 3.13.3 이다[1]. 아래 기록은 2.31.1 로 구성한 것이라 경로만 다르고 절차는 같다.
wget https://github.com/prometheus/prometheus/releases/download/v3.14.0/prometheus-3.14.0.linux-amd64.tar.gz
tar -xzf prometheus-3.14.0.linux-amd64.tar.gz
위 버전과 저장소는 오래된 것이라 저장소가 존재하지 않을 수 있다.
[haedong@haedongg.net:~]$ wget https://github.com/prometheus/prometheus/releases/download/v2.31.1/prometheus-2.31.1.linux-amd64.tar.gz
[haedong@haedongg.net:~]$ tar -xvzf prometheus-2.31.1.linux-amd64.tar.gz
prometheus-2.31.1.linux-amd64/
prometheus-2.31.1.linux-amd64/consoles/
prometheus-2.31.1.linux-amd64/consoles/index.html.example
prometheus-2.31.1.linux-amd64/consoles/node-cpu.html
prometheus-2.31.1.linux-amd64/consoles/node-disk.html
prometheus-2.31.1.linux-amd64/consoles/node-overview.html
prometheus-2.31.1.linux-amd64/consoles/node.html
prometheus-2.31.1.linux-amd64/consoles/prometheus-overview.html
prometheus-2.31.1.linux-amd64/consoles/prometheus.html
prometheus-2.31.1.linux-amd64/console_libraries/
prometheus-2.31.1.linux-amd64/console_libraries/menu.lib
prometheus-2.31.1.linux-amd64/console_libraries/prom.lib
prometheus-2.31.1.linux-amd64/prometheus.yml
prometheus-2.31.1.linux-amd64/LICENSE
prometheus-2.31.1.linux-amd64/NOTICE
prometheus-2.31.1.linux-amd64/prometheus
prometheus-2.31.1.linux-amd64/promtool
관리 편의를 위해 /etc/prometheus/ 경로에 설정 파일들을 넣는 것으로 했지만, 실행 시 지정할 수 있으므로 경로를 꼭 맞출 필요는 없다.
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/prometheus.rules
alerting:
alertmanagers:
- scheme: http
static_configs:
- targets:
- "alertmanager.monitoring.svc:9093"
scrape_configs:
- job_name: 'kubernetes-apiservers'
kubernetes_sd_configs:
- role: endpoints
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: default;kubernetes;https
- job_name: 'kubernetes-nodes'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
- job_name: 'kube-state-metrics'
static_configs:
- targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']
- job_name: 'kubernetes-cadvisor'
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
kubernetes_sd_configs:
- role: node
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
action: replace
target_label: __scheme__
regex: (https?)
- source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
action: replace
target_label: __address__
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
- action: labelmap
regex: __meta_kubernetes_service_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
# Kubernetes cluster를 운영 중일 때
- job_name: 'Cluster_current_state'
static_configs:
- targets: ['192.168.192.168:9100', '192.168.192.169:9100', '192.168.192.170:9100', '192.168.192.171:9100']
# node exporter 가 필요하다
- job_name: 'Kafka_status'
static_configs:
- targets: ['192.168.192.172:9101', '192.168.192.173:9101', '192.168.192.174:9101']
# kafka exporter가 필요하다
#필요시 아래에 Job 추가
- job_name: 'JOB_NAME'
static_configs:
- targets: ['HOSTNAME:PORT','HOSTNAME:PORT','HOSTNAME:PORT','HOSTNAME:PORT']
groups:
- name: container memory alert
rules:
- alert: container memory usage rate is very high( > 55%)
expr: sum(container_memory_working_set_bytes{pod!="", name=""})/ sum (kube_node_status_allocatable_memory_bytes) * 100 > 55
for: 1m
labels:
severity: fatal
annotations:
summary: High Memory Usage on {{ $labels.instance }}
identifier: "{{ $labels.instance }}"
description: "{{ $labels.job }} Memory Usage: {{ $value }}"
- name: container CPU alert
rules:
- alert: container CPU usage rate is very high( > 10%)
expr: sum (rate (container_cpu_usage_seconds_total{pod!=""}[1m])) / sum (machine_cpu_cores) * 100 > 10
for: 1m
labels:
severity: fatal
annotations:
summary: High Cpu Usage
# rule 의 경우 grafana 등에서 Query를 입력 하는 것으로 정의할 수 있다.
Prometheus 설정은 global 설정과 scrape_configs 와 rule 로 이루어져 있다. 각각의 파일을 만들어 로드해도 되고, 하나의 파일에 내용을 기재해도 된다.
./promtool check config /etc/prometheus/prometheus.yaml
curl -s http://localhost:9090/-/ready
curl -s 'http://localhost:9090/api/v1/targets?state=active' | python3 -m json.tool | grep -E 'job|health'
최신 버전 3.14.0 · LTS 3.13.3 — 2026-09-20 확인. https://prometheus.io/download/ ↩︎