Kubernetes

[쿠버네티스 DevOps 구축] - Prometheus 와 Grafana 설치하기

목차

✅ 메트릭 수집을 위한 Prometheus

서비스를 올렸다고 해서 끝이 아니다! 서비스를 운용하는 입장에서는 시스템이 안정적인지 지속적으로 확인을 해야 하는데, 운영하는 서비스의 CPU 사용률, 메모리 사용률과 같은 메트릭 정보를 지속적으로 수집하는 시스템을 구축해야 합니다.

저는 대중적으로 많이 알려진 Prometheus 를 이용해 쿠버네티스에서 운영중인 서비스에 대한 메트릭 정보를 수집하려고 합니다.

Helm 을 이용한 Prometheus 설치

# helm repo 추가
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# prometheus 설치
helm upgrade --install -f ./values-prometheus.yaml prometheus prometheus-community/prometheus -n monitoring --create-namespace
Release "prometheus" does not exist. Installing it now.
NAME: prometheus
LAST DEPLOYED: Fri Nov 15 10:11:47 2024
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=prometheus,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Server pod is terminated.                             #####
#################################################################################


The Prometheus alertmanager can be accessed via port 9093 on the following DNS name from within your cluster:
prometheus-alertmanager.monitoring.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=alertmanager,app.kubernetes.io/instance=prometheus" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the AlertManager pod is terminated.                       #####
#################################################################################
#################################################################################
######   WARNING: Pod Security Policy has been disabled by default since    #####
######            it deprecated after k8s 1.25+. use                        #####
######            (index .Values "prometheus-node-exporter" "rbac"          #####
###### .          "pspEnabled") with (index .Values                         #####
######            "prometheus-node-exporter" "rbac" "pspAnnotations")       #####
######            in case you still need it.                                #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-prometheus-pushgateway.monitoring.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus-pushgateway,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

✅ 모니터링을 위한 Grafana

메트릭 수집을 위한 Prometheus 를 설치 했는데! 단순 메트릭 정보만 있으면 운영하는 입장에서 가시적으로 와 닿지 않습니다.

이번에는 메트릭 정보를 사용자에게 보여주는 대시보드 툴인 Grafana 를 설치해 대시보드를 구성하려고 합니다.

Helm 을 이용한 Grafana 설치

# repo 추가
helm repo add grafana https://grafana.github.io/helm-charts
# grafana 설치
helm upgrade --install -f values-grafana.yaml grafana grafana/grafana -n monitoring --create-namespace
Release "grafana" does not exist. Installing it now.
NAME: grafana
LAST DEPLOYED: Fri Nov 15 11:22:42 2024
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo


2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.monitoring.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:
     export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace monitoring port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################

✅ Grafana KeyCloak 연동

1. KeyCloak - Grafana Client 생성

2. KeyCloak - Grafana User Client Role 추가

3. Grafana values.yaml 수정

server:
  root_url: http://monitoring.crunchychoco.com/
auth.generic_oauth:
  enabled: true
  name: ddanzit
  allow_sign_up: true
  client_id: grafana
  client_secret: FYFRju1uhbUnZeQq3BUz1n7G1cK7DSNc
  scopes: openid email profile offline_access roles
  email_attribute_path: email
  login_attribute_path: username
  name_attribute_path: full_name
  auth_url: http://auth.crunchychoco.com/realms/ddanzit/protocol/openid-connect/auth
  token_url: http://auth.crunchychoco.com/realms/ddanzit/protocol/openid-connect/token
  api_url: http://auth.crunchychoco.com/realms/ddanzit/protocol/openid-connect/userinfo
  role_attribute_path: contains(roles[*], 'admin') && 'Admin' || contains(roles[*], 'editor') && 'Editor' || 'Viewer'

Share