[쿠버네티스 DevOps 구축] - OpenSearch 설치하기

목차

• [쿠버네티스 DevOps 구축] - OpenSearch 설치하기
• [쿠버네티스 DevOps 구축] - Prometheus 와 Grafana 설치하기
• [쿠버네티스 DevOps 구축] - Keycloak 설치하기
• [쿠버네티스 DevOps 구축] - Ingress Nginx Controller 설치하기
• [쿠버네티스 DevOps 구축] - Local PC 쿠버네티스 동적 프로비저닝을 위한 StorageClass 설치
• [쿠버네티스 DevOps 구축] - Local PC 에 쿠버네티스 설치하기

참고

• https://blog.pages.kr/2844
• https://opendistro.github.io/for-elasticsearch-docs/docs/security/access-control/api/#reserved-and-hidden-resources
• https://forum.opensearch.org/t/default-password-reset/102/10
• Admin Password 변경
  • https://github.com/opendistro-for-elasticsearch/opendistro-build/issues/558
  • https://opendistro.github.io/for-elasticsearch-docs/docs/security/configuration/yaml/#internal_usersyml
  • https://github.com/opensearch-project/helm-charts/issues/161

✅ OpenSearch 설치

# Helm Repo 추가
helm repo add opensearch https://opensearch-project.github.io/helm-charts/
# OpenSearch 설치
helm upgrade --install opensearch \
  -f ./values-opensearch.yaml opensearch/opensearch \
  -n logging --create-namespace

Release "opensearch" has been upgraded. Happy Helming!
NAME: opensearch
LAST DEPLOYED: Thu Jan 18 15:55:28 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
Watch all cluster members come up.
  $ kubectl get pods --namespace=logging -l app.kubernetes.io/component=opensearch-cluster-master -w

✅ OpenSearch Dashboard 설치

helm upgrade --install opensearch-dashboard \
  -f ./values-opensearch-dashboards.yaml opensearch/opensearch-dashboards \
  -n logging --create-namespace

Release "opensearch" has been upgraded. Happy Helming!
NAME: opensearch
LAST DEPLOYED: Thu Feb  6 23:13:09 2025
NAMESPACE: logging
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace logging -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch" -o jsonpath="{.items[0].metadata.name}")
  export CONTAINER_PORT=$(kubectl get pod --namespace logging $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl --namespace logging port-forward $POD_NAME 8080:$CONTAINER_PORT

OpenSearch Demo User

• https://docs.search-guard.com/latest/demo-users-roles

Admin 비밀번호 변경

필요한 명령어 및 파일 위치

/usr/share/elasticsearch/plugins/opendistro_security/tools/hash.sh
/usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh

securityConfig:
  enabled: true
  config:
    securityConfigSecret: ""
    dataComplete: true
    data:
      internal_users.yml: |-
        _meta:
          type: "internalusers"
          config_version: 2
        admin:
          hash: "YWtxanF0azFxMnczZTRyIUA="
          reserved: true
        backend_roles:
        - "admin"
        description: "Demo admin user"

문제 - Not yet initialized (you may need to run securityadmin)

[2023-12-30T15:22:45,111][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:45,115][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:45,118][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:45,120][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:47,614][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:47,617][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:47,620][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)
[2023-12-30T15:22:47,623][ERROR][o.o.s.a.BackendRegistry  ] [opensearch-cluster-master-0] Not yet initialized (you may need to run securityadmin)

kubectl -n logging exec -it opensearch-cluster-master-0 -- /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh

• pod 내에서

/usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
  -cd /usr/share/opensearch/config/opensearch-security/ \
  -icl \
  -nhnv \
  -cacert /usr/share/opensearch/config/root-ca.pem \
  -cert /usr/share/opensearch/config/kirk.pem \
  -key /usr/share/opensearch/config/kirk-key.pem

• 쿠버네티스 명령어로 실행

kubectl exec \
    -n logging \
    -it opensearch-cluster-master-0 \
    -- /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -cd /usr/share/opensearch/config/opensearch-security/ \
    -icl \
    -nhnv \
    -cacert /usr/share/opensearch/config/root-ca.pem \
    -cert /usr/share/opensearch/config/kirk.pem \
    -key /usr/share/opensearch/config/kirk-key.pem

securityConfig.internalUsersSecret 을 이용한 Admin 비밀번호 변경

비밀번호 Hash 값 생성

kubectl -n logging exec -it opensearch-cluster-master-0 -- /usr/share/opensearch/plugins/opensearch-security/tools/hash.sh

Secret 파일 생성

apiVersion: v1
kind: Secret
metadata:
  name: opensearch-internal-users
  namespace: logging
type: Opaque
stringData:
  internal_users.yml: |
    ---
    # This is the internal user database
    # The hash value is a bcrypt hash and can be generated using:
    # $ plugins/opensearch-security/tools/hash.sh -p <new-password>
    
    _meta:
      type: "internalusers"
      config_version: 2
    
    # Define your internal users here
    
    ## internal users
    admin:
      hash: "$2y$12$yGwPWlea1ghTu9NFl00h6.SqRJY1G.MuFyX8PTQnwUnYaGgUsFKBu"
      reserved: true
      backend_roles:
      - "admin"
      description: "admin user"

securityConfig.internalUsersSecret 에 Secret 이름 설정

securityConfig:
  enabled: true
  path: "/usr/share/opensearch/config/opensearch-security"
  actionGroupsSecret:
  configSecret:
  internalUsersSecret: opensearch-internal-users

✅ OpenSearch Dashboards basePath 및 Ingress 설정

config:
  opensearch_dashboards.yml: |
    server:
      host: '0.0.0.0'
      basePath: "/logging"
      rewriteBasePath: true
    opensearch:
      hosts: [https://localhost:9200]
      ssl.verificationMode: none
      username: kibanaserver
      password: kibanaserver
      requestHeadersWhitelist: [authorization, securitytenant]

    opensearch_security.multitenancy:
      enabled: true
    opensearch_security.multitenancy.tenants:
      preferred: [Private, Global]
    opensearch_security:
      readonly_mode.roles: [kibana_read_only]
      cookie.secure: false

Ingress 설정

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: logging
  namespace: logging
  annotations:
      kubernetes.io/ingress.class: nginx
spec:
  rules:
    - http:
        paths:
        - path: /logging
          pathType: Prefix
          backend:
            service:
              name: opensearch-dashboards
              port:
                number: 5601

✅ KeyCloak OpenSearch 연동

• 참고
  • https://velog.io/@ziggy/Keyclock-%EA%B5%AC%EC%B6%95-%EB%B0%8F-Opensearch-%EC%97%B0%EB%8F%99
  • https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/
  • https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/#opensearch-security-configuration
  • https://opensearch.org/docs/latest/security/authentication-backends/openid-connect/#opensearch-dashboards-single-sign-on
  • https://forum.opensearch.org/t/opensearch-integration-with-keycloak-using-k8s-opster-operator/15388/5

1. OpenSearch Client 생성

2. KeyCloak OpenSearch User Client Role 생성

3. OpenSearch 연동 및 설정파일 생성

3.1 KeyClock Connect URL 확인

3.2 OpenSearch Client Credential 확인

3.3 config.yml 생성

• https://forum.opensearch.org/t/opensearch-saml-ad-working-but-with-no-basic-authorization-header-warning/13883/2
• https://forum.opensearch.org/t/openid-401-unauthorized-with-keycloak/10364/2

apiVersion: v1
kind: Secret
metadata:
  name: opensearch-openid-keycloak
  namespace: logging
type: Opaque
stringData:
  config.yml: |-
    _meta:
      type: "config"
      config_version: 2
    config:
      dynamic:
        authz: {}
        authc:
          openid_auth_domain:
            http_enabled: true
            transport_enabled: true
            order: 0
            http_authenticator:
              type: openid
              challenge: false
              config:
                subject_key: preferred_username
                roles_key: roles
                openid_connect_url: http://auth.crunchychoco.com/realms/ddanzit/.well-known/openid-configuration
                skip_users:
                  - kibanaserver
                  - admin
            authentication_backend:
              type: noop
          basic_internal_auth_domain:
            http_enabled: true
            transport_enabled: true
            order: 1
            http_authenticator:
              type: basic
              challenge: false
            authentication_backend:
              type: internal

config.yml 을 통해 인증 방식과 순서를 설정하면 dashboard 에서 설정된 인증 방식을 확인할 수 있습니다.

3.4 Opensearch values.yaml 수정

securityConfig:
  enabled: true
  configSecret: opensearch-openid-keycloak

3.5 opensearch_dashboards.yml 수정

server:
    host: '0.0.0.0'
opensearch:
    hosts: [https://opensearch-cluster-master:9200]
    ssl.verificationMode: none
    username: kibanaserver
    password: kibanaserver
    requestHeadersWhitelist: [authorization, securitytenant]

opensearch_security.multitenancy:
    enabled: true
opensearch_security.multitenancy.tenants:
    preferred: [Private, Global]
opensearch_security:
    readonly_mode.roles: [kibana_read_only]
    cookie.secure: false

opensearch_security.auth.type: ["openid"]
opensearch_security.openid.connect_url: "http://auth.crunchychoco.com/realms/ddanzit/.well-known/openid-configuration"
opensearch_security.openid.client_id: "opensearch"
opensearch_security.openid.client_secret: "9EH2nHCcpKk05F8l8TUsqnEslA26h2KU"
opensearch_security.openid.base_redirect_url: "http://logging.crunchychoco.com"

3.6 OpenSearch SSO 및 Basic Auth 사용 설정

opensearch_security.auth.type: ["basicauth","openid"]
opensearch_security.auth.multiple_auth_enabled: true

3.7 skeycloak 인증 테스트

export CLIENT_ID=opensearch
export CLIENT_SECRET='9EH2nHCcpKk05F8l8TUsqnEslA26h2KU'
export PW='1q2w3e4r!@#'

RESULT=$(curl -k --noproxy '*' -d 'username=ddanzit' -d "password=$PW" -d 'grant_type=password' -d "client_id=$CLIENT_ID" -d "client_secret=$CLIENT_SECRET" -d 'scope=openid' 'http://auth.crunchychoco.com/realms/ddanzit/protocol/openid-connect/token')

TOKEN=$(echo $RESULT | sed 's/.*access_token":"\([^"]*\).*/\1/')

curl -H "Authorization: Bearer $TOKEN" http://auth.crunchychoco.com/realms/ddanzit/protocol/openid-connect/userinfo
{
    "sub": "ce6f75ba-26fb-4cf4-9e7d-06f20d7d749f",
    "email_verified": true,
    "name": "동우 양",
    "preferred_username": "kwangill",
    "given_name": "동우",
    "family_name": "양",
    "email": "askain@hanmail.net"
}

3.8 keycloak Json Web Token(JWT) 내용 확인

{
  "exp": 1705381539,
  "iat": 1705381239,
  "jti": "e6cf69c7-f674-4a10-bfbc-444d3c19ad25",
  "iss": "http://auth.crunchychoco.com/realms/ddanzit",
  "aud": "account",
  "sub": "4eac345b-bfb9-4c4f-aba3-d52af277bbf0",
  "typ": "Bearer",
  "azp": "opensearch",
  "session_state": "c7c47cf3-34bf-40d5-891a-6d5ae52cc855",
  "acr": "1",
  "allowed-origins": [
    ""
  ],
  "realm_access": {
    "roles": [
      "default-roles-ddanzit",
      "offline_access",
      "admin",
      "uma_authorization",
      "Admin"
    ]
  },
  "resource_access": {
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "openid email profile",
  "sid": "c7c47cf3-34bf-40d5-891a-6d5ae52cc855",
  "email_verified": false,
  "name": "동우 양",
  "preferred_username": "ddanzit",
  "given_name": "동우",
  "family_name": "양",
  "email": "ckck803@naver.com"
}

No ‘Basic Authorization’ header, send 401 and ‘WWW-Authenticate Basic’

• https://forum.opensearch.org/t/opensearch-dashboards-getting-no-basic-authorization-header-send-401-and-www-authenticate-basic-when-trying-to-login-via-sso/16101/7
• https://forum.search-guard.com/t/no-basic-authorization-header-send-401-and-www-authenticate-basic-while-using-openid-and-basic-auth-domain/1930
• https://opensearch.org/docs/latest/security/configuration/security-admin/

위와 같은 에러가 발생한다면 securityadmin.sh 를 이용하여 설정파일을 적용한다.

kubectl exec \
    -n logging \
    -it opensearch-cluster-master-0 \
    -- /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh \
    -cd /usr/share/opensearch/config/opensearch-security/ \
    -icl \
    -nhnv \
    -cacert /usr/share/opensearch/config/root-ca.pem \
    -cert /usr/share/opensearch/config/kirk.pem \
    -key /usr/share/opensearch/config/kirk-key.pem

FATAL Error: Multiple Authentication Mode is disabled. To enable this feature, please set up opensearch_security.auth.multiple_auth_enabled: true

opensearch_security.auth.multiple_auth_enabled: true