수평형 포드 자동 확장에 대한 사용자 정의 커스텀 측정항목 사용 설정

이 주제에서는 Google Distributed Cloud에서 수평형 포드 자동 확장 처리(HPA)에 대한 사용자 정의 측정항목을 구성하는 방법을 설명합니다.

Prometheus 및 측정항목 어댑터 배포

이 섹션에서는 Prometheus를 배포하여 사용자 정의 측정항목을 스크래핑하고, prometheus-adapter를 배포하여 Prometheus를 백엔드로 사용하는 Kubernetes Custom Metrics API를 처리합니다.

다음 매니페스트를 custom-metrics-adapter.yaml이라는 파일에 저장합니다.

Prometheus 및 측정항목 어댑터용 매니페스트 파일 콘텐츠

# Copyright 2018 Google Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: stackdriver-prometheus
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: stackdriver-prometheus
  namespace: kube-system
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: stackdriver-prometheus
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: stackdriver-prometheus
subjects:
- kind: ServiceAccount
  name: stackdriver-prometheus
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  name: stackdriver-prometheus-app
  namespace: kube-system
  labels:
    app: stackdriver-prometheus-app
spec:
  clusterIP: "None"
  ports:
    - name: http
      port: 9090
      protocol: TCP
      targetPort: 9090
  sessionAffinity: ClientIP
  selector:
    app: stackdriver-prometheus-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stackdriver-prometheus-app
  namespace: kube-system
  labels:
    app: stackdriver-prometheus-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: stackdriver-prometheus-app
  template:
    metadata:
      labels:
        app: stackdriver-prometheus-app
    spec:
      serviceAccount: stackdriver-prometheus
      containers:
      - name: prometheus-server
        image: prom/prometheus:v2.45.0
        args:
        - "--config.file=/etc/prometheus/config/prometheus.yaml"
        - "--storage.tsdb.path=/data"
        - "--storage.tsdb.retention.time=2h"
        ports:
        - name: prometheus
          containerPort: 9090
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 9090
          periodSeconds: 5
          timeoutSeconds: 3
          # Allow up to 10m on startup for data recovery
          failureThreshold: 120
        livenessProbe:
          httpGet:
            path: /-/healthy
            port: 9090
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 6
        resources:
          requests:
            cpu: 250m
            memory: 500Mi
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus/config
        - name: stackdriver-prometheus-app-data
          mountPath: /data
      volumes:
      - name: config-volume
        configMap:
          name: stackdriver-prometheus-app
      - name: stackdriver-prometheus-app-data
        emptyDir: {}
      terminationGracePeriodSeconds: 300
      nodeSelector:
        kubernetes.io/os: linux
---
apiVersion: v1
data:
  prometheus.yaml: |
    global:
      scrape_interval: 1m
    rule_files:
    - /etc/config/rules.yaml
    - /etc/config/alerts.yaml
    scrape_configs:
    - job_name: prometheus-io-endpoints
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scrape
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_scheme
        target_label: __scheme__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_service_annotation_prometheus_io_port
        target_label: __address__
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: keep
        regex: (.+)
        source_labels:
        - __meta_kubernetes_endpoint_port_name
    - job_name: prometheus-io-services
      kubernetes_sd_configs:
      - role: service
      metrics_path: /probe
      params:
        module:
        - http_2xx
      relabel_configs:
      - action: replace
        source_labels:
        - __address__
        target_label: __param_target
      - action: replace
        replacement: blackbox
        target_label: __address__
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_service_annotation_prometheus_io_probe
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
    - job_name: prometheus-io-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - action: replace
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_path
        target_label: __metrics_path__
      - action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels:
        - __address__
        - __meta_kubernetes_pod_annotation_prometheus_io_port
        target_label: __address__
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
kind: ConfigMap
metadata:
  name: stackdriver-prometheus-app
  namespace: kube-system
---

# The main section of custom metrics adapter.
kind: ServiceAccount
apiVersion: v1
metadata:
  name: custom-metrics-apiserver
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-server-resources
rules:
- apiGroups:
  - custom.metrics.k8s.io
  resources: ["*"]
  verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: custom-metrics-resource-reader
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - pods
  - services
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: custom-metrics-resource-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-resource-reader
subjects:
- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: custom-metrics-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: custom-metrics-apiserver
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: kube-system
data:
  config.yaml: |
    rules:
    default: false
      # fliter all metrics
    - seriesQuery: '{pod=~".+"}'
      seriesFilters: []
      resources:
        # resource name is mapped as it is. ex. namespace -> namespace
        template: <<.Resource>>
      name:
        matches: ^(.*)$
        as: ""
      # Aggregate metric on resource level
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: custom-metrics-apiserver
  name: custom-metrics-apiserver
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: custom-metrics-apiserver
  template:
    metadata:
      labels:
        app: custom-metrics-apiserver
      name: custom-metrics-apiserver
    spec:
      serviceAccountName: custom-metrics-apiserver
      containers:
      - name: custom-metrics-apiserver
        resources:
          requests:
            cpu: 15m
            memory: 20Mi
          limits:
            cpu: 100m
            memory: 150Mi
        image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.0
        args:
        - /adapter
        - --cert-dir=/var/run/serving-cert
        - --secure-port=6443
        - --prometheus-url=http://stackdriver-prometheus-app.kube-system.svc:9090/
        - --metrics-relist-interval=1m
        - --config=/etc/adapter/config.yaml
        ports:
        - containerPort: 6443
        volumeMounts:
        - name: serving-cert
          mountPath: /var/run/serving-cert
        - mountPath: /etc/adapter/
          name: config
          readOnly: true
      nodeSelector:
        kubernetes.io/os: linux
      volumes:
      - name: serving-cert
        emptyDir:
          medium: Memory
      - name: config
        configMap:
          name: adapter-config
---
apiVersion: v1
kind: Service
metadata:
  name: custom-metrics-apiserver
  namespace: kube-system
spec:
  ports:
  - port: 443
    targetPort: 6443
  selector:
    app: custom-metrics-apiserver
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta1.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: kube-system
  group: custom.metrics.k8s.io
  version: v1beta1
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  name: v1beta2.custom.metrics.k8s.io
spec:
  service:
    name: custom-metrics-apiserver
    namespace: kube-system
  group: custom.metrics.k8s.io
  version: v1beta2
  insecureSkipTLSVerify: true
  groupPriorityMinimum: 100
  versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: hpa-controller-custom-metrics
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: custom-metrics-server-resources
subjects:
- kind: ServiceAccount
  name: horizontal-pod-autoscaler
  namespace: kube-system

배포와 서비스를 만듭니다.

kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f custom-metrics-adapter.yaml

다음 단계는 측정항목 수집을 위해 사용자 애플리케이션에 주석을 추가하는 것입니다.

측정항목 수집을 위해 사용자 애플리케이션에 주석 추가

스크레이핑할 사용자 애플리케이션과 Cloud Monitoring으로 전송된 로그에 주석을 추가하려면 해당 annotations를 서비스, pod, 엔드포인트의 메타데이터에 추가해야 합니다.

  metadata:
    name: "example-monitoring"
    namespace: "default"
    annotations:
      prometheus.io/scrape: "true"
      prometheus.io/path: "" - Overriding metrics path (default "/metrics")
  

예시 사용자 애플리케이션 배포

이 섹션에서는 로그 및 Prometheus 호환 측정항목과 함께 샘플 애플리케이션을 배포합니다.

  1. 다음 서비스 및 배포 매니페스트를 my-app.yaml 파일에 저장합니다. 서비스에는 prometheus.io/scrape: "true" 주석이 있습니다.

    kind: Service
    apiVersion: v1
    metadata:
      name: "example-monitoring"
      namespace: "default"
      annotations:
        prometheus.io/scrape: "true"
    spec:
      selector:
        app: "example-monitoring"
      ports:
        - name: http
          port: 9090
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: "example-monitoring"
      namespace: "default"
      labels:
        app: "example-monitoring"
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: "example-monitoring"
      template:
        metadata:
          labels:
            app: "example-monitoring"
        spec:
          containers:
          - image: gcr.io/google-samples/prometheus-dummy-exporter:v0.2.0
            name: prometheus-example-exporter
            command:
            - ./prometheus-dummy-exporter
            args:
            - --metric-name=example_monitoring_up
            - --metric-value=1
            - --port=9090
            resources:
              requests:
                cpu: 100m
    
  2. 배포와 서비스를 만듭니다.

    kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml
    

HPA에서 커스텀 측정항목 사용

HPA 객체를 배포하여 이전 단계에서 노출된 측정항목을 사용합니다. 다양한 유형의 커스텀 측정항목에 대한 자세한 내용은 여러 측정항목 및 커스텀 측정항목에 대한 자동 확장을 참조하세요.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: example-monitoring-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-monitoring
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: Pods
    pods:
      metric:
        name: example_monitoring_up
      target:
        type: AverageValue
        averageValue: 20

포드 유형 측정항목에는 대상 포드의 라벨에 대한 기본 측정항목 선택자가 있으며, 이에 따라 kube-controller-manager가 작동합니다. 이 예시에서는 대상 pod에서 사용할 수 있는 {matchLabels: {app: example-monitoring}} 선택기로 example_monitoring_up 측정항목을 쿼리할 수 있습니다. 지정된 다른 선택기가 목록에 추가됩니다. 기본 선택기를 사용하지 않으려면 대상 pod에서 라벨을 삭제하거나 객체 유형 측정항목을 사용하면 됩니다.

사용자 정의 애플리케이션 측정항목이 HPA에서 사용되는지 확인

사용자 정의 애플리케이션 측정항목이 HPA에서 사용되는지 확인합니다.

kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG describe hpa example-monitoring-hpa

출력은 다음과 같습니다.

  Name:               example-monitoring-hpa
  Namespace:          default
  Labels:             
  Annotations:        autoscaling.alpha.kubernetes.io/conditions:
                        [{"type":"AbleToScale","status":"True","lastTransitionTime":"2023-08-23T22:07:24Z","reason":"ReadyForNewScale","message":"recommended size...
                      autoscaling.alpha.kubernetes.io/current-metrics: [{"type":"Pods","pods":{"metricName":"example_monitoring_up","currentAverageValue":"1"}}]
                      autoscaling.alpha.kubernetes.io/metrics: [{"type":"Pods","pods":{"metricName":"example_monitoring_up","targetAverageValue":"20"}}]
  CreationTimestamp:  Wed, 23 Aug 2023 22:07:09 +0000
  Reference:          Deployment/example-monitoring
  Min replicas:       1
  Max replicas:       5
  Deployment pods:    1 current / 1 desired
  

비용

HPA의 커스텀 측정항목을 사용해도 추가 Cloud Monitoring 비용이 발생하지 않습니다. 커스텀 측정항목을 사용 설정하는 포드는 스크래핑하는 측정항목의 양에 따라 추가 CPU 및 메모리를 사용합니다.