このトピックでは、Google Distributed Cloud で水平 Pod 自動スケーリング（HPA）のユーザー定義指標を構成する方法について説明します。
Prometheus と Metrics Adapter をデプロイする
このセクションでは、Prometheus をデプロイしてユーザー定義の指標を取得し、prometheus-adapter をデプロイして Prometheus をバックエンドとして Kubernetes カスタム指標 API を実行します。
次のマニフェストを custom-metrics-adapter.yaml という名前のファイルに保存します。
Prometheus および Metrics Adapter のマニフェスト ファイルの内容
# Copyright 2018 Google Inc
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: v1
kind: ServiceAccount
metadata:
name: stackdriver-prometheus
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: stackdriver-prometheus
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- nodes
- services
- endpoints
- pods
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: stackdriver-prometheus
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: stackdriver-prometheus
subjects:
- kind: ServiceAccount
name: stackdriver-prometheus
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
name: stackdriver-prometheus-app
namespace: kube-system
labels:
app: stackdriver-prometheus-app
spec:
clusterIP: "None"
ports:
- name: http
port: 9090
protocol: TCP
targetPort: 9090
sessionAffinity: ClientIP
selector:
app: stackdriver-prometheus-app
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: stackdriver-prometheus-app
namespace: kube-system
labels:
app: stackdriver-prometheus-app
spec:
replicas: 1
selector:
matchLabels:
app: stackdriver-prometheus-app
template:
metadata:
labels:
app: stackdriver-prometheus-app
spec:
serviceAccount: stackdriver-prometheus
containers:
- name: prometheus-server
image: prom/prometheus:v2.45.0
args:
- "--config.file=/etc/prometheus/config/prometheus.yaml"
- "--storage.tsdb.path=/data"
- "--storage.tsdb.retention.time=2h"
ports:
- name: prometheus
containerPort: 9090
readinessProbe:
httpGet:
path: /-/ready
port: 9090
periodSeconds: 5
timeoutSeconds: 3
# Allow up to 10m on startup for data recovery
failureThreshold: 120
livenessProbe:
httpGet:
path: /-/healthy
port: 9090
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 6
resources:
requests:
cpu: 250m
memory: 500Mi
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus/config
- name: stackdriver-prometheus-app-data
mountPath: /data
volumes:
- name: config-volume
configMap:
name: stackdriver-prometheus-app
- name: stackdriver-prometheus-app-data
emptyDir: {}
terminationGracePeriodSeconds: 300
nodeSelector:
kubernetes.io/os: linux
---
apiVersion: v1
data:
prometheus.yaml: |
global:
scrape_interval: 1m
rule_files:
- /etc/config/rules.yaml
- /etc/config/alerts.yaml
scrape_configs:
- job_name: prometheus-io-endpoints
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scrape
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: (https?)
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_scheme
target_label: __scheme__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_service_annotation_prometheus_io_port
target_label: __address__
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: keep
regex: (.+)
source_labels:
- __meta_kubernetes_endpoint_port_name
- job_name: prometheus-io-services
kubernetes_sd_configs:
- role: service
metrics_path: /probe
params:
module:
- http_2xx
relabel_configs:
- action: replace
source_labels:
- __address__
target_label: __param_target
- action: replace
replacement: blackbox
target_label: __address__
- action: keep
regex: true
source_labels:
- __meta_kubernetes_service_annotation_prometheus_io_probe
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- job_name: prometheus-io-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: true
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_scrape
- action: replace
regex: (.+)
source_labels:
- __meta_kubernetes_pod_annotation_prometheus_io_path
target_label: __metrics_path__
- action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
source_labels:
- __address__
- __meta_kubernetes_pod_annotation_prometheus_io_port
target_label: __address__
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
kind: ConfigMap
metadata:
name: stackdriver-prometheus-app
namespace: kube-system
---
# The main section of custom metrics adapter.
kind: ServiceAccount
apiVersion: v1
metadata:
name: custom-metrics-apiserver
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: custom-metrics:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: custom-metrics-apiserver
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: custom-metrics-server-resources
rules:
- apiGroups:
- custom.metrics.k8s.io
resources: ["*"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: custom-metrics-resource-reader
rules:
- apiGroups:
- ""
resources:
- nodes
- namespaces
- pods
- services
verbs:
- get
- watch
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: custom-metrics-resource-reader
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: custom-metrics-resource-reader
subjects:
- kind: ServiceAccount
name: custom-metrics-apiserver
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: custom-metrics-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: custom-metrics-apiserver
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: adapter-config
namespace: kube-system
data:
config.yaml: |
rules:
default: false
# fliter all metrics
- seriesQuery: '{pod=~".+"}'
seriesFilters: []
resources:
# resource name is mapped as it is. ex. namespace -> namespace
template: <<.Resource>>
name:
matches: ^(.*)$
as: ""
# Aggregate metric on resource level
metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: custom-metrics-apiserver
template:
metadata:
labels:
app: custom-metrics-apiserver
name: custom-metrics-apiserver
spec:
serviceAccountName: custom-metrics-apiserver
containers:
- name: custom-metrics-apiserver
resources:
requests:
cpu: 15m
memory: 20Mi
limits:
cpu: 100m
memory: 150Mi
image: registry.k8s.io/prometheus-adapter/prometheus-adapter:v0.11.0
args:
- /adapter
- --cert-dir=/var/run/serving-cert
- --secure-port=6443
- --prometheus-url=http://stackdriver-prometheus-app.kube-system.svc:9090/
- --metrics-relist-interval=1m
- --config=/etc/adapter/config.yaml
ports:
- containerPort: 6443
volumeMounts:
- name: serving-cert
mountPath: /var/run/serving-cert
- mountPath: /etc/adapter/
name: config
readOnly: true
nodeSelector:
kubernetes.io/os: linux
volumes:
- name: serving-cert
emptyDir:
medium: Memory
- name: config
configMap:
name: adapter-config
---
apiVersion: v1
kind: Service
metadata:
name: custom-metrics-apiserver
namespace: kube-system
spec:
ports:
- port: 443
targetPort: 6443
selector:
app: custom-metrics-apiserver
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta1.custom.metrics.k8s.io
spec:
service:
name: custom-metrics-apiserver
namespace: kube-system
group: custom.metrics.k8s.io
version: v1beta1
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
name: v1beta2.custom.metrics.k8s.io
spec:
service:
name: custom-metrics-apiserver
namespace: kube-system
group: custom.metrics.k8s.io
version: v1beta2
insecureSkipTLSVerify: true
groupPriorityMinimum: 100
versionPriority: 100
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: hpa-controller-custom-metrics
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: custom-metrics-server-resources
subjects:
- kind: ServiceAccount
name: horizontal-pod-autoscaler
namespace: kube-system
Deployment と Service を作成します。
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f custom-metrics-adapter.yaml
次の手順では、指標の収集について、ユーザー アプリケーションにアノテーションを付けます。
指標を収集するユーザー アプリケーションにアノテーションを付ける
収集対象のユーザー アプリケーションと Cloud Monitoring に送信されるログにアノテーションを付けるには、対応する annotations をサービス、Pod、エンドポイントのメタデータに追加する必要があります。
metadata:
name: "example-monitoring"
namespace: "default"
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "" - Overriding metrics path (default "/metrics")
サンプルのユーザー アプリケーションをデプロイする
このセクションでは、ログと Prometheus 互換の指標の両方を持つサンプル アプリケーションをデプロイします。
次の Service マニフェストと Deployment マニフェストを
my-app.yamlという名前のファイルに保存します。Service にprometheus.io/scrape: "true"というアノテーションが付いていることに注意してください。kind: Service apiVersion: v1 metadata: name: "example-monitoring" namespace: "default" annotations: prometheus.io/scrape: "true" spec: selector: app: "example-monitoring" ports: - name: http port: 9090 --- apiVersion: apps/v1 kind: Deployment metadata: name: "example-monitoring" namespace: "default" labels: app: "example-monitoring" spec: replicas: 1 selector: matchLabels: app: "example-monitoring" template: metadata: labels: app: "example-monitoring" spec: containers: - image: gcr.io/google-samples/prometheus-dummy-exporter:v0.2.0 name: prometheus-example-exporter command: - ./prometheus-dummy-exporter args: - --metric-name=example_monitoring_up - --metric-value=1 - --port=9090 resources: requests: cpu: 100mDeployment と Service を作成します。
kubectl --kubeconfig USER_CLUSTER_KUBECONFIG apply -f my-app.yaml
HPA でカスタム指標を使用する
HPA オブジェクトをデプロイして、前のステップで公開された指標を使用します。さまざまな種類のカスタム指標の詳細については、複数の指標とカスタム指標の自動スケーリングをご覧ください。
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: example-monitoring-hpa
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: example-monitoring
minReplicas: 1
maxReplicas: 5
metrics:
- type: Pods
pods:
metric:
name: example_monitoring_up
target:
type: AverageValue
averageValue: 20
Pod タイプの指標には、ターゲット Pod のラベルのデフォルトの指標セレクタがあり、これが kube-controller-manager の機能です。この例では、ターゲット Pod で利用できる example_monitoring_up 指標を {matchLabels: {app: example-monitoring}} というセレクタでクエリします。指定したその他のセレクタは、リストに追加されます。デフォルト セレクタを使用しないようにするには、ターゲット Pod のラベルをすべて削除するか、オブジェクト タイプの指標を使用します。
ユーザー定義のアプリケーション指標を HPA が使用していることを確認する
ユーザー定義のアプリケーション指標を HPA が使用していることを確認します。
kubectl --kubeconfig=USER_CLUSTER_KUBECONFIG describe hpa example-monitoring-hpa
出力は次のようになります。
Name: example-monitoring-hpa Namespace: default Labels:Annotations: autoscaling.alpha.kubernetes.io/conditions: [{"type":"AbleToScale","status":"True","lastTransitionTime":"2023-08-23T22:07:24Z","reason":"ReadyForNewScale","message":"recommended size... autoscaling.alpha.kubernetes.io/current-metrics: [{"type":"Pods","pods":{"metricName":"example_monitoring_up","currentAverageValue":"1"}}] autoscaling.alpha.kubernetes.io/metrics: [{"type":"Pods","pods":{"metricName":"example_monitoring_up","targetAverageValue":"20"}}] CreationTimestamp: Wed, 23 Aug 2023 22:07:09 +0000 Reference: Deployment/example-monitoring Min replicas: 1 Max replicas: 5 Deployment pods: 1 current / 1 desired
費用
HPA にカスタム指標を使用しても、Cloud Monitoring の追加料金は発生しません。カスタム指標を有効にする Pod は、取得する指標の量に基づいて、追加の CPU とメモリを消費します。