本頁面由 Cloud Translation API 翻譯而成。

部署 GKE 推論閘道

自動駕駛標準

本頁說明如何部署 GKE Inference Gateway。

本頁面適用於負責管理 GKE 基礎架構的網路專家，以及管理 AI 工作負載的平台管理員。

閱讀本頁面之前，請先熟悉下列概念：

關於 GKE Inference Gateway
GKE 的 AI/機器學習自動化調度管理機制。
生成式 AI 詞彙表。
負載平衡Google Cloud，特別是負載平衡器與 GKE 的互動方式。
GKE Service Extensions。詳情請參閱 GKE Gateway 控制器說明文件。
使用服務擴充功能自訂 GKE 閘道流量。

GKE Inference Gateway 可強化 Google Kubernetes Engine (GKE) Gateway，進而提升生成式 AI 應用程式的服務品質。GKE 推論閘道可讓您在 GKE 上，以最佳化方式提供生成式 AI 工作負載。可有效管理及擴充 AI 工作負載、達成工作負載專屬的效能目標 (例如延遲)，並提升資源使用率、可觀測性及 AI 安全性。

事前準備

開始之前，請確認你已完成下列工作：

啟用 Google Kubernetes Engine API。

啟用 Google Kubernetes Engine API

如要使用 Google Cloud CLI 執行這項工作，請安裝並初始化 gcloud CLI。如果您先前已安裝 gcloud CLI，請執行 gcloud components update，取得最新版本。
注意：如果是現有的 gcloud CLI 安裝項目，請務必設定 compute/region 和 compute/zone 屬性。設定預設位置後，即可避免 gcloud CLI 發生下列錯誤： One of [--zone, --region] must be supplied: Please specify location.

視需要啟用 Compute Engine API、Network Services API 和 Model Armor API。

前往「啟用 API 存取權」，然後按照操作說明進行。

GKE Gateway 控制器需求

GKE 1.32.3 版。
Google Cloud CLI 407.0.0 以上版本。
閘道 API 僅支援虛擬私有雲原生叢集。
您必須啟用僅限 Proxy 的子網路。
叢集必須啟用 HttpLoadBalancing 外掛程式。
如果您使用 Istio，請務必將 Istio 升級至下列其中一個版本：
- 1.15.2 以上版本
- 1.14.5 以上版本
- 1.13.9 以上版本
如果您使用共用虛擬私有雲，則必須在主機專案中，將 Compute Network User 角色指派給服務專案的 GKE 服務帳戶。

規定與限制

請注意下列限制：

不支援多叢集閘道。
GKE Inference Gateway 僅支援 gke-l7-regional-external-managed 和 gke-l7-rilb GatewayClass 資源。
不支援跨區域內部應用程式負載平衡器。

設定 GKE Inference Gateway

如要設定 GKE Inference Gateway，請參考這個範例。團隊會執行 vLLM 和 Llama3 模型，並積極實驗兩種不同的 LoRA 微調轉接程式：「food-review」和「cad-fabricator」。

設定 GKE Inference Gateway 的大致工作流程如下：

準備環境：設定必要的基礎架構和元件。
建立推論集區：使用 InferencePool 自訂資源定義模型伺服器集區。
指定模型服務目標：使用 InferenceModel 自訂資源指定模型目標。
建立 Gateway：使用 Gateway API 公開推論服務。
建立 HTTPRoute：定義 HTTP 流量如何路由至推論服務。
傳送推論要求：向已部署的模型提出要求。

準備環境

安裝 Helm。
建立 GKE 叢集：
- 建立 GKE Autopilot 或 Standard 叢集，版本為 1.31 以上。如需操作說明，請參閱建立 GKE 叢集。
- 使用偏好的運算系列和加速器設定節點。
- 根據您選取的加速器、模型和效能需求，使用透過 GKE Inference 快速入門食譜執行最佳做法推論，取得預先設定及測試的部署資訊清單。
如要在 GKE 叢集中安裝 InferencePool 和 InferenceModel 自訂資源定義 (CRD)，請執行下列指令：
```
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/releases/download/v0.3.0/manifests.yaml
```
將 VERSION 替換成要安裝的 CRD 版本 (例如 v0.3.0)。

如果您使用的 GKE 版本早於 v1.32.2-gke.1182001，且想搭配 GKE Inference Gateway 使用 Model Armor，請務必安裝流量和路徑擴充功能 CRD：

kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-gateway-api/refs/heads/main/config/crd/networking.gke.io_gcptrafficextensions.yaml
kubectl apply -f https://raw.githubusercontent.com/GoogleCloudPlatform/gke-gateway-api/refs/heads/main/config/crd/networking.gke.io_gcproutingextensions.yaml

如要設定授權來擷取指標，請建立 inference-gateway-sa-metrics-reader-secret 密鑰：

kubectl apply -f - <<EOF
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: inference-gateway-metrics-reader
rules:
- nonResourceURLs:
  - /metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: inference-gateway-sa-metrics-reader
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: inference-gateway-sa-metrics-reader-role-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: inference-gateway-sa-metrics-reader
  namespace: default
roleRef:
  kind: ClusterRole
  name: inference-gateway-metrics-reader
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
  name: inference-gateway-sa-metrics-reader-secret
  namespace: default
  annotations:
    kubernetes.io/service-account.name: inference-gateway-sa-metrics-reader
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: inference-gateway-sa-metrics-reader-secret-read
rules:
- resources:
  - secrets
  apiGroups: [""]
  verbs: ["get", "list", "watch"]
  resourceNames: ["inference-gateway-sa-metrics-reader-secret"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: gmp-system:collector:inference-gateway-sa-metrics-reader-secret-read
  namespace: default
roleRef:
  name: inference-gateway-sa-metrics-reader-secret-read
  kind: ClusterRole
  apiGroup: rbac.authorization.k8s.io
subjects:
- name: collector
  namespace: gmp-system
  kind: ServiceAccount
EOF

建立模型伺服器和模型部署作業

本節說明如何部署模型伺服器和模型。這個範例使用 vLLM 模型伺服器和 Llama3 模型。部署作業會標示為 app:vllm-llama3-8b-instruct。這項部署作業也會使用來自 Hugging Face 的兩個 LoRA 適應器，分別命名為 food-review 和 cad-fabricator。

您可以根據自己的模型伺服器容器和模型、服務通訊埠和部署名稱，調整這個範例。您也可以在部署中設定 LoRA 配接器，或部署基礎模型。下列步驟說明如何建立必要的 Kubernetes 資源。

建立 Kubernetes Secret 來儲存 Hugging Face 權杖。這個權杖用於存取 LoRA 適應器：
```
kubectl create secret generic hf-token --from-literal=token=HF_TOKEN
```
將 HF_TOKEN 替換為您的 Hugging Face 權杖。

如要在 nvidia-h100-80gb 加速器類型上部署，請將下列資訊清單儲存為 vllm-llama3-8b-instruct.yaml。這個資訊清單會定義含有模型和模型伺服器的 Kubernetes 部署：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b-instruct
spec:
  replicas: 3
  selector:
    matchLabels:
      app: vllm-llama3-8b-instruct
  template:
    metadata:
      labels:
        app: vllm-llama3-8b-instruct
    spec:
      containers:
        - name: vllm
          image: "vllm/vllm-openai:latest"
          imagePullPolicy: Always
          command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
          args:
          - "--model"
          - "meta-llama/Llama-3.1-8B-Instruct"
          - "--tensor-parallel-size"
          - "1"
          - "--port"
          - "8000"
          - "--enable-lora"
          - "--max-loras"
          - "2"
          - "--max-cpu-loras"
          - "12"
          env:
            # Enabling LoRA support temporarily disables automatic v1, we want to force it on
            # until 0.8.3 vLLM is released.
            - name: PORT
              value: "8000"
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-token
                  key: token
            - name: VLLM_ALLOW_RUNTIME_LORA_UPDATING
              value: "true"
          ports:
            - containerPort: 8000
              name: http
              protocol: TCP
          lifecycle:
            preStop:
              # vLLM stops accepting connections when it receives SIGTERM, so we need to sleep
              # to give upstream gateways a chance to take us out of rotation. The time we wait
              # is dependent on the time it takes for all upstreams to completely remove us from
              # rotation. Older or simpler load balancers might take upwards of 30s, but we expect
              # our deployment to run behind a modern gateway like Envoy which is designed to
              # probe for readiness aggressively.
              sleep:
                # Upstream gateway probers for health should be set on a low period, such as 5s,
                # and the shorter we can tighten that bound the faster that we release
                # accelerators during controlled shutdowns. However, we should expect variance,
                # as load balancers may have internal delays, and we don't want to drop requests
                # normally, so we're often aiming to set this value to a p99 propagation latency
                # of readiness -> load balancer taking backend out of rotation, not the average.
                #
                # This value is generally stable and must often be experimentally determined on
                # for a given load balancer and health check period. We set the value here to
                # the highest value we observe on a supported load balancer, and we recommend
                # tuning this value down and verifying no requests are dropped.
                #
                # If this value is updated, be sure to update terminationGracePeriodSeconds.
                #
                seconds: 30
              #
              # IMPORTANT: preStop.sleep is beta as of Kubernetes 1.30 - for older versions
              # replace with this exec action.
              #exec:
              #  command:
              #  - /usr/bin/sleep
              #  - 30
          livenessProbe:
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            # vLLM's health check is simple, so we can more aggressively probe it.  Liveness
            # check endpoints should always be suitable for aggressive probing.
            periodSeconds: 1
            successThreshold: 1
            # vLLM has a very simple health implementation, which means that any failure is
            # likely significant. However, any liveness triggered restart requires the very
            # large core model to be reloaded, and so we should bias towards ensuring the
            # server is definitely unhealthy vs immediately restarting. Use 5 attempts as
            # evidence of a serious problem.
            failureThreshold: 5
            timeoutSeconds: 1
          readinessProbe:
            httpGet:
              path: /health
              port: http
              scheme: HTTP
            # vLLM's health check is simple, so we can more aggressively probe it.  Readiness
            # check endpoints should always be suitable for aggressive probing, but may be
            # slightly more expensive than readiness probes.
            periodSeconds: 1
            successThreshold: 1
            # vLLM has a very simple health implementation, which means that any failure is
            # likely significant,
            failureThreshold: 1
            timeoutSeconds: 1
          # We set a startup probe so that we don't begin directing traffic or checking
          # liveness to this instance until the model is loaded.
          startupProbe:
            # Failure threshold is when we believe startup will not happen at all, and is set
            # to the maximum possible time we believe loading a model will take. In our
            # default configuration we are downloading a model from HuggingFace, which may
            # take a long time, then the model must load into the accelerator. We choose
            # 10 minutes as a reasonable maximum startup time before giving up and attempting
            # to restart the pod.
            #
            # IMPORTANT: If the core model takes more than 10 minutes to load, pods will crash
            # loop forever. Be sure to set this appropriately.
            failureThreshold: 3600
            # Set delay to start low so that if the base model changes to something smaller
            # or an optimization is deployed, we don't wait unnecessarily.
            initialDelaySeconds: 2
            # As a startup probe, this stops running and so we can more aggressively probe
            # even a moderately complex startup - this is a very important workload.
            periodSeconds: 1
            httpGet:
              # vLLM does not start the OpenAI server (and hence make /health available)
              # until models are loaded. This may not be true for all model servers.
              path: /health
              port: http
              scheme: HTTP

          resources:
            limits:
              nvidia.com/gpu: 1
            requests:
              nvidia.com/gpu: 1
          volumeMounts:
            - mountPath: /data
              name: data
            - mountPath: /dev/shm
              name: shm
            - name: adapters
              mountPath: "/adapters"
      initContainers:
        - name: lora-adapter-syncer
          tty: true
          stdin: true
          image: us-central1-docker.pkg.dev/k8s-staging-images/gateway-api-inference-extension/lora-syncer:main
          restartPolicy: Always
          imagePullPolicy: Always
          env:
            - name: DYNAMIC_LORA_ROLLOUT_CONFIG
              value: "/config/configmap.yaml"
          volumeMounts: # DO NOT USE subPath, dynamic configmap updates don't work on subPaths
          - name: config-volume
            mountPath:  /config
      restartPolicy: Always

      # vLLM allows VLLM_PORT to be specified as an environment variable, but a user might
      # create a 'vllm' service in their namespace. That auto-injects VLLM_PORT in docker
      # compatible form as `tcp://<IP>:<PORT>` instead of the numeric value vLLM accepts
      # causing CrashLoopBackoff. Set service environment injection off by default.
      enableServiceLinks: false

      # Generally, the termination grace period needs to last longer than the slowest request
      # we expect to serve plus any extra time spent waiting for load balancers to take the
      # model server out of rotation.
      #
      # An easy starting point is the p99 or max request latency measured for your workload,
      # although LLM request latencies vary significantly if clients send longer inputs or
      # trigger longer outputs. Since steady state p99 will be higher than the latency
      # to drain a server, you may wish to slightly this value either experimentally or
      # via the calculation below.
      #
      # For most models you can derive an upper bound for the maximum drain latency as
      # follows:
      #
      #   1. Identify the maximum context length the model was trained on, or the maximum
      #      allowed length of output tokens configured on vLLM (llama2-7b was trained to
      #      4k context length, while llama3-8b was trained to 128k).
      #   2. Output tokens are the more compute intensive to calculate and the accelerator
      #      will have a maximum concurrency (batch size) - the time per output token at
      #      maximum batch with no prompt tokens being processed is the slowest an output
      #      token can be generated (for this model it would be about 100ms TPOT at a max
      #      batch size around 50)
      #   3. Calculate the worst case request duration if a request starts immediately
      #      before the server stops accepting new connections - generally when it receives
      #      SIGTERM (for this model that is about 4096 / 10 ~ 40s)
      #   4. If there are any requests generating prompt tokens that will delay when those
      #      output tokens start, and prompt token generation is roughly 6x faster than
      #      compute-bound output token generation, so add 20% to the time from above (40s +
      #      16s ~ 55s)
      #
      # Thus we think it will take us at worst about 55s to complete the longest possible
      # request the model is likely to receive at maximum concurrency (highest latency)
      # once requests stop being sent.
      #
      # NOTE: This number will be lower than steady state p99 latency since we stop receiving
      #       new requests which require continuous prompt token computation.
      # NOTE: The max timeout for backend connections from gateway to model servers should
      #       be configured based on steady state p99 latency, not drain p99 latency
      #
      #   5. Add the time the pod takes in its preStop hook to allow the load balancers have
      #      stopped sending us new requests (55s + 30s ~ 85s)
      #
      # Because termination grace period controls when the Kubelet forcibly terminates a
      # stuck or hung process (a possibility due to a GPU crash), there is operational safety
      # in keeping the value roughly proportional to the time to finish serving. There is also
      # value in adding a bit of extra time to deal with unexpectedly long workloads.
      #
      #   6. Add a 50% safety buffer to this time since the operational impact should be low
      #      (85s * 1.5 ~ 130s)
      #
      # One additional source of drain latency is that some workloads may run close to
      # saturation and have queued requests on each server. Since traffic in excess of the
      # max sustainable QPS will result in timeouts as the queues grow, we assume that failure
      # to drain in time due to excess queues at the time of shutdown is an expected failure
      # mode of server overload. If your workload occasionally experiences high queue depths
      # due to periodic traffic, consider increasing the safety margin above to account for
      # time to drain queued requests.
      terminationGracePeriodSeconds: 130
      nodeSelector:
        cloud.google.com/gke-accelerator: "nvidia-h100-80gb"
      volumes:
        - name: data
          emptyDir: {}
        - name: shm
          emptyDir:
            medium: Memory
        - name: adapters
          emptyDir: {}
        - name: config-volume
          configMap:
            name: vllm-llama3-8b-adapters
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: vllm-llama3-8b-adapters
data:
  configmap.yaml: |
      vLLMLoRAConfig:
        name: vllm-llama3.1-8b-instruct
        port: 8000
        defaultBaseModel: meta-llama/Llama-3.1-8B-Instruct
        ensureExist:
          models:
          - id: food-review
            source: Kawon/llama3.1-food-finetune_v14_r8
          - id: cad-fabricator
            source: redcathode/fabricator
---
kind: HealthCheckPolicy
apiVersion: networking.gke.io/v1
metadata:
  name: health-check-policy
  namespace: default
spec:
  targetRef:
    group: "inference.networking.x-k8s.io"
    kind: InferencePool
    name: vllm-llama3-8b-instruct
  default:
    config:
      type: HTTP
      httpHealthCheck:
          requestPath: /health
          port: 8000

將範例資訊清單套用至叢集：

kubectl apply -f vllm-llama3-8b-instruct.yaml

套用資訊清單後，請考量下列重要欄位和參數：

replicas：指定 Deployment 的 Pod 數量。
image：指定模型伺服器的 Docker 映像檔。
command：指定容器啟動時要執行的指令。
args：指定要傳遞至指令的引數。
env：指定容器的環境變數。
ports：指定容器公開的通訊埠。
resources：指定容器的資源要求和限制，例如 GPU。
volumeMounts：指定磁碟區在容器中的掛接方式。
initContainers：指定在應用程式容器之前執行的容器。
restartPolicy：指定 Pod 的重新啟動政策。
terminationGracePeriodSeconds：指定 Pod 終止的緩衝期。
volumes：指定 Pod 使用的磁碟區。

您可以修改這些欄位，以符合特定需求。

建立推論集區

InferencePool Kubernetes 自訂資源會定義一組 Pod，這些 Pod 具有共同的基礎大型語言模型 (LLM) 和運算設定。selector 欄位會指定哪些 Pod 屬於這個集區。這個選取器中的標籤必須與套用至模型伺服器 Pod 的標籤完全一致。targetPort 欄位會定義模型伺服器在 Pod 中使用的通訊埠。extensionRef 欄位會參照擴充功能服務，為推論集區提供額外功能。InferencePool 可讓 GKE 推論閘道將流量轉送至模型伺服器 Pod。

建立 InferencePool 前，請確認 InferencePool 選取的 Pod 已在執行中。

如要使用 Helm 建立 InferencePool，請執行下列步驟：

helm install vllm-llama3-8b-instruct \
  --set inferencePool.modelServers.matchLabels.app=vllm-llama3-8b-instruct \
  --set provider.name=gke \
  --version v0.3.0 \
  oci://registry.k8s.io/gateway-api-inference-extension/charts/inferencepool

變更下列欄位，以符合您的部署作業：

inferencePool.modelServers.matchLabels.app：用於選取模型伺服器 Pod 的標籤鍵。

Helm 安裝作業會自動安裝必要的逾時政策、端點選擇器，以及可觀測性所需的 Pod。

這會建立 InferencePool 物件：vllm-llama3-8b-instruct 參照 Pod 內的模型端點服務。此外，系統也會為這個建立的 InferencePool 建立名為 app:vllm-llama3-8b-instruct-epp 的 Endpoint Picker 部署作業。

指定模型服務目標

InferenceModel 自訂資源會定義要放送的特定模型，包括支援 LoRA 微調模型及其放送重要性。您必須建立 InferenceModel 資源，定義 InferencePool 上提供的模型。這些 InferenceModel 資源可以參照基礎模型，或 InferencePool 中模型伺服器支援的 LoRA 配接器。

modelName 欄位會指定基礎模型或 LoRA 配接器的名稱。Criticality 欄位會指定模型的服務重要性。poolRef 欄位會指定提供此模型的 InferencePool。

如要建立 InferenceModel，請執行下列步驟：

將下列範例資訊清單儲存為 inferencemodel.yaml：
```
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: inferencemodel-sample
spec:
  modelName: MODEL_NAME
  criticality: VALUE
  poolRef:
    name: INFERENCE_POOL_NAME
```
更改下列內容：
- MODEL_NAME：基礎模型或 LoRA 轉接程式的名稱。例如：food-review。
- VALUE：所選的放送重要性。選擇 Critical、Standard 或 Sheddable。例如：Standard。
- INFERENCE_POOL_NAME：您在上一個步驟中建立的InferencePool名稱。例如：vllm-llama3-8b-instruct。
將範例資訊清單套用至叢集：
```
kubectl apply -f inferencemodel.yaml
```

以下範例會建立 InferenceModel 物件，在 vllm-llama3-8b-instruct InferencePool 上設定 food-review LoRA 模型，並使用 Standard 服務重要性。InferenceModel 物件也會設定要以 Critical 優先順序層級放送的基礎模型。

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: food-review
spec:
  modelName: food-review
  criticality: Standard
  poolRef:
    name: vllm-llama3-8b-instruct
  targetModels:
  - name: food-review
    weight: 100

---
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: llama3-base-model
spec:
  modelName: meta-llama/Llama-3.1-8B-Instruct
  criticality: Critical
  poolRef:
    name: vllm-llama3-8b-instruct

建立閘道

Gateway 資源是外部流量進入 Kubernetes 叢集的進入點。定義接受連入連線的接聽程式。

GKE 推論閘道支援下列閘道類別：

gke-l7-rilb：適用於區域性內部應用程式負載平衡器。
gke-l7-regional-external-managed

詳情請參閱 GatewayClasses 說明文件。

如要建立閘道，請執行下列步驟：

將下列範例資訊清單儲存為 gateway.yaml：

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: GATEWAY_NAME
spec:
  gatewayClassName: GATEWAY_CLASS
  listeners:
    - protocol: HTTP
      port: 80
      name: http

將 GATEWAY_NAME 替換為閘道資源的專屬名稱 (例如 inference-gateway)，並將 GATEWAY_CLASS 替換為要使用的閘道類別 (例如 gke-l7-regional-external-managed)。

將資訊清單套用至叢集：
```
kubectl apply -f gateway.yaml
```

注意：如要進一步瞭解如何設定 TLS，透過 HTTPS 保護 Gateway 安全，請參閱 GKE 說明文件中的 TLS 設定。

建立 `HTTPRoute`

HTTPRoute 資源定義 GKE 閘道如何將傳入的 HTTP 要求轉送至後端服務，在此情況下，後端服務就是 InferencePool。HTTPRoute 資源會指定相符規則 (例如標頭或路徑)，以及流量應轉送至的後端。

如要建立 HTTPRoute，請將下列範例資訊清單儲存為 httproute.yaml：
```
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: HTTPROUTE_NAME
spec:
  parentRefs:
  - name: GATEWAY_NAME
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: PATH_PREFIX
    backendRefs:
    - name: INFERENCE_POOL_NAME
      group: inference.networking.x-k8s.io
      kind: InferencePool
```
更改下列內容：
- HTTPROUTE_NAME：HTTPRoute資源的專屬名稱。例如：my-route。
- GATEWAY_NAME：您建立的 Gateway 資源名稱。例如：inference-gateway。
- PATH_PREFIX：用於比對傳入要求的路徑前置字元。例如，/ 可比對所有項目。
- INFERENCE_POOL_NAME：要將流量導向的InferencePool資源名稱。例如：vllm-llama3-8b-instruct。
將資訊清單套用至叢集：
```
kubectl apply -f httproute.yaml
```

傳送推論要求

設定 GKE Inference Gateway 後，您可以將推論要求傳送至已部署的模型。您可以根據輸入的提示和指定的參數生成文字。

如要傳送推論要求，請執行下列步驟：

如要取得 Gateway 端點，請執行下列指令：
```
IP=$(kubectl get gateway/GATEWAY_NAME -o jsonpath='{.status.addresses[0].value}')
PORT=PORT_NUMBER # Use 80 for HTTP
```
更改下列內容：
- GATEWAY_NAME：Gateway 資源的名稱。
- PORT_NUMBER：您在 Gateway 中設定的通訊埠號碼。
如要使用 curl 將要求傳送至 /v1/completions 端點，請執行下列指令：
```
curl -i -X POST ${IP}:${PORT}/v1/completions \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer $(gcloud auth print-access-token)' \
-d '{
    "model": "MODEL_NAME",
    "prompt": "PROMPT_TEXT",
    "max_tokens": MAX_TOKENS,
    "temperature": "TEMPERATURE"
}'
```
更改下列內容：
- MODEL_NAME：要使用的模型或 LoRA 轉接器名稱。
- PROMPT_TEXT：模型的輸入提示。
- MAX_TOKENS：回覆中生成的權杖數量上限。
- TEMPERATURE：控制輸出內容的隨機性。如要取得確定性輸出內容，請使用 0 值；如要取得更多創意輸出內容，請使用較大的數字。

以下範例說明如何將範例要求傳送至 GKE Inference Gateway：

curl -i -X POST ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -H 'Authorization: Bearer $(gcloud auth print-access-token)' -d '{
    "model": "food-review",
    "prompt": "What is the best pizza in the world?",
    "max_tokens": 2048,
    "temperature": "0"
}'

請注意下列行為：

要求主體：要求主體可以包含其他參數，例如 stop 和 top_p。如需完整的選項清單，請參閱 OpenAI API 規格。
錯誤處理：在用戶端程式碼中實作適當的錯誤處理機制，處理回應中可能發生的錯誤。舉例來說，請檢查 curl 回應中的 HTTP 狀態碼。非 200 狀態碼通常表示發生錯誤。
驗證和授權：針對實際工作環境部署作業，請使用驗證和授權機制保護 API 端點。在要求中加入適當的標頭 (例如 Authorization)。