Run LLM inference on Cloud Run GPUs with Hugging Face TGI

Preview — GPU support for Cloud Run services

This feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

The following example shows how to run a backend service that runs the Hugging Face Text Generation Inference (TGI) toolkit, which is a toolkit for deploying and serving Large Language Models (LLMs), using Llama 3.

See the entire example at Deploy Llama 3.1 8B with TGI DLC on Cloud Run.