Diffuser des réponses à partir de modèles d'IA générative

La diffusion en streaming implique de recevoir des réponses aux invites au fur et à mesure de leur génération. Autrement dit, les jetons de sortie sont envoyés dès lors qu'ils ont été générés par le modèle.

Vous pouvez envoyer des requêtes en streaming au modèle LDD (Vert Language Model) Vertex AI à l'aide des éléments suivants :

Les API de diffusion et de non diffusion en continu utilisent les mêmes paramètres, et la tarification et les quotas ne diffèrent pas.

Vertex AI Studio

Vous pouvez utiliser Vertex AI Studio pour concevoir et exécuter des invites et recevoir des réponses en flux. Sur la page de conception de l'invite, cliquez sur le bouton Réponse en flux pour activer le flux.

Bouton de réponse en flux

Examples

Vous pouvez appeler l'API de traitement par flux à l'aide de l'une des méthodes suivantes :

API REST avec événements envoyés par le serveur (SSE)

Les paramètres sont différents selon les types de modèles utilisés dans les exemples suivants :

Texte

Les modèles actuellement compatibles sont text-bison et text-unicorn. Consultez les versions disponibles.

Requête

  PROJECT_ID=YOUR_PROJECT_ID
  PROMPT="PROMPT"
  MODEL_ID=text-bison

  curl \
  -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict?alt=sse -d \
  '{
    "inputs": [
      {
        "struct_val": {
          "prompt": {
            "string_val": [ "'"${PROMPT}"'" ]
          }
        }
      }
    ],
    "parameters": {
      "struct_val": {
        "temperature": { "float_val": 0.8 },
        "maxOutputTokens": { "int_val": 1024 },
        "topK": { "int_val": 40 },
        "topP": { "float_val": 0.95 }
      }
    }
  }'

Response (Réponse)

Les réponses sont des messages d'événement envoyés par le serveur.

  data: {"outputs": [{"structVal": {"content": {"stringVal": [RESPONSE]},"safetyAttributes": {"structVal": {"blocked": {"boolVal": [BOOLEAN]},"categories": {"listVal": [{"stringVal": [Safety category name]}]},"scores": {"listVal": [{"doubleVal": [Safety category score]}]}}},"citationMetadata": {"structVal": {"citations": {}}}}}]}

Chat

Le modèle actuellement compatible est chat-bison. Consultez les versions disponibles.

Requête

PROJECT_ID=YOUR_PROJECT_ID
PROMPT="PROMPT"
AUTHOR="USER"
MODEL_ID=chat-bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict?alt=sse -d \
$'{
  "inputs": [
    {
      "struct_val": {
        "messages": {
          "list_val": [
            {
              "struct_val": {
                "content": {
                  "string_val": [ "'"${PROMPT}"'" ]
                },
                "author": {
                  "string_val": [ "'"${AUTHOR}"'"]
                }
              }
            }
          ]
        }
      }
    }
  ],
  "parameters": {
    "struct_val": {
      "temperature": { "float_val": 0.5 },
      "maxOutputTokens": { "int_val": 1024 },
      "topK": { "int_val": 40 },
      "topP": { "float_val": 0.95 }
    }
  }
}'

Response (Réponse)

Les réponses sont des messages d'événement envoyés par le serveur.

data: {"outputs": [{"structVal": {"candidates": {"listVal": [{"structVal": {"author": {"stringVal": [AUTHOR]},"content": {"stringVal": [RESPONSE]}}}]},"citationMetadata": {"listVal": [{"structVal": {"citations": {}}}]},"safetyAttributes": {"structVal": {"blocked": {"boolVal": [BOOLEAN]},"categories": {"listVal": [{"stringVal": [Safety category name]}]},"scores": {"listVal": [{"doubleVal": [Safety category score]}]}}}}}]}

Code

Le modèle actuellement compatible est code-bison. Consultez les versions disponibles.

Requête

PROJECT_ID=YOUR_PROJECT_ID
PROMPT="PROMPT"
MODEL_ID=code-bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict?alt=sse -d \
$'{
  "inputs": [
    {
      "struct_val": {
        "prefix": {
          "string_val": [ "'"${PROMPT}"'" ]
        }
      }
    }
  ],
  "parameters": {
    "struct_val": {
      "temperature": { "float_val": 0.8 },
      "maxOutputTokens": { "int_val": 1024 },
      "topK": { "int_val": 40 },
      "topP": { "float_val": 0.95 }
    }
  }
}'

Response (Réponse)

Les réponses sont des messages d'événement envoyés par le serveur.

data: {"outputs": [{"structVal": {"citationMetadata": {"structVal": {"citations": {}}},"safetyAttributes": {"structVal": {"blocked": {"boolVal": [BOOLEAN]},"categories": {"listVal": [{"stringVal": [Safety category name]}]},"scores": {"listVal": [{"doubleVal": [Safety category score]}]}}},"content": {"stringVal": [RESPONSE]}}}]}

Chat de code

Le modèle actuellement compatible est codechat-bison. Consultez les versions disponibles.

Requête

PROJECT_ID=YOUR_PROJECT_ID
PROMPT="PROMPT"
AUTHOR="USER"
MODEL_ID=codechat-bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict?alt=sse -d \
$'{
  "inputs": [
    {
      "struct_val": {
        "messages": {
          "list_val": [
            {
              "struct_val": {
                "content": {
                  "string_val": [ "'"${PROMPT}"'" ]
                },
                "author": {
                  "string_val": [ "'"${AUTHOR}"'"]
                }
              }
            }
          ]
        }
      }
    }
  ],
  "parameters": {
    "struct_val": {
      "temperature": { "float_val": 0.5 },
      "maxOutputTokens": { "int_val": 1024 },
      "topK": { "int_val": 40 },
      "topP": { "float_val": 0.95 }
    }
  }
}'

Response (Réponse)

Les réponses sont des messages d'événement envoyés par le serveur.

data: {"outputs": [{"structVal": {"safetyAttributes": {"structVal": {"blocked": {"boolVal": [BOOLEAN]},"categories": {"listVal": [{"stringVal": [Safety category name]}]},"scores": {"listVal": [{"doubleVal": [Safety category score]}]}}},"citationMetadata": {"listVal": [{"structVal": {"citations": {}}}]},"candidates": {"listVal": [{"structVal": {"content": {"stringVal": [RESPONSE]},"author": {"stringVal": [AUTHOR]}}}]}}}]}

API REST

Les paramètres sont différents selon les types de modèles utilisés dans les exemples suivants :

Texte

Les modèles actuellement compatibles sont text-bison et text-unicorn. Consultez les versions disponibles.

Requête

  PROJECT_ID=YOUR_PROJECT_ID
  PROMPT="PROMPT"
  MODEL_ID=text-bison

  curl \
  -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict -d \
  '{
    "inputs": [
      {
        "struct_val": {
          "prompt": {
            "string_val": [ "'"${PROMPT}"'" ]
          }
        }
      }
    ],
    "parameters": {
      "struct_val": {
        "temperature": { "float_val": 0.8 },
        "maxOutputTokens": { "int_val": 1024 },
        "topK": { "int_val": 40 },
        "topP": { "float_val": 0.95 }
      }
    }
  }'

Response (Réponse)

{
  "outputs": [
    {
      "structVal": {
        "citationMetadata": {
          "structVal": {
            "citations": {}
          }
        },
        "safetyAttributes": {
          "structVal": {
            "categories": {},
            "scores": {},
            "blocked": {
              "boolVal": [
                false
              ]
            }
          }
        },
        "content": {
          "stringVal": [
            RESPONSE
          ]
        }
      }
    }
  ]
}

Chat

Le modèle actuellement compatible est chat-bison. Consultez les versions disponibles.

Requête

PROJECT_ID=YOUR_PROJECT_ID
PROMPT="PROMPT"
AUTHOR="USER"
MODEL_ID=chat-bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict -d \
$'{
  "inputs": [
    {
      "struct_val": {
        "messages": {
          "list_val": [
            {
              "struct_val": {
                "content": {
                  "string_val": [ "'"${PROMPT}"'" ]
                },
                "author": {
                  "string_val": [ "'"${AUTHOR}"'"]
                }
              }
            }
          ]
        }
      }
    }
  ],
  "parameters": {
    "struct_val": {
      "temperature": { "float_val": 0.5 },
      "maxOutputTokens": { "int_val": 1024 },
      "topK": { "int_val": 40 },
      "topP": { "float_val": 0.95 }
    }
  }
}'

Response (Réponse)

{
  "outputs": [
    {
      "structVal": {
        "candidates": {
          "listVal": [
            {
              "structVal": {
                "content": {
                  "stringVal": [
                    RESPONSE
                  ]
                },
                "author": {
                  "stringVal": [
                    AUTHOR
                  ]
                }
              }
            }
          ]
        },
        "citationMetadata": {
          "listVal": [
            {
              "structVal": {
                "citations": {}
              }
            }
          ]
        },
        "safetyAttributes": {
          "listVal": [
            {
              "structVal": {
                "categories": {},
                "blocked": {
                  "boolVal": [
                    false
                  ]
                },
                "scores": {}
              }
            }
          ]
        }
      }
    }
  ]
}

Code

Le modèle actuellement compatible est code-bison. Consultez les versions disponibles.

Requête

PROJECT_ID=YOUR_PROJECT_ID
PROMPT="PROMPT"
MODEL_ID=code-bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict -d \
$'{
  "inputs": [
    {
      "struct_val": {
        "prefix": {
          "string_val": [ "'"${PROMPT}"'" ]
        }
      }
    }
  ],
  "parameters": {
    "struct_val": {
      "temperature": { "float_val": 0.8 },
      "maxOutputTokens": { "int_val": 1024 },
      "topK": { "int_val": 40 },
      "topP": { "float_val": 0.95 }
    }
  }
}'

Response (Réponse)

{
  "outputs": [
    {
      "structVal": {
        "safetyAttributes": {
          "structVal": {
            "categories": {},
            "scores": {},
            "blocked": {
              "boolVal": [
                false
              ]
            }
          }
        },
        "citationMetadata": {
          "structVal": {
            "citations": {}
          }
        },
        "content": {
          "stringVal": [
            RESPONSE
          ]
        }
      }
    }
  ]
}

Chat de code

Le modèle actuellement compatible est codechat-bison. Consultez les versions disponibles.

Requête

PROJECT_ID=YOUR_PROJECT_ID
PROMPT="PROMPT"
AUTHOR="USER"
MODEL_ID=codechat-bison

curl \
-X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
https://us-central1-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/us-central1/publishers/google/models/${MODEL_ID}:serverStreamingPredict -d \
$'{
  "inputs": [
    {
      "struct_val": {
        "messages": {
          "list_val": [
            {
              "struct_val": {
                "content": {
                  "string_val": [ "'"${PROMPT}"'" ]
                },
                "author": {
                  "string_val": [ "'"${AUTHOR}"'"]
                }
              }
            }
          ]
        }
      }
    }
  ],
  "parameters": {
    "struct_val": {
      "temperature": { "float_val": 0.5 },
      "maxOutputTokens": { "int_val": 1024 },
      "topK": { "int_val": 40 },
      "topP": { "float_val": 0.95 }
    }
  }
}'

Response (Réponse)

{
  "outputs": [
    {
      "structVal": {
        "candidates": {
          "listVal": [
            {
              "structVal": {
                "content": {
                  "stringVal": [
                    RESPONSE
                  ]
                },
                "author": {
                  "stringVal": [
                    AUTHOR
                  ]
                }
              }
            }
          ]
        },
        "citationMetadata": {
          "listVal": [
            {
              "structVal": {
                "citations": {}
              }
            }
          ]
        },
        "safetyAttributes": {
          "listVal": [
            {
              "structVal": {
                "categories": {},
                "blocked": {
                  "boolVal": [
                    false
                  ]
                },
                "scores": {}
              }
            }
          ]
        }
      }
    }
  ]
}

SDK Vertex AI pour Python

Pour en savoir plus sur l'installation du SDK Vertex AI pour Python, consultez la section Installer le SDK Vertex AI pour Python.

Texte 

  import vertexai
  from vertexai.language_models import TextGenerationModel

  def streaming_prediction(
      project_id: str,
      location: str,
  ) -> str:
      """Streaming Text Example with a Large Language Model"""

  vertexai.init(project=project_id, location=location)

  text_generation_model = TextGenerationModel.from_pretrained("text-bison")
  parameters = {
      "temperature": temperature,  # Temperature controls the degree of randomness in token selection.
      "max_output_tokens": 256,  # Token limit determines the maximum amount of text output.
      "top_p": 0.8,  # Tokens are selected from most probable to least until the sum of their probabilities equals the top_p value.
      "top_k": 40,  # A top_k of 1 means the selected token is the most probable among all tokens.
  }

  responses = text_generation_model.predict_streaming(prompt="Give me ten interview questions for the role of program manager.", **parameters)
  for response in responses:
      `print(response)`

Chat 

import vertexai
from vertexai.language_models import ChatModel, InputOutputTextPair

def streaming_prediction(
    project_id: str,
    location: str,
) -> str:
    """Streaming Chat Example with a Large Language Model"""

    vertexai.init(project=project_id, location=location)

    chat_model = ChatModel.from_pretrained("chat-bison")

    parameters = {
        "temperature": 0.8,  # Temperature controls the degree of randomness in token selection.
        "max_output_tokens": 256,  # Token limit determines the maximum amount of text output.
        "top_p": 0.95,  # Tokens are selected from most probable to least until the sum of their probabilities equals the top_p value.
        "top_k": 40,  # A top_k of 1 means the selected token is the most probable among all tokens.
    }

    chat = chat_model.start_chat(
        context="My name is Miles. You are an astronomer, knowledgeable about the solar system.",
        examples=[
            InputOutputTextPair(
                input_text="How many moons does Mars have?",
                output_text="The planet Mars has two moons, Phobos and Deimos.",
            ),
        ],
    )

    responses = chat.send_message_streaming(
        message="How many planets are there in the solar system?", **parameters)
    for response in responses:
        `print(response)`

Code 

import vertexai
from vertexai.language_models import CodeGenerationModel

def streaming_prediction(
    project_id: str,
    location: str,
) -> str:
    """Streaming Chat Example with a Large Language Model"""

    vertexai.init(project=project_id, location=location)

    code_model = CodeGenerationModel.from_pretrained("code-bison")
    parameters = {
        "temperature": 0.8,  # Temperature controls the degree of randomness in token selection.
        "max_output_tokens": 256,  # Token limit determines the maximum amount of text output.
    }

    responses = code.predict_streaming(
        prefix="Write a function that checks if a year is a leap year.", **parameters)
    for response in responses:
        `print(response)`

Chat de code 

import vertexai
from vertexai.language_models import CodeChatModel

def streaming_prediction(
    project_id: str,
    location: str,
) -> str:
    """Streaming Chat Example with a Large Language Model"""

    vertexai.init(project=project_id, location=location)

    codechat_model = CodeChatModel.from_pretrained("codechat-bison")
    parameters = {
        "temperature": 0.8,  # Temperature controls the degree of randomness in token selection.
        "max_output_tokens": 1024,  # Token limit determines the maximum amount of text output.
    }
    codechat = codechat_model.start_chat()

    responses = codechat.send_message_streaming(
        message="Please help write a function to calculate the min of two numbers", **parameters)
    for response in responses:
        `print(response)`

Bibliothèques clientes disponibles

Vous pouvez utiliser l'une des bibliothèques clientes suivantes pour diffuser des réponses :

  • Python
  • Node.js
  • Java

Pour afficher des exemples de requêtes et de réponses de code à l'aide de l'API REST, consultez la page Exemples d'utilisation de l'API REST.

Pour afficher des exemples de requêtes et de réponses de code à l'aide du SDK Vertex AI pour Python, consultez la page Exemples d'utilisation du SDK Vertex AI pour Python.

Une IA responsable

Les filtres d'intelligence artificielle responsable (RAI) analysent la sortie du flux à mesure que le modèle le génère. Si une violation est détectée, les filtres bloquent les jetons de sortie mis en cause et renvoient une sortie avec une option bloquée sous safetyAttributes, ce qui met fin au flux.

Étapes suivantes