Invoke predictions with model endpoint management

This page describes how to invoke predictions using the model endpoints registered in the Model endpoint management.

Before you begin

Make sure that you have registered your model endpoint with Model endpoint management. For more information, see Register a model endpoint with model endpoint management

Invoke predictions for generic models

Use the google_ml.predict_row() SQL function to call a registered generic model endpoint to invoke predictions.

SELECT
  google_ml.predict_row(
    model_id => 'MODEL_ID',
    request_body => 'REQUEST_BODY');

Replace the following:

• MODEL_ID: the model ID you defined when registering the model endpoint.
• REQUEST_BODY: the parameters to the prediction function, in JSON format.

Examples

This section includes some examples for invoking predictions using registered model endpoints.

To generate predictions for a registered gemini-1.5-pro:streamGenerateContent model endpoint, run the following statement:

  SELECT
  json_array_elements( google_ml.predict_row( model_id => 'gemini-1.5-pro:streamGenerateContent',
      request_body => '{ "contents": [ { "role": "user", "parts": [ { "text": "For TPCH database schema as mentioned here https://www.tpc.org/TPC_Documents_Current_Versions/pdf/TPC-H_v3.0.1.pdf , generate a SQL query to find all supplier names which are located in the India nation." } ] } ] }'))-> 'candidates' -> 0 -> 'content' -> 'parts' -> 0 -> 'text';

To generate predictions for a registered facebook/bart-large-mnli model endpoint on Hugging Face, run the following statement:

  SELECT
  google_ml.predict_row(
    model_id => 'facebook/bart-large-mnli',
    request_body =>
      '{
       "inputs": "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!",
    "parameters": {"candidate_labels": ["refund", "legal", "faq"]}
    }'
  );

To generate predictions for a registered Anthropic claude-3-opus-20240229 model endpoint, run the following statement:

  SELECT
  google_ml.predict_row('anthropic-opus', '{
    "model": "claude-3-opus-20240229",
    "max_tokens": 1024,
    "messages": [
        {"role": "user", "content": "Hello, world"}
    ]
  }');