Migrate to PaLM API from Azure OpenAI

This user guide outlines the steps required to migrate to Vertex AI PaLM API from Microsoft Azure OpenAI.

Objectives

Vertex AI PaLM API is a service that allows you to create and train generative models using Google Cloud. It's a fully managed service that provides a simple and intuitive interface for creating and training generative models.

Both Azure OpenAI and Vertex AI PaLM API are cloud-based services that provide access to powerful large language models (LLMs). LLMs can be used to create a variety of applications, including chatbots, content generators, and creative tools. The services provided are backed by enterprise-grade security and compliance and include built-in responsible AI features.

Prerequisites

To migrate an OpenAI service from Microsoft Azure Open AI to Vertex AI PaLM API, you must first create a Google Cloud project and development environment. For more information, see Set up a project and a development environment.

How to migrate to PaLM API from Azure OpenAI

Use the following topics to learn how to migrate to Vertex AI PaLM API from OpenAI project in Microsoft Azure.

Use equivalent PaLM API parameters

The following are some common Azure OpenAI parameters and their equivalent parameters in the PaLM API

Parameter

Azure

Parameter

Google Cloud

Description Valid values
prompt prompt A prompt is a natural language request submitted to a language model to receive a response back. Prompts can contain questions, instructions, contextual information, examples, and text for the model to complete or continue. Text
temperature temperature The temperature is used for sampling during response generation, which occurs when topP and topK are applied. Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that require a less open-ended or creative response, while higher temperatures can lead to more diverse or creative results. A temperature of 0 means that the highest probability tokens are always selected. In this case, responses for a given prompt are mostly deterministic, but a small amount of variation is still possible.

For most use cases, try starting with a temperature of 0.2. If the model returns a response that's too generic, too short, or the model gives a fallback response, try increasing the temperature.

 0.0–1.0

max_tokens maxOutputTokens Maximum number of tokens that can be generated in the response. A token is approximately four characters. 100 tokens correspond to roughly 60-80 words.

Specify a lower value for shorter responses and a higher value for longer responses.

 1-2048 (Azure)

1–1024 (Google Cloud)

Not available topK Top-K changes how the model selects tokens for output. A top-K of 1 means the next selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-K of 3 means that the next token is selected from among the three most probable tokens by using temperature.

For each token selection step, the top-K tokens with the highest probabilities are sampled. Then tokens are further filtered based on top-P with the final token selected using temperature sampling.

Specify a lower value for less random responses and a higher value for more random responses. The default top-K is 40.

 1–40

top_p topP Top-P changes how the model selects tokens for output. Tokens are selected from the most (see top-K) to least probable until the sum of their probabilities equals the top-P value. For example, if tokens A, B, and C have a probability of 0.3, 0.2, and 0.1 and the top-P value is 0.5, then the model will select either A or B as the next token by using temperature and excludes C as a candidate.

Specify a lower value for less random responses and a higher value for more random responses. The default top-P is 0.95.

 0.0–1.0

Use the equivalent PaLM API model

The following table describes the foundation models available.

Type Description Microsoft Azure endpoints Vertex AI LLM endpoints
Text Fine-tuned to follow natural language instructions and is suitable for a variety of language tasks. text-davinci-003 text-bison@001
Chat Fine-tuned for multi-turn conversation use cases. gpt-35-turbo or gpt-4 chat-bison@001
Embedding Fine-tuned to return model embeddings for text inputs. text-embedding-ada-002 textembedding-gecko@001

Learn how to get started with Generative AI support on Vertex AI

The following shows you the equivalent Vertex AI SDK for Python methods for Azure OpenAI methods. Use the Vertex AI SDK for Python to get started with Generative AI support on Vertex AI.

Install Vertex AI PaLM API

Azure OpenAI

$ pip install --upgrade openai

Vertex AI PaLM API

$ pip install google-cloud-aiplatform

Import Vertex AI PaLM API

Azure OpenAI

import openai

Vertex AI PaLM API

from vertexai.preview.language_models import TextGenerationModel

Authenticate Vertex AI PaLM API

Azure OpenAI

openai.api_key = os.getenv("OPENAI_API_KEY")

Vertex AI PaLM API

from google.colab import auth as google_auth
google_auth.authenticate_user()

Vertex AI PaLM API and Azure comparisons and sample code

Generate text with the Vertex AI SDK for Python

Azure OpenAI

import openai

response = openai.Completion.create(
    prompt="Hello",
    max_tokens=256,
    temperature=0.3,
    deployment_id="text-davinci-003",
    #engine="text-davinci-003",)
print(f"Response from Model: {response['choices'][0]['text']}")

Vertex AI PaLM API

from vertexai.preview.language_models import TextGenerationModel

model = TextGenerationModel.from_pretrained("text-bison@001")

response = model.predict(
    "Hello",
    max_output_tokens=256,
    temperature=0.3,)

print(f"Response from Model: {response.text}")

Use chat completion with the Vertex AI SDK for Python

Azure OpenAI

import openai

parameters = {
    "deployment_id"="gpt-4",
    #"engine": "gpt-4",
    "temperature": 0.2,
    "max_tokens": 256,
    "top_p": 0.95,
}

chat = openai.ChatCompletion.create(
    messages=[
      {"role": "system", "content": "My name is Miles. You are an astronomer, knowledgeable about the solar system."},
      {"role": "user", "name":"example_user", "content": "How many planets are there in the solar system?"}
      ],
    **parameters
)

response = chat['choices'][0]['message']['content']

print(f"Response from Azure OpenAI Model: {response.text}")

Vertex AI PaLM API

from vertexai.preview.language_models import ChatModel

chat_model = ChatModel.from_pretrained("chat-bison@001")

parameters = {
    "temperature": 0.2,
    "max_output_tokens": 256,
    "top_p": 0.95,
}

chat = chat_model.start_chat(context="My name is Miles. You are an astronomer, knowledgeable about the solar system.")

response = chat.send_message(
    "How many planets are there in the solar system?",
    **parameters)

print(f"Response from Google GenAI Model: {response.text}")

Use text embedding with the Vertex AI SDK for Python

Azure OpenAI

import openai

embeddings = openai.Embedding.create(
  deployment_id="text-embedding-ada-002",
  #engine="text-embedding-ada-002",
  input="What is life?"
)["data"][0]["embedding"]

print(f'Length of Embedding Vector: {len(embeddings)}')

Vertex AI PaLM API

from vertexai.preview.language_models import TextEmbeddingModel

model = TextEmbeddingModel.from_pretrained("textembedding-gecko@001")
embeddings = model.get_embeddings(["What is life?"])

for embedding in embeddings:
  vector = embedding.values

print(f'Length of Embedding Vector: {len(vector)}')

What's next