List and count tokens

Preview

This product or feature is subject to the "Pre-GA Offerings Terms" in the General Service Terms section of the Service Specific Terms. Pre-GA products and features are available "as is" and might have limited support. For more information, see the launch stage descriptions.

The Vertex AI SDK for Python (1.60.0 and later) includes an integrated tokenizer, which lets you list and count the tokens of a prompt locally without having to make API calls. This page shows you how to list the tokens and their token IDs of a prompt and how to get a total token count of a prompt by using the Vertex AI SDK for Python.

Tokens and the importance of token listing and counting

Generative AI models break down text and other data in a prompt into units called tokens for processing. The way that data is converted into tokens depends on the tokenizer used. A token can be characters, words, or phrases.

Each model has a maximum number of tokens that it can handle in a prompt and response. Knowing the token count of your prompt lets you know whether you've exceeded this limit or not. Additionally, counting tokens also returns the billable characters for the prompt, which helps you estimate cost.

Listing tokens returns a list of the tokens that your prompt is broken down into. Each listed token is associated with a token ID, which helps you perform troubleshooting and analyze model behavior.

Supported models

The following table shows you the models that support token listing and token counting:

List tokens	Count tokens
gemini-1.5-flash-002	gemini-1.5-flash-002
gemini-1.5-pro-002	gemini-1.5-pro-002
	gemini-1.0-pro-002
	gemini-1.0-pro-vision-001

Get a list of tokens and token IDs for a prompt

The following code sample shows you how to get a list of tokens and token IDs for a prompt. The prompt must contain only text. Multimodal prompts are not supported.

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation.

from vertexai.preview.tokenization import get_tokenizer_for_model

# init local tokenzier
tokenizer = get_tokenizer_for_model("gemini-1.5-flash-001")

# Count Tokens
prompt = "why is the sky blue?"
response = tokenizer.count_tokens(prompt)
print(f"Tokens count: {response.total_tokens}")
# Example response:
#       Tokens count: 6

# Compute Tokens
response = tokenizer.compute_tokens(prompt)
print(f"Tokens list: {response.tokens_info}")
# Example response:
#     Tokens list: [TokensInfo(token_ids=[18177, 603, 573, 8203, 3868, 235336],
#          tokens=[b'why', b' is', b' the', b' sky', b' blue', b'?'], role='user')]

Get the token count and billable characters of a prompt

The following code sample shows you how to Get the token count and the number of billable characters of a prompt. Both text-only and multimodal prompts are supported.

Python

To learn how to install or update the Vertex AI SDK for Python, see Install the Vertex AI SDK for Python. For more information, see the Python API reference documentation.

from vertexai.preview.tokenization import get_tokenizer_for_model

# Using local tokenzier
tokenizer = get_tokenizer_for_model("gemini-1.5-flash-002")

prompt = "hello world"
response = tokenizer.count_tokens(prompt)
print(f"Prompt Token Count: {response.total_tokens}")
# Example response:
# Prompt Token Count: 2

prompt = ["hello world", "what's the weather today"]
response = tokenizer.count_tokens(prompt)
print(f"Prompt Token Count: {response.total_tokens}")
# Example response:
# Prompt Token Count: 8