Boost BigQuery with Python: Managed Python UDFs now generally available

SQL is the industry standard for high-performance structured data analysis. However, expressing complex procedural logic, scientific computations, advanced string manipulations, or machine learning workflows in pure SQL can be highly challenging, if not impossible. That kind of work is better done with Python. Data practitioners often take on additional infrastructure management tasks — maintaining custom images and containers, and working with additional compute services — just to run simple helper functions with custom Python code and libraries. 

Today, we are thrilled to announce the general availability (GA) of BigQuery Managed Python User-Defined Functions (UDFs).

This launch represents a major milestone in BigQuery’s extensibility strategy, allowing data scientists, engineers, and analysts to execute custom Python code directly and securely inside BigQuery using standard SQL queries or BigQuery DataFrames (BigFrames) in Python. With this release, Python UDFs are fully supported for production enterprise workloads and completely integrated into BigQuery's billing SKUs. 

Bridging SQL and the Rich Python Ecosystem

BigQuery Managed Python UDFs run on BigQuery-managed serverless resources that automatically scales to billions of rows, without having to set up infrastructure or manage containers. BigQuery automatically handles the compilation, image building, security patching, deployment, and execution of your Python code, making it super simple to use Python functions in your SQL.

Core benefits

• Flexibility: Access the vast Python ecosystem — including top-tier scientific and mathematical libraries like NumPy, SciPy, pandas, scikit-learn and more — directly in your SQL select statements.

• Tight external API integration: Clean and enrich your BigQuery tables in real time by calling external web APIs or Google Cloud services such as Cloud Translation, Gemini Enterprise Agent Platform or custom microservices securely within your queries.

• Fully managed and serverless: BigQuery handles the underlying container infrastructure and auto-scales performance dynamically.

Code example 

Here is an example of a Python UDF that utilizes a popular Python package — beautifulsoup — to remove HTML tags. We use this function to process 

StackOverflow answer bodies that are stored in a BigQuery public table:

How to query it:

Advanced capabilities

For advanced users, Python UDF adds a set of capabilities to tune the performance as well as monitor the usage. Here are some examples. 

Vectorized processing with Pandas PyArrow
To maximize throughput, the GA release supports direct processing of vectorized input as PyArrow RecordBatches. By processing columns of data in bulk rather than row-by-row, PyArrow eliminates Python serialization and conversion overhead, boosting performance by up to 10x for data-intensive calculations.

Configurable container resources
For heavy-duty data science and ML data preparation, you can now provision container memory (up to 16 GB) and CPU (up to 4 vCPUs) per function. This enables memory-intensive workloads (such as loading large serialized models or geospatial datasets) to run directly within the sandbox.

Customizable concurrency
Optimize your throughput and resource efficiency by configuring concurrent requests per container (up to 1,000 concurrent operations). This helps ensure that your scale-out execution is highly cost-effective and performs exceptionally well under heavy parallel loads.

Streaming logs and real-time metrics
Easily debug and monitor your production workloads. The BigQuery console now features a direct link from your query results to real-time CPU, memory, and concurrency metrics in Cloud Monitoring.

Billing

BigQuery Managed Python UDF are billed with BigQuery Services SKU. This SKU is fully eligible for BigQuery spend commitment-based usage discounts (CUDs), allowing you to maximize budget efficiency.

You can also get cost observability through INFORMATION_SCHEMA.JOBS as well as using billing labels MANAGED_ROUTINE_EXECUTION and MANAGED_ROUTINE_BUILD).

See more details in the Pricing section of the documentation. 

Getting started 

To get started with BigQuery Python UDFs, first check out product documentation. 

Then, try out the functions published in the public BigQuery dataset. For example, run the following code in a BigQuery project to tokenize country names data from BigQuery public data. Under the hood, the token UDF utilizes the o200k_base tokenizer library.

Or, try out this code lab to explore some advanced scenarios. 

Then, to learn how to implement other advanced design patterns, we encourage you to explore our official public documentation guides: 

• Calling Google Cloud or online services (with connections): To connect to first-party Google Cloud services such as Gemini Enterprise Agent Platform or Cloud Translation, or external API endpoints securely using Cloud Resource connections, - check out the Call Google Cloud or online services in Python code guide.

• BigQuery DataFrames (BigFrames) Python UDFs:To learn how to write, deploy, and scale custom Python functions natively from standard Jupyter notebook or Colab environments using BigQuery DataFrames, visit the Customize Python functions for BigQuery DataFrames guide.

Bring your Python workflows out of isolation and directly into the heart of your data warehouse today!