Manage featurestores

Learn how to create, list, describe, update, and delete featurestores. A featurestore is a top-level container for entity types, features, and feature values.

Online and offline storage

Vertex AI Feature Store uses two storage methods classified as online storage and offline storage, which are priced differently. All featurestores have offline storage and optionally, online storage.

Online storage retains the latest timestamp values of your features to efficiently handle online serving requests. When you run an ingestion job by using the API, you can control the job if the data is written to the online store. Skipping the online store prevents any load on the online serving nodes. For example, when you run backfill jobs, you can disable writes to the online store and write only to the offline store. For more information, see the disableOnlineServing flag in the API reference.

Vertex AI Feature Store uses offline storage to permanently store data until the data reaches the retention limit or until you delete the data. You can control offline storage costs by managing how much data you keep.

Use the Cloud console to view the amount of online and offline storage you are currently using. View your featurestore's Total online storage and Total offline storage monitoring metrics to determine your usage.

Online serving nodes

Online serving nodes provide the compute resources used to store and serve feature values for low-latency online serving. These nodes are always running even when they aren't serving data. You are charged for each node hour.

If you don't require online serving, create or modify featurestores to have zero nodes, which prevent you from incurring charges for online serving nodes.

The number of online serving nodes that you require is directly proportional to the following two factors:

  • The number of online serving requests (queries per second) that the featurestore receives
  • The number of ingestion jobs that write to online storage

Both factors contribute to the CPU utilization and performance of the nodes. From the Cloud console, view the metrics of the following:

  • Queries per second - your featurestore's queries per second
  • Node count - number of your online serving nodes
  • CPU utilization - CPU utilization of your nodes

If CPU utilization is consistently high, consider increasing the number of online serving nodes for your featurestore.

Scaling Options

You can switch between the following two options to configure your number of online serving nodes:

  • Autoscaling (preview)
  • Allocating a fixed node count

With autoscaling, the featurestore automatically changes the number of nodes based on CPU utilization. Autoscaling reviews traffic patterns to maintain performance and optimize your cost by adding nodes when the traffic increases and removing nodes when the traffic decreases. Autoscaling performs well for traffic patterns that experience gradual growth and decline. However, autoscaling is not effective for traffic patterns that encounter frequent bursts of traffic. See Additional Considerations for more details.

Allocating a fixed node count will maintain a consistent number of nodes regardless of traffic patterns. The fixed node count will keep costs predictable, and the nodes should perform well when there are enough nodes to handle the traffic. You can manually change the fixed node count to handle changes in traffic patterns.

Additional Considerations

If you choose autoscaling, there are four additional points to consider that include:

  • After adding online serving nodes, the online store needs time to rebalance the data. It can take up to 20 minutes under load before you see a significant improvement in performance. As a result, scaling the number of nodes might not help for short bursts of traffic. This limitation applies for both manual scaling and autoscaling.

  • If you update the number of online serving nodes from 0 to 1 or greater, Vertex AI Feature Store doesn't migrate any data to the online store. Responses to online serving requests are returned empty as if no data was ingested. To populate the online store, ingest your data again. For example, you can export your existing data and then ingest it again. When you provision online serving nodes, wait for the long-running operation to complete before ingesting data. While a node is being provisioned, in-progress ingestion jobs don't write to the online store.

  • If you update the number of online serving nodes from 1 or greater to 0, data from the online store is deleted and cannot be restored. Zero online serving nodes deletes the entire online store. You can't, for example, temporarily turn down your online store and then restore it. However, reducing the number of online serving nodes to 0 does not impact the offline store.

  • Submitting online serving requests to a featurestore without online serving nodes returns an error.

Create a featurestore

Create a featurestore resource to contain entity types and features. The location of your featurestore must be in the same location as your source data. For example, if your featurestore is in us-central, you can ingest data from files in Cloud Storage buckets that are in us-central1 or in the US multi-region location, though source data from dual-region buckets is not supported. Similarly for BigQuery, you can ingest data from tables that are in us-central1 or in the US multi-region location. For more information, see Source data requirements.

Vertex AI Feature Store availability can vary by location. For more information, see Feature availability.

Web UI

Use another method. You cannot create a featurestore from the Cloud console.

REST & CMD LINE

To create a featurestore, send a POST request by using the featurestores.create method.

The following sample creates a featurestore with a fixed node count of 1. The node count specifies the number of online serving nodes, which affects the number of online serving requests that the featurestore can handle. The latency could increase if the number of nodes cannot support the number of incoming requests.

Before using any of the request data, make the following replacements:

  • LOCATION: Region where the featurestore is created. For example, us-central1.
  • PROJECT: Your project ID or project number.
  • FEATURESTORE_ID: ID for the featurestore.

HTTP method and URL: 

POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores?featurestoreId=FEATURESTORE_ID

Request JSON body: 

{
  "online_serving_config": {
    "fixed_node_count": 1
  },
  "labels": {
    "environment": "testing"
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file called request.json, and execute the following command: 

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores?featurestoreId=FEATURESTORE_ID"

PowerShell

Save the request body in a file called request.json, and execute the following command: 

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -InFile request.json `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores?featurestoreId=FEATURESTORE_ID" | Select-Object -Expand Content

You should see output similar to the following. You can use the OPERATION_ID in the response to get the status of the operation.

{
"name": "projects/PROJECT_NUMBER/locations/LOCATION/featurestores/FEATURESTORE_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1.CreateFeaturestoreOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-02-26T00:44:40.722474Z",
      "updateTime": "2021-02-26T00:44:40.722474Z"
    }
  }
}

Python

To learn how to install and use the client library for Vertex AI, see Vertex AI client libraries. For more information, see the Vertex AI Python API reference documentation. 

from google.cloud import aiplatform


def create_featurestore_sample(
    project: str,
    location: str,
    featurestore_id: str,
    online_store_fixed_node_count: int = 1,
    sync: bool = True,
):

    aiplatform.init(project=project, location=location)

    fs = aiplatform.Featurestore.create(
        featurestore_id=featurestore_id,
        online_store_fixed_node_count=online_store_fixed_node_count,
        sync=sync,
    )

    fs.wait()

    return fs

Additional languages

You can install and use the following Vertex AI client libraries to call the Vertex AI API. Cloud Client Libraries provide an optimized developer experience by using each supported language's natural conventions and styles.

Create a featurestore that uses a CMEK

Before you begin, if you don't have an existing CMEK, use Cloud Key Management Service to configure a customer-managed encryption key and set up permissions. The following sample creates a featurestore that uses a CMEK.

Web UI

Use another method. You cannot create a featurestore from the Cloud console.

REST & CMD LINE

Before using any of the request data, make the following replacements:

  • LOCATION: Region where the featurestore is created. For example, us-central1.
  • PROJECT: Your project ID or project number.
  • FEATURESTORE_ID: ID for the featurestore.
  • CMEK_PROJECT: The project ID or project number that contains your CMEK.
  • KEY_RING: The name of the Cloud Key Management Service key ring that your encryption key is on.
  • KEY_NAME: The name of the encryption key to use.

HTTP method and URL: 

POST https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores?featurestoreId=FEATURESTORE_ID

Request JSON body: 

{
  "online_serving_config": {
    "fixed_node_count": 1
  },
  "encryption_spec":{
    "kms_key_name": "projects/CMEK_PROJECT/locations/LOCATION/keyRings/KEY_RING/cryptoKeys/KEY_NAME"
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file called request.json, and execute the following command: 

curl -X POST \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores?featurestoreId=FEATURESTORE_ID"

PowerShell

Save the request body in a file called request.json, and execute the following command: 

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method POST `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -InFile request.json `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores?featurestoreId=FEATURESTORE_ID" | Select-Object -Expand Content

You should see output similar to the following. You can use the OPERATION_ID in the response to get the status of the operation.

{
"name": "projects/PROJECT_NUMBER/locations/LOCATION/featurestores/FEATURESTORE_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1.CreateFeaturestoreOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-02-26T00:44:40.722474Z",
      "updateTime": "2021-02-26T00:44:40.722474Z"
    }
  }
}

List featurestores

List all featurestores in a project.

Web UI

  1. In the Vertex AI section of the Google Cloud console, go to the Features page.

    Go to the Features page

  2. Select a region from the Region drop-down list.
  3. In the features table, view the Featurestore column to see the featurestores in your project for the selected region.

REST & CMD LINE

To list featurestores for a particular region in your project, send a GET request by using the featurestores.list method.

Before using any of the request data, make the following replacements:

  • LOCATION: Region where the featurestore is located, such as us-central1.
  • PROJECT: Your project ID or project number.

HTTP method and URL: 

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores

To send your request, choose one of these options:

curl

Execute the following command: 

curl -X GET \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores"

PowerShell

Execute the following command: 

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "featurestores": [
    {
      "name": "projects/PROJECT_NUMBER/locations/us-central1/featurestores/test",
      "createTime": "2021-02-26T00:44:44.216805Z",
      "updateTime": "2021-02-26T00:44:44.364916Z",
      "etag": "AMEw9yNL0s7qZh8lZVZ5T3BEuhoEgFR7JmjbbCSAkRZjeKDXkkIYnxxA4POe5BWT8cCn",
      "labels": {
        "environment": "testing"
      },
      "onlineServingConfig": {
        "fixedNodeCount": 2
      },
      "state": "STABLE"
    },
    {
      "name": "projects/PROJECT_NUMBER/locations/us-central1/featurestores/featurestore_demo",
      "createTime": "2021-02-25T00:39:40.598781Z",
      "updateTime": "2021-02-25T00:39:40.744038Z",
      "etag": "AMEw9yO_e0vm-9W_yeCz4rJm-XnnEMYQ-vQesevxya_sz-FckuysnDwo3cEXHdWWSeda",
      "labels": {
        "environment": "testing"
      },
      "onlineServingConfig": {
        "fixedNodeCount": 3
      },
      "state": "STABLE"
    }
  ]
}

Additional languages

You can install and use the following Vertex AI client libraries to call the Vertex AI API. Cloud Client Libraries provide an optimized developer experience by using each supported language's natural conventions and styles.

View featurestore details

Get details about a featurestore such as its name and online serving configuration. If you use the Cloud console, you can also view Cloud Monitoring metrics for featurestores.

Web UI

  1. In the Vertex AI section of the Google Cloud console, go to the Features page.

    Go to the Features page

  2. Select a region from the Region drop-down list.
  3. In the features table, view the Featurestore column and find the featurestore that you want to view information for.
  4. Click the name of the featurestore to view its Monitoring metrics.
  5. Click the Properties tab to view the featurestore's online serving configuration.

REST & CMD LINE

To get details about a single featurestore, send a GET request by using the featurestores.get method.

Before using any of the request data, make the following replacements:

  • LOCATION: Region where the featurestore is located, such as us-central1.
  • PROJECT: Your project ID or project number.
  • FEATURESTORE_ID: ID of the featurestore.

HTTP method and URL: 

GET https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID

To send your request, choose one of these options:

curl

Execute the following command: 

curl -X GET \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID"

PowerShell

Execute the following command: 

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method GET `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT_NUMBER/locations/us-central1/featurestores/FEATURESTORE_ID",
  "createTime": "2021-02-25T00:39:40.598781Z",
  "updateTime": "2021-02-25T00:39:40.744038Z",
  "etag": "AMEw9yNy_b4IaMIvw1803ZT38cpUtjfwlyLkR709oBCY6pQrm6dHophLcqhrvsNqkQQZ",
  "onlineServingConfig": {
    "fixedNodeCount": 3
  },
  "state": "STABLE"
}

Additional languages

You can install and use the following Vertex AI client libraries to call the Vertex AI API. Cloud Client Libraries provide an optimized developer experience by using each supported language's natural conventions and styles.

Update a featurestore

Update a featurestore, for example, to change the number of online serving nodes or update labels on a featurestore.

Web UI

You can update only the number of online serving nodes. To update labels, use the API.

  1. In the Vertex AI section of the Google Cloud console, go to the Features page.

    Go to the Features page

  2. Select a region from the Region drop-down list.
  3. In the features table, view the Featurestore column and click the name of the featurestore to update.
  4. Click Edit configuration to open the Edit featurestore configuration pane.
  5. Edit the featurestore configuration.
  6. Click Update to apply your changes.

REST & CMD LINE

To update a featurestore, send a PATCH request by using the featurestores.patch method.

The following sample updates the number of online serving nodes to 2 for the featurestore. All other settings remain the same.

Before using any of the request data, make the following replacements:

  • LOCATION: Region where the featurestore is located, such as us-central1.
  • PROJECT: Your project ID or project number.
  • FEATURESTORE_ID: ID of the featurestore.

HTTP method and URL: 

PATCH https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID

Request JSON body: 

{
  "online_serving_config": {
    "fixed_node_count": 2
  }
}

To send your request, choose one of these options:

curl

Save the request body in a file called request.json, and execute the following command: 

curl -X PATCH \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID"

PowerShell

Save the request body in a file called request.json, and execute the following command: 

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method PATCH `
  -Headers $headers `
  -ContentType: "application/json; charset=utf-8" `
  -InFile request.json `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID" | Select-Object -Expand Content

You should see output similar to the following. You can use the OPERATION_ID in the response to get the status of the operation.

{
"name": "projects/PROJECT_NUMBER/locations/LOCATION/featurestores/FEATURESTORE_ID/operations/OPERATION_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1.UpdateFeaturestoreOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-03-18T21:12:08.373664Z",
      "updateTime": "2021-03-18T21:12:08.373664Z"
    }
  }
}

Additional languages

You can install and use the following Vertex AI client libraries to call the Vertex AI API. Cloud Client Libraries provide an optimized developer experience by using each supported language's natural conventions and styles.

Delete a featurestore

Delete a featurestore. If the featurestore includes existing entity types and features, enable the force query parameter to delete the featurestore and all of its contents.

Web UI

Use another method. You cannot delete a featurestore from the Cloud console.

REST & CMD LINE

To delete a featurestore and all of its content, send a DELETE request by using the featurestores.delete method.

Before using any of the request data, make the following replacements:

  • LOCATION: Region where the featurestore is located, such as us-central1.
  • PROJECT: Your project ID or project number.
  • FEATURESTORE_ID: ID of the featurestore.
  • BOOLEAN: Whether to delete the featurestore even if it contains entity types and features. The force query parameter is optional and is false by default.

HTTP method and URL: 

DELETE https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID?force=BOOLEAN

To send your request, choose one of these options:

curl

Execute the following command: 

curl -X DELETE \
-H "Authorization: Bearer "$(gcloud auth application-default print-access-token) \
"https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID?force=BOOLEAN"

PowerShell

Execute the following command: 

$cred = gcloud auth application-default print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }

Invoke-WebRequest `
  -Method DELETE `
  -Headers $headers `
  -Uri "https://LOCATION-aiplatform.googleapis.com/v1/projects/PROJECT/locations/LOCATION/featurestores/FEATURESTORE_ID?force=BOOLEAN" | Select-Object -Expand Content

You should receive a JSON response similar to the following:

{
  "name": "projects/PROJECT_NUMBER/locations/LOCATION/operations/OPERATIONS_ID",
  "metadata": {
    "@type": "type.googleapis.com/google.cloud.aiplatform.v1.DeleteOperationMetadata",
    "genericMetadata": {
      "createTime": "2021-05-03T17:50:21.813112Z",
      "updateTime": "2021-05-03T17:50:21.813112Z"
    }
  },
  "done": true,
  "response": {
    "@type": "type.googleapis.com/google.protobuf.Empty"
  }
}

Python

To learn how to install and use the client library for Vertex AI, see Vertex AI client libraries. For more information, see the Vertex AI Python API reference documentation. 

from google.cloud import aiplatform


def delete_featurestore_sample(
    project: str,
    location: str,
    featurestore_name: str,
    sync: bool = True,
    force: bool = True,
):

    aiplatform.init(project=project, location=location)

    fs = aiplatform.featurestore.Featurestore(featurestore_name=featurestore_name)
    fs.delete(sync=sync, force=force)

Additional languages

You can install and use the following Vertex AI client libraries to call the Vertex AI API. Cloud Client Libraries provide an optimized developer experience by using each supported language's natural conventions and styles.

What's next