The following best practices will help you plan and use Vertex AI Feature Store in various scenarios. This guide is not intended to be exhaustive..
Model features that jointly describe multiple entities
Some features might apply to multiple entity types. For example, you might have a calculated value that records clicks per product by user. This feature jointly describes product-user pairs.
The best practice, in this case, is to create a separate entity type to group
shared features. You can create an entity type, such as
contain shared features.
For the entity IDs, concatenate the IDs of the individual entities, such as the entity IDs of the individual product and user. The only requirement is that the IDs must be strings. These combined entity types are referred to as composite entity types.
For more information, see creating an entity type.
Use IAM policies to control access across multiple teams
Use IAM roles and policies to set different levels of access to different groups of users. For example, ML researchers, data scientists, DevOps, and site reliability engineers all require access to the same featurestore, but their level of access can differ. For example, DevOps users might require permissions to manage a featurestore, but they don't require access to the contents of the featurestore.
You can also restrict access to a particular featurestore or entity type by using resource-level IAM policies.
As an example, imagine that your organization includes the following personas. Because each persona requires a different level of access, each persona is assigned a different predefined IAM role. You can also create and use your own custom roles.
|ML researcher or business analyst||Users who only view data on specific entity types||
|Data scientists or data engineers||Users who work with specific entity type resources. For the resources they own, they can delegate access to other principals.||
|IT or DevOps||Users who must maintain and tune the performance of specific featurestores but don't require access to the data.||
|Automated data import pipeline||Applications that write data to specific entity types.||
|Site reliability engineer||Users who manage particular featurestores or all featurestores in a project||
|Global (any Vertex AI Feature Store user)||
Allow users to view and search for existing features. If they find a feature they want to work with, they can request access from the feature owners.
For Google Cloud console users, this role is also required to view the Vertex AI Feature Store landing page, ingestion jobs page, and batch serving jobs page.
Monitor and tune resources accordingly to optimize batch ingestion
Batch ingestion jobs require workers to process and write data, which can increase the CPU utilization of your featurestore and affect online serving performance. If preserving online serving performance is a priority, start with one worker for every ten online serving nodes. During ingestion, monitor the CPU usage of the online storage. If CPU usage is lower than expected, increase the number of workers for future batch ingestion jobs to increase throughput. If CPU usage is higher than expected, increase the number of online serving nodes to increase CPU capacity or lower the batch ingestion worker count, both of which can lower CPU usage.
If you do increase the number of online serving nodes, note that Vertex AI Feature Store takes roughly 15 minutes to reach optimal performance after you make the update.
For more information about featurestore monitoring, see Cloud Monitoring metrics.
disableOnlineServing field when backfilling historical data
Backfilling is the process of ingesting historical feature values and don't impact the most recent feature values. In this case, you can disable online serving, which skips any changes to the online store. For more information, see Backfill historical data.
Use autoscaling to reduce costs during load fluctuations
If you use Vertex AI Feature Store extensively and encounter frequent load fluctuations in your traffic patterns, use autoscaling to optimize costs. Autoscaling lets Vertex AI Feature Store review traffic patterns and automatically adjust the number of nodes up or down depending on CPU utilization, instead of maintaining a high node count. This option works well for traffic patterns that encounter gradual growth and decline.
For more information about autoscaling, see Scaling options.
Test the performance of online serving nodes for real-time serving
You can ensure the performance of your featurestore during real-time online serving by testing the performance of your online serving nodes. You can perform these tests based on various benchmarking parameters, such as QPS, latency, and API. Follow these guidelines to test the performance of online serving nodes:
Run all test clients from the same region, preferably on Compute Engine or Google Kubernetes Engine: This prevents discrepancies due to network latency resulting from hops across regions.
Use the gRPC API in the SDK: The gRPC API performs better than the REST API. If you need to use the REST API, enable the HTTP keep-alive option to reuse HTTP connections. Otherwise, each request results in the creation of a new HTTP connection, which increases the latency.
Run longer duration tests: Run tests with longer duration (15 minutes or more) and a minimum of 5 QPS to calculate more accurate metrics.
Add a "warm-up" period: If you start testing after a period of inactivity, you might observe high latency while connections are reestablished. To account for the initial period of high latency, you can designate this period as a "warm-up period", when the initial data reads are ignored. As an alternative, you can send a low but consistent rate of artificial traffic to the featurestore to keep the connection active.
If required, enable autoscaling: If you anticipate gradual growth and decline in your online traffic, enable autoscaling. If you choose autoscaling, Vertex AI automatically changes the number of online serving nodes based on CPU utilization.
Learn Vertex AI Feature Store best practices for implementing custom-trained ML models on Vertex AI.