This page shows you how to create and manage AML AI backtest
prediction results. Backtest prediction uses a trained model on existing
historical data. For these months, the dataset must contain a party with
suspicious activity or investigation start dates leading to an AML_EXIT
event,
for more information on how AML AI labels various parties, see
lifecycle of a risk case.
The same dataset can be used for training and backtest as long as the training end date is before the backtest dates.
Before you begin
-
To get the permissions that you need to create and manage backtest results, ask your administrator to grant you the Financial Services Admin (
financialservices.admin
) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.You might also be able to get the required permissions through custom roles or other predefined roles.
- Create an instance
- Create a model
- Create a dataset
Create backtest results
Some API methods return a long-running operation (LRO). These methods are asynchronous and return an Operation object; for details, see REST Reference. The operation might not be completed when the method returns a response. For these methods, send the request and then check for the result. In general, all POST, PUT, UPDATE, and DELETE operations are long-running.
Send the request
To create backtest results, use the
projects.locations.instances.backtestResults.create
method.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
INSTANCE_ID
: a user-defined identifier for the instanceBACKTEST_RESULTS_ID
: a user-defined identifier for the backtest resultsMODEL_ID
: a user-defined identifier for the modelBACKTEST_DATASET_ID
: the user-defined identifier for the dataset used for the backtest; tables should not have the training label columnsBACKTEST_END_DATE
: The latest time from which data is used to generate features for backtests. This date should be the same or earlier than the end time of the datasets. Use RFC3339 UTC "Zulu" format (for example,2014-10-02T15:01:23Z
).PARTY_INVESTIGATIONS_PER_PERIOD_HINT
: A number that gives the tuner a hint on the number of parties from this data that will be investigated per period (monthly). This is used to control how the model is evaluated. For example, when trying AML AI for the first time, we recommend setting this to the number of parties investigated in an average month, based on alerts from your existing automated alerting system.
Request JSON body:
{ "model": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/models/MODEL_ID", "dataset": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/datasets/BACKTEST_DATASET_ID", "endTime": "BACKTEST_END_DATE", "backtestPeriods": 5, "performanceTarget": { "partyInvestigationsPerPeriodHint": "PARTY_INVESTIGATIONS_PER_PERIOD_HINT" } }
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
cat > request.json << 'EOF' { "model": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/models/MODEL_ID", "dataset": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/datasets/BACKTEST_DATASET_ID", "endTime": "BACKTEST_END_DATE", "backtestPeriods": 5, "performanceTarget": { "partyInvestigationsPerPeriodHint": "PARTY_INVESTIGATIONS_PER_PERIOD_HINT" } } EOF
Then execute the following command to send your REST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults?backtest_result_id=BACKTEST_RESULTS_ID"
PowerShell
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
@' { "model": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/models/MODEL_ID", "dataset": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/datasets/BACKTEST_DATASET_ID", "endTime": "BACKTEST_END_DATE", "backtestPeriods": 5, "performanceTarget": { "partyInvestigationsPerPeriodHint": "PARTY_INVESTIGATIONS_PER_PERIOD_HINT" } } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults?backtest_result_id=BACKTEST_RESULTS_ID" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.financialservices.v1.OperationMetadata", "createTime": CREATE_TIME, "target": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "verb": "create", "requestedCancellation": false, "apiVersion": "v1" }, "done": false }
Check for the result
To check if backtest results have been created, use the
projects.locations.operations.get
method. If the response contains "done": false
, repeat the command until the
response contains "done": true
. These operations can take a few minutes to
several hours to complete.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
OPERATION_ID
: the identifier for the operation
To send your request, choose one of these options:
curl
Execute the following command:
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.financialservices.v1.OperationMetadata", "createTime": "2023-03-14T15:52:55.358979323Z", "endTime": "2023-03-14T16:52:55.358979323Z", "target": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "verb": "create", "requestedCancellation": false, "apiVersion": "v1" }, "done": true, "response": { "@type": "type.googleapis.com/dataresidency.monitoring.DataResidencyAugmentedView", "tpIds": [ "i608e8cf4abb2a7d9-tp" ] } }
Export metadata
To export metadata from backtest results, use the
projects.locations.instances.backtestResults.exportMetadata
method.
For more information, see Exported metadata in the AML output data model.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
INSTANCE_ID
: the user-defined identifier for the instanceBACKTEST_RESULTS_ID
: a user-defined identifier for the backtest resultsBQ_OUTPUT_DATASET_NAME
: a BigQuery dataset in which to export a table that describes the structured metadata of the backtest resultsSTRUCTURED_METADATA_TABLE
: the table to write the structured metadata toWRITE_DISPOSITION
: the action that occurs if the destination table already exists; use one of the following values:-
WRITE_EMPTY
: Only export data if the BigQuery table is empty. -
WRITE_TRUNCATE
: Erase all existing data in the BigQuery table before writing to the table.
-
Request JSON body:
{ "structuredMetadataDestination": { "tableUri": "bq://PROJECT_ID.BQ_OUTPUT_DATASET_NAME.STRUCTURED_METADATA_TABLE", "writeDisposition": "WRITE_DISPOSITION" } }
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
cat > request.json << 'EOF' { "structuredMetadataDestination": { "tableUri": "bq://PROJECT_ID.BQ_OUTPUT_DATASET_NAME.STRUCTURED_METADATA_TABLE", "writeDisposition": "WRITE_DISPOSITION" } } EOF
Then execute the following command to send your REST request:
curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID:exportMetadata"
PowerShell
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
@' { "structuredMetadataDestination": { "tableUri": "bq://PROJECT_ID.BQ_OUTPUT_DATASET_NAME.STRUCTURED_METADATA_TABLE", "writeDisposition": "WRITE_DISPOSITION" } } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID:exportMetadata" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.financialservices.v1.OperationMetadata", "createTime": CREATE_TIME, "target": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "verb": "exportMetadata", "requestedCancellation": false, "apiVersion": "v1" }, "done": false }
For more information on how to get the result of the long-running operation (LRO), see Check for the result.
Get backtest results
To get backtest results, use the
projects.locations.instances.backtestResults.get
method.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
INSTANCE_ID
: the user-defined identifier for the instanceBACKTEST_RESULTS_ID
: the user-defined identifier for the backtest results
To send your request, choose one of these options:
curl
Execute the following command:
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "createTime": CREATE_TIME, "updateTime": UPDATE_TIME, "state": "ACTIVE", "dataset": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/datasets/BACKTEST_DATASET_ID", "model": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/models/MODEL_ID", "endTime": "BACKTEST_END_DATE", "performanceTarget": { "partyInvestigationsPerPeriodHint": "PARTY_INVESTIGATIONS_PER_PERIOD_HINT" }, "lineOfBusiness": "RETAIL" }
Update backtest results
To update backtest results, use the
projects.locations.instances.backtestResults.patch
method.
Only the labels
field in backtest results can be updated. The following
example updates the key-value pair
user labels
associated with the backtest results.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
INSTANCE_ID
: the user-defined identifier for the instanceBACKTEST_RESULTS_ID
: the user-defined identifier for the backtest resultsKEY
: The key in a key-value pair used to organize backtest results. Seelabels
for more information.VALUE
: The value in a key-value pair used to organize backtest results. Seelabels
for more information.
Request JSON body:
{ "labels": { "KEY": "VALUE" } }
To send your request, choose one of these options:
curl
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
cat > request.json << 'EOF' { "labels": { "KEY": "VALUE" } } EOF
Then execute the following command to send your REST request:
curl -X PATCH \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json; charset=utf-8" \
-d @request.json \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID?updateMask=labels"
PowerShell
Save the request body in a file named request.json
.
Run the following command in the terminal to create or overwrite
this file in the current directory:
@' { "labels": { "KEY": "VALUE" } } '@ | Out-File -FilePath request.json -Encoding utf8
Then execute the following command to send your REST request:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method PATCH `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID?updateMask=labels" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.financialservices.v1.OperationMetadata", "createTime": CREATE_TIME, "target": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "verb": "update", "requestedCancellation": false, "apiVersion": "v1" }, "done": false }
For more information on how to get the result of the long-running operation (LRO), see Check for the result.
List the backtest results
To list the backtest results for a given instance, use the
projects.locations.instances.backtestResults.list
method.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
INSTANCE_ID
: the user-defined identifier for the instance
To send your request, choose one of these options:
curl
Execute the following command:
curl -X GET \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "backtestResults": [ { "name": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "createTime": CREATE_TIME, "updateTime": UPDATE_TIME, "state": "ACTIVE", "dataset": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/datasets/BACKTEST_DATASET_ID", "model": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/models/MODEL_ID", "endTime": "BACKTEST_END_DATE", "performanceTarget": { "partyInvestigationsPerPeriodHint": "PARTY_INVESTIGATIONS_PER_PERIOD_HINT" }, "lineOfBusiness": "RETAIL" } ] }
Delete backtest results
To delete backtest results, use the
projects.locations.instances.backtestResults.delete
method.
Before using any of the request data, make the following replacements:
PROJECT_ID
: your Google Cloud project ID listed in the IAM SettingsLOCATION
: the location of the instance; use one of the supported regionsShow locationsus-central1
us-east1
asia-south1
europe-west1
europe-west2
europe-west4
northamerica-northeast1
southamerica-east1
australia-southeast1
INSTANCE_ID
: the user-defined identifier for the instanceBACKTEST_RESULTS_ID
: the user-defined identifier for the backtest results
To send your request, choose one of these options:
curl
Execute the following command:
curl -X DELETE \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
"https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID"
PowerShell
Execute the following command:
$cred = gcloud auth print-access-token
$headers = @{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method DELETE `
-Headers $headers `
-Uri "https://financialservices.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID" | Select-Object -Expand Content
You should receive a JSON response similar to the following:
{ "name": "projects/PROJECT_ID/locations/LOCATION/operations/OPERATION_ID", "metadata": { "@type": "type.googleapis.com/google.cloud.financialservices.v1.OperationMetadata", "createTime": CREATE_TIME, "target": "projects/PROJECT_ID/locations/LOCATION/instances/INSTANCE_ID/backtestResults/BACKTEST_RESULTS_ID", "verb": "delete", "requestedCancellation": false, "apiVersion": "v1" }, "done": false }
For more information on how to get the result of the long-running operation (LRO), see Check for the result.