Configure an engine

This page briefly covers concepts behind configuring an engine.

When to tune or re-tune

AML AI tunes hyperparameters when you create an EngineConfig resource. This is a necessary step before an engine trains a model, for any engine version, because the engine config must be specified when you create a model.

For best performance, you should consider engine tuning when any of the following occur:

  • You make significant changes to dataset logic. For example, when any of the following change:
    • The logic by which fields are populated
    • The selection of RECOMMENDED fields that are populated
    • The logic or selection of data provided in the PartySupplementaryData table
  • You're about to have an engine train a model for a new region.

How to tune an engine

To trigger tuning, see Create and manage engine configs.

In particular, you need to select the following:

  • The data to use for engine tuning:

    Specify a dataset and an end time within the date range of the dataset.

    Engine tuning uses labels and features based on complete calendar months up to, but not including, the month of the selected end time. For more information, see Dataset time ranges.

  • The engine version to use for engine tuning:

    Select an engine version that matches the line of business (retail or commercial) that you will use the associated models for.

  • The volume of investigations you expect based on the models:

    Specify partyInvestigationsPerPeriodHint. This is used by engine tuning, training, and backtesting to ensure AML AI delivers performance at your investigation volume.

Engine tuning output

Engine tuning generates an EngineConfig resource, which can be used to create a Model resource.

The engine config metadata contains the following metrics. In particular, these metrics show you the following:

  • Expected performance gain from engine tuning versus using the default hyperparameters

  • Any large changes in which feature families the dataset supports (between engine tuning, training, evaluation, and prediction)

Metric name Metric description Example metric value
ExpectedRecallPreTuning Recall metric measured on a test set when using default hyperparameters of the engine version.

This recall measurement assumes the number of investigations per month specified in partyInvestigationsPerPeriodHint. 

{
  "recallValues": [
    {
      "partyInvestigationsPerPeriod": 5000,
      "recallValue": 0.72,
      "scoreThreshold": 0.42,
    },
  ],
}
ExpectedRecallPostTuning Recall metric measured on a test set when using tuned hyperparameters.

This recall measurement assumes the number of investigations per month specified in partyInvestigationsPerPeriodHint. 

{
  "recallValues": [
    {
      "partyInvestigationsPerPeriod": 5000,
      "recallValue": 0.80,
      "scoreThreshold": 0.43,
    },
  ],
}
Missingness

Share of missing values across all features in each feature family.

Ideally, all AML AI feature families should have a Missingness near to 0. Exceptions may occur where the data underlying those feature families is unavailable for integration.

A significant change in this value for any feature family between tuning, training, evaluation, and prediction can indicate inconsistency in the datasets used. 

{
  "featureFamilies": [
    {
      "featureFamily": "unusual_wire_credit_activity",
      "missingnessValue": 0.00,
    },
    ...
    ...
    {
      "featureFamily": "party_supplementary_data_id_3",
      "missingnessValue": 0.45,
    },
  ],
}