Collecting custom data with the DataCapture policy

This page applies to Apigee and Apigee hybrid.

View Apigee Edge documentation.

In addition to the usual API data collected by Apigee, you can also collect custom data using the DataCapture policy. There are two main steps required, as described in the following sections:

1. Create a Data Collector REST resource.
2. Specify the data you want to collect using the DataCapture policy's <Collect> element.

Note:If you have an Apigee organization with Pay-as-you-go billing, you must enable the Apigee API Analytics add-on to use this policy. For more information, see Manage the Apigee API Analytics add-on.

If you are unsure whether you are using a Subscription or Pay-as-you-go Apigee organization, contact your Apigee organization administrator.

Create a Data Collector REST resource

To use the DataCapture policy, you must first create a Data Collector REST resource. To do so, send an API request like the following:

curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -X POST -H "content-type:application/json" \
  -d '
{
  "name": "dc_my_custom_data",
  "description": "Collects data for analysis.",
  "type": "STRING",
}' \
  "https://apigee.googleapis.com/v1/organizations/$PROJECT_ID/datacollectors"

This creates a resource named dc_my_custom_data, which you can use with the DataCapture policy.

Notes:

• The name of the Data Collector resource must begin with the prefix dc_.
• If you use a Data Collector in multiple policies, the captured data will be overwritten by the last policy that executes.

Warning: Do not create two data collectors whose names differ only by case, such as dc_my_data and dc_My_Data, as then exporting data will result in a "duplicate column names" error.

Specify the data to collect

You can specify the data to collect using the DataCapture policy's <Collect> element. There are various ways to do this, as shown in the Examples section in the DataCapure reference page. One way is to specify a flow variable, which stores data passed in API requests and responses. For example, you could collect data that customers enter in a web page form using the flow variable message.formparam.param_name.values, where param_name is the name of a parameter in the form. Suppose the form has a parameter named payment_amount. The corresponding flow variable would be message.formparam.payment_amount.values.

The following code specifies this variable by the ref attribute of the <Collect> element.

<DataCapture name="capturepayment">
    <Capture>
        <DataCollector>dc_my_custom_data</DataCollector>
        <Collect ref="message.form_param.payment_amount.values" >
    </Capture>
</DataCapture>

With this setting, the data entered in the form's payment_amount field will be collected by DataCapture and stored in a variable named dc_my_custom_data.

Creating a custom report from data collected with DataCapture

Once you have set up data collection with the DataCapture policy, you can use the data to create a custom report by the procedure described in Adding a custom report. To do so, when you select a metric for the report, as shown in step 4 of the procedure, choose the variable that has the same name as the Data Collector REST resource you created for DataCapture—for example, dc_my_custom_data— as shown below.

You can also use the same variable as a dimension in a custom report. Since dimensions group data by the values of the variable, it is usually preferable for the variable to be a string that only takes on a limited number of values.