Custom metrics are metrics defined by users. This page describes how to create metric descriptors for custom metrics and how to write custom metric data.
Custom metrics use the same elements that the built-in Cloud Monitoring metrics use:
- A set of data points.
- Metric-type information, which tells you what the data points represent.
- Monitored-resource information, which tells you where the data points originated.
Custom metrics can be used in the same way as built-in metrics. That is, you can create charts and alerts for your custom metric data.
Before you begin
If you aren't familiar with the structures that underlie all metrics, see Metrics, time series, and resources.
Getting started with custom metrics
To use a custom metric, you must have a metric descriptor for your new metric type. The metric descriptor defines how your data is organized. Before you start writing custom metric data, determine how you want to use the data you collect, and ensure that the data organization supports those objectives.
Creating a metric descriptor
Cloud Monitoring can create the metric descriptor for you automatically,
or you can use the metricDescriptors.create
method
in the Monitoring API to create it yourself. You can't manipulate
metric descriptors by using the Google Cloud Console for Monitoring.
To have Cloud Monitoring create the metric descriptor for you, you simply write time series data for your metric, and Cloud Monitoring creates a descriptor based on the data you are writing. For more information, see Auto-creation of metric descriptors. There are limits to auto-creation, so it's helpful to know what information goes into a metric definition. After you have a new custom metric descriptor, whether you or Monitoring created it, you can use the metric descriptor with the metric descriptor API methods and the time series API methods.
Whether you explicitly create the metric descriptor or let Monitoring create it for you based on your data, you still need to ensure the organization of the data matches how you want to query it.
Organizing your data
Suppose you have a program that runs on a single machine, and this program
calls auxiliary programs A
and B
. You want to count how often programs
A
and B
are called. You also want to know when program A
is called more
than 10 times per minute and when program B
is called more than 5 times per
minute.
To simplify the discussion, assume that you have a single Google Cloud
project and you plan to use the global
monitored resource. For more
information about this choice, see
Monitored resources for custom metrics.
Consider the following approaches:
You use two custom metrics:
Metric-type-A
counts calls to programA
andMetric-type-B
counts calls to programB
. In this case,Metric-type-A
contains 1 time series, andMetric-type-B
contains 1 time series.With this data model, you can create a single alerting policy with two conditions, or you can create two alerting policies each with one condition. An alerting policy can support multiple conditions, but it has a single configuration for the notification channels.
This model might be appropriate when you aren't interested in similarities in the data between the activities being monitored. In this example, the activities are the rate of calls to programs
A
andB
.You use a single custom metric and use a label to store a program identifier. For example, the label might store the value
A
orB
. Because Monitoring creates a time series for each unique combination of labels, there are two time series for this single metric type.As with the previous model, you can create a single alerting policy or two alerting policies. However, the conditions for the alerting policy are more complicated. To create a condition that generates an incident if the rate of calls for program
A
exceeds a threshold, then the condition needs to include a filter that includes data points whose label value isA
and excludes all other data points.One advantage of this model is that it is simple to compute ratios. For example, you can determine how much of the total is due to calls to
A
.You use a single custom metric to count the number of calls, but you don't use a label to record which program was called. In this model, there is a single time series that combines the data for the two programs. However, you can't create an alerting policy that meets your objectives because the data for two programs can't be separated.
Naming custom metrics
When you create a custom metric, you define a string identifier that represents
the metric type. This string must be unique among the custom metrics in your
Google Cloud project and it must use a prefix that marks the metric as a
user-defined metric. For Monitoring, the allowable prefixes are
custom.googleapis.com/
and external.googleapis.com/prometheus
.
The prefix is followed by a name that describes what you are collecting.
For details on the recommended way to name a custom metric, see
Naming conventions.
Here are examples of the two kinds of identifiers for metric types:
custom.googleapis.com/cpu_utilization
custom.googleapis.com/instance/cpu/utilization
All metric types have globally unique identifiers called resource names. The structure of a resource name for a metric type is:
projects/[PROJECT_ID]/metricDescriptors/[METRIC_TYPE]
where METRIC_TYPE
is the string identifier of the metric type.
If the previous custom metric examples were created in project my-project-id
,
then their resource names for these metrics would be the following:
projects/my-project-id/metricDescriptors/custom.googleapis.com/cpu_utilization
projects/my-project-id/metricDescriptors/custom.googleapis.com/instance/cpu/utilization
Name or type? In the metric descriptor, the name
field stores the
metric type's resource name and the type
field stores the METRIC_TYPE
string.
Choosing a monitored-resource type
When you write your data to a time series, you must indicate where the data came from. To do this, you choose a monitored-resource type that represents where your data comes from, and then use that to describe the specific origin. The monitored resource isn't part of the metric type; instead, the time series to which you write data includes a reference to the metric type, which describes the data, and a reference to the monitored resource, which indicates where the data originated.
Consider the monitored resource before creating your metric descriptor. The monitored-resource type you use affects which labels you need to include in the metric descriptor. For example, if you use a VM instance monitored resource with your data, then you won't need a metric label to specify the instance. For more information, see Monitored resources for custom metrics.
Each of your metric's data points must be associated with a monitored resource object. Points from different monitored resource objects are held in different time series.
Monitored resources for custom metrics
You can use only the following monitored-resource types in your custom metrics.
aws_ec2_instance
: Amazon EC2 instance.dataflow_job
: Dataflow job.gae_instance
: App Engine instance.gce_instance
: Compute Engine instance.generic_node
: User-specified computing node.generic_task
: User-defined task.gke_container
: GKE container instance.global
: Use this resource when no other resource type is suitable. For most use cases,generic_node
orgeneric_task
are better choices thanglobal
.k8s_cluster
: Kubernetes cluster.k8s_container
: Kubernetes container.k8s_node
: Kubernetes node.k8s_pod
: Kubernetes pod.
A common practice is to use the monitored resource objects that represent the physical resources where your application code is running. This approach has several advantages:
- You get better performance compared with using a single resource type.
- You avoid out-of-order data caused by multiple processes writing to the same time series.
- Your custom metric data can be grouped with other metric data from the same resources.
global
versus generic resources
The generic_task
and generic_node
resource types are useful in situations
where none of the more specific resource types are appropriate.
The generic_task
type is useful for defining task-like resources such as
applications. The generic_node
type is useful for defining node-like
resources such as virtual machines.
Both generic_*
types
have several common labels you can use to define unique resource objects,
making it easy to use them in metric filters for aggregations and reductions.
In contrast, the global
resource type has only project_id
and location
labels. If you have many sources of metrics within a project, using the
same global
resource object can cause
collisions and over-writes of your metric data.
Creating metric descriptors
The data you collect for a custom metric must be associated with a
MetricDescriptor
object
that specifies various information about the metric.
You can create the metric descriptor yourself, or Monitoring can create it for you.
- Auto-creation of metric descriptors describes how Monitoring creates metric descriptors.
- Creating metric descriptors manually describes how you can create the descriptor, and it provides code samples.
You can change an existing descriptor in limited ways; see Modifying metric descriptors for more information.
Auto-creation of metric descriptors
If you write metric data when a metric descriptor for that custom metric doesn't yet exist, then a metric descriptor is created automatically. However, this new metric descriptor might not be exactly what you want; auto-creation of metric descriptors involves some assumptions and defaults.
Specifically, during a call to timeSeries.create
,
if a TimeSeries
object includes a Metric
object that specifies a nonexistent metric-type name, then
Cloud Monitoring creates a new
MetricDescriptor
with the following fields:
type
: The type is copied from theMetric
object'stype
field.name
: The name is created from the project ID in the method call and the value of thetype
in theMetric
object.labels
: The labels are those that appear in theMetric
object. Each label descriptor in the new metric descriptor has the following fields:key
: the label key in theMetric
object.valueType
:STRING
description
: not set
metricKind
: The default metric kind isGAUGE
, but if you specify themetricKind
parameter of theTimeSeries
object, then the new metric has that kind. You can specify onlyGAUGE
andCUMULATIVE
kinds.valueType
: The value type is taken from the typed value of thePoint
being written. which must beBOOL
,INT64
,DOUBLE
, orDISTRIBUTION
. If you specify a value type in thevalueType
field of theTimeSeries
, then that type must match the type of thePoint
.unit
: not setdescription
:"Auto created custom metric."
.displayName
: not set
In a single timeSeries.create
call, you can
include multiple TimeSeries
objects that refer to the same nonexistent metric type. In that case,
the new type's fields are set in the same way, except that the labels in the new
metric descriptor consist of the union of all the labels in the
Metric
objects in all the time series in this call to create
.
Next step: See Writing metric data.
Creating metric descriptors manually
This section describes how you can create the metric descriptor for your custom metric.
Before you begin
Complete the following steps:
Choose a metric name or identifier for your custom metric.
Choose a display name and description for your metric. The display name is used in the Google Cloud Console.
Choose a project or projects in which to define your custom metric and write its time series data. If you need the same metric in several projects, then make identical definitions of the metric in each project.
To write custom metrics from resources managed by an AWS account, create the metric descriptor in the AWS connector project for that account.
Determine the metric's kind, value type, and (optionally) units. Not all value types and metric kinds are supported for custom metrics. For more information on these fields, see Value types and metric kinds.
Choose the metric's labels—their names, value types, and descriptions.
For more help in making these choices, you can browse the built-in metrics and look at their time series data.
Creating a metric descriptor
To create the descriptor for your custom metric,
call the metricDescriptors.create
method, passing in a MetricDescriptor
object.
It is usually an error to call
metricDescriptors.create
using the same type
name as an existing custom metric descriptor. However, if all the fields of the
new MetricDescriptor
object match the fields of the
existing descriptor exactly, then it isn't an error but it has no effect.
In the following example, you create a gauge custom metric,
custom.googleapis.com/stores/daily_sales
. The metric has a single dimension
label, store_id
.
Protocol
To create a metric descriptor, use the
metricDescriptors.create
method.
You can execute this method by using the API Explorer widget on the
method's reference page. See API Explorer for more
information.
The following are the sample parameters to
metricDescriptors.create
:
- name (URL):
projects/[PROJECT_ID]
Request body: supply a
MetricDescriptor
object such as the following:{ "name": "", "description": "Daily sales records from all branch stores.", "displayName": "Sales", "type": "custom.googleapis.com/stores/sales", "metricKind": "CUMULATIVE", "valueType": "DOUBLE", "unit": "{USD}", "labels": [ { "key": "store_id", "valueType": "STRING", "description": "The ID of the store." }, ], }
Supply these values to the fields in the widget, using your project's ID
in place of [PROJECT_ID
]:
Click the Execute button to run the method.
When creating a new custom metric, the name
field in the
MetricDescriptor
is ignored and can be omitted. The create
method returns
the new metric descriptor with the name
field filled in, which in this
example would be the following:
"name": "projects/[PROJECT_ID]/metricDescriptors/custom.googleapis.com/stores/daily_sales"
If, for example, you want to get a metric's descriptor, you use this name.
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Next step: See Writing metric data.
Deleting metrics
To delete a custom metric, delete its metric descriptor. You can't delete the time-series data stored in your Google Cloud project; however, deleting the metric descriptor renders the data inaccessible. The data expires and is deleted according to the data retention policy.
You can't delete the metric descriptor for a built-in metric.
To delete your custom metric descriptor, call the
metricDescriptors.delete
method.
Protocol
To delete a metric descriptor, use the
metricDescriptors.delete
method.
You can execute this method by using the API Explorer widget on the
method's reference page. See API Explorer for more
information.
To delete the stores/daily_sales
custom metric created in
Creating metric descriptors manually:
- Go to the
reference page for
metricDescriptors.delete
: Supply the name of the metric descriptor to the API Explorer widget:
name:
projects/[PROJECT_ID]/metricDescriptors/custom.googleapis.com/stores/daily_sales
Click the Execute button.
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Writing metric data
You can write data only to metric types for custom metrics. To write your data,
use the timeSeries.create
method.
If the time series already exists, this
method appends a new data point to the existing time series. If the
time series doesn't exist, this method creates it and appends the data.
You write data points by passing a list of TimeSeries
objects to
timeSeries.create
.
The maximum list size is 200
and each object in the list must specify a different time series:
- Each time series is identified by the
metric
andresource
fields of theTimeSeries
object. These fields represent the metric type of the data and the monitored resource from which the data was collected. - Omit the fields
metricKind
andvalueType
; they are ignored when writing data points. Each
TimeSeries
object must contain only a singlePoint
object:- The point's value and time interval must be consistent with the metric
type's definition. For information on time intervals for different
metric kinds, see
TimeInterval
. - The point's time interval must be later than any point already in the time series.
- The end time of the interval must not be more than 25 hours in the past or more than five minutes in the future.
- The point's value and time interval must be consistent with the metric
type's definition. For information on time intervals for different
metric kinds, see
If you want to write more than one point to the same time series, then use a separate call to the
timeSeries.create
method for each point. Don't write data to a single time series faster than one point each 10 seconds. If you are adding data points to different time series, then there is no rate limitation.
Protocol
To write metric data, use the timeSeries.create
method.
You can execute this method by using the API Explorer widget on the
method's reference page. See API Explorer for more
information.
To write a point to the stores/daily_sales
custom metric created in the
Creating metric descriptors manually:
- Go to the reference page for
timeSeries.create
. - Supply the parameters below to the API Explorer widget.
- Click the Execute button.
Use the following sample parameters:
- name:
projects/[PROJECT_ID]
request body: include a list of
TimeSeries
objects. The following sample has only one time series in the list.{ "timeSeries": [ { "metric": { "type": "custom.googleapis.com/my_metric", "labels": { "my_label": "my_value" } }, "resource": { "type": "gce_instance", "labels": { "project_id": "[PROJECT_ID]", "instance_id": "1234567890123456789", "zone": "us-central1-f" } }, "points": [ { "interval": { "endTime": "2018-06-01T10:00:00-04:00" }, "value": { "doubleValue": 123.45 } } ] } ] }
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Modifying metric descriptors
To add labels to an existing custom metric, use the
timeSeries.create
method and include the new
labels with the time-series data.
If the labels you attempt to write are valid and don't cause the number
of labels in the metric descriptor to exceed the label limit of
10, then the labels
are added. The time series data is then written as though the label had been
there from the beginning.
If you want to do more than add new labels, you must delete and recreate the metric descriptor. In this case, you lose all the time series data previously collected for the old metric descriptor. See Deleting metrics for more information.
You can't rename a metric.
What's next
- To browse the built-in metrics, see Metrics list.
- To browse the available monitored resources, see Resource list.