Custom metrics are metrics defined by users. Custom metrics use the same elements that the built-in Cloud Monitoring metrics use:
- A set of data points.
- Metric-type information, which tells you what the data points represent.
- Monitored-resource information, which tells you where the data points originated.
Custom metrics can be used in the same way as built-in metrics. That is, you can create charts and alerts for your custom metric data.
For an introduction to the structures that underlie both the built-in and custom metrics, see Metrics, time series, and resources. To browse the built-in metrics, see Metrics list. To browse the available monitored resources, see Resource list.
Using a custom metric
To use a custom metric, you must have a metric descriptor for your new metric type. The metric descriptor defines how your data is organized. Before you start writing custom metric data, determine how you want to use the data you collect, and ensure that the data organization supports those objectives.
Creating a metric descriptor
Cloud Monitoring can create the metric descriptor for you automatically,
or you can use the metricDescriptors.create
method
in the Monitoring API to create it yourself. You can't manipulate
metric descriptors by using the Google Cloud Console for Monitoring.
To have Cloud Monitoring create the metric descriptor for you, you simply write time series data for your metric, and Cloud Monitoring creates a descriptor based on the data you are writing. For more information, see Auto-creation of metric descriptors. There are limits to auto-creation, so it's helpful to know what information goes into a metric definition. After you have a new custom metric descriptor, whether you or Monitoring created it, you can use the metric descriptor with the metric descriptor API methods and the time series API methods.
Whether you explicitly create the metric descriptor or let Monitoring create it for you based on your data, you still need to ensure the organization of the data matches how you want to query it.
Organizing your data
Suppose you have a program that runs on a single machine, and this program
calls auxiliary programs A
and B
. You want to count how often programs
A
and B
are called. You also want to know when program A
is called more
than 10 times per minute and when program B
is called more than 5 times per
minute.
To simplify the discussion, assume that you have a single Google Cloud
project and you plan to use the global
monitored resource. For more
information about this choice, see
Monitored resources for custom metrics.
Consider the following approaches:
You use two custom metric types:
Metric-type-A
counts calls to programA
andMetric-type-B
counts calls to programB
. In this case,Metric-type-A
contains 1 time series, andMetric-type-B
contains 1 time series.With this data model, you can create a single alerting policy with two conditions, or you can create two alerting policies each with one condition. An alerting policy can support multiple conditions, but it has a single configuration for the notification channels.
This model might be appropriate when you aren't interested in similarities in the data between the activities being monitored. In this example, the activities are the rate of calls to programs
A
andB
.You use a single custom metric type and use a label with that stores the value
A
orB
. Because Monitoring creates a time series for each unique combination of labels, there are two time series for this single metric type.As with the previous model, you can create a single alerting policy or two alerting policies. However, the conditions for the alerting policy are more complicated. To create a condition that generates an incident if the rate of calls for program
A
exceeds a threshold, then the condition needs to include a filter that includes data points whose label value isA
and excludes all other data points.One advantage of this model is that it is simple to compute ratios. For example, you can determine how much of the total is due to calls to
A
.You use a single custom metric type to count the number of calls, but you don't use a label to record which program was called. In this model, there is a single time series that combines the data for the two programs. However, you can't create an alerting policy that meets your objectives because the data for two programs can't be separated.
Custom metric names
When you create a custom metric, you create a string identifier that represents
the metric type. This string must be unique among the custom metrics in your
Google Cloud
project. The string must begin with a prefix that marks it as a user-defined
metric. For Monitoring, the only supported prefixes are
custom.googleapis.com/
and external.googleapis.com/prometheus
.
The use of arbitrary prefixes is not supported.
The prefix is followed by a name that describes what you are
collecting. See Naming conventions for more details.
Here are examples of the two kinds of identifiers for metric types:
custom.googleapis.com/cpu_utilization
custom.googleapis.com/instance/cpu/utilization
Custom metric types have resource names that are unique among all Google Cloud resources in all projects. The format of a resource name for a metric type is:
projects/[PROJECT_ID]/metricDescriptors/[METRIC_TYPE]
where METRIC_TYPE
is the string identifier of the metric type.
If the previous custom metric examples were created in project my-project-id
,
then their resource names would be the following:
projects/my-project-id/metricDescriptors/custom.googleapis.com/cpu_utilization
projects/my-project-id/metricDescriptors/custom.googleapis.com/instance/cpu/utilization
Name or type? In the metric descriptor, the name
field stores the
metric type's resource name and the type
field stores the METRIC_TYPE
string.
Defining your custom metric descriptor
To collect custom metric data, you must provide a
MetricDescriptor
object
that specifies various information about the metric. Cloud Monitoring
can create the descriptor for you automatically; see Auto-creation of metric
descriptors for more information.
If you want to create the metric descriptor explicitly, here is the information you'll need:
Choose an identifier for the custom metric type, as described in the previous section.
Choose a project in which to define your custom metric and write its time series data.
If you need the same metric in several projects, then make identical definitions of the metric in each project.
Custom metrics for AWS should be created in the AWS connector project for your AWS account.
Choose a display name and description of the metric. The display name is used in the Google Cloud Console.
Choose the metric's kind, value type, and (optionally) units. Not all value types and metric kinds are supported for custom metrics. See Value types and metric kinds.
Choose the metric's labels—their names, value types, and descriptions.
For more help in making these choices, you can browse the built-in metrics and look at their time series data.
Creating metric descriptors manually includes code samples for manually creating a metric descriptor.
Choosing a monitored-resource type
When you write your data to a time series, you must indicate where the data came from. To do this, you choose a monitored-resource type that represents where your data comes from, and then use that to describe the specific origin. The monitored resource is not part of the metric type; instead, the time series to which you write data includes a reference to the metric type, which describes the data, and a reference to the monitored resource, which indicates where the data originated.
Consider the monitored resource before creating your metric descriptor. The monitored-resource type you use affects which labels you need to include in the metric descriptor. For example, if you use a VM instance monitored resource with your data, then you won't need a metric label to specify the instance. For more information, see Monitored resources for custom metrics.
Each of your metric's data points must be associated with a monitored resource object. Points from different monitored resource objects are held in different time series.
Monitored resources for custom metrics
You can use only the following monitored-resource types in your custom metrics.
aws_ec2_instance
: Amazon EC2 instance.dataflow_job
: Dataflow job.gae_instance
: App Engine instance.gce_instance
: Compute Engine instance.generic_node
: User-specified computing node.generic_task
: User-defined task.gke_container
: GKE container instance.global
: Use this resource when no other resource type is suitable. For most use cases,generic_node
orgeneric_task
are better choices thanglobal
.k8s_cluster
: Kubernetes cluster.k8s_container
: Kubernetes container.k8s_node
: Kubernetes node.k8s_pod
: Kubernetes pod.
A common practice is to use the monitored resource objects that represent the physical resources where your application code is running. This approach has several advantages:
- You get better performance compared with using a single resource type.
- You avoid out-of-order data caused by multiple processes writing to the same time series.
- Your custom metric data can be grouped with other metric data from the same resources.
global
versus generic resources
The generic_task
and generic_node
resource types are useful in situations
where none of the more specific resource types are appropriate.
The generic_task
type is useful for defining task-like resources such as
applications. The generic_node
type is useful for defining node-like
resources such as virtual machines.
Both generic_*
types
have several common labels you can use to define unique resource objects,
making it easy to use them in metric filters for aggregations and reductions.
In contrast, the global
resource type has only project_id
and location
labels. If you have many sources of metrics within a project, using the
same global
resource object can cause
collisions and over-writes of your metric data.
Creating metric descriptors
The data you collect for a custom metric must be associated with a descriptor for a custom metric type. You can create the metric descriptor yourself, or Monitoring can create it for you.
- Auto-creation of metric descriptors describes how Monitoring creates metric descriptors.
- Creating metric descriptors manually provides more information and code samples.
You can change an existing descriptor in limited ways; see Modifying metric descriptors for more information.
Auto-creation of metric descriptors
If you write metric data to a custom metric type that does not yet exist, then a new custom metric descriptor is created for you automatically. However, this new metric descriptor might not be exactly what you want; auto-creation of metric descriptors involves some assumptions and defaults.
Specifically, during a call to timeSeries.create
,
if a TimeSeries
object includes a Metric
object that specifies a nonexistent metric-type name, then
Cloud Monitoring creates a new
MetricDescriptor
with the following fields:
type
: The type is copied from theMetric
object'stype
field.name
: The name is created from the project ID in the method call and the value of thetype
in theMetric
object.labels
: The labels are those that appear in theMetric
object. Each label descriptor in the new metric descriptor has the following fields:key
: the label key in theMetric
object.valueType
:STRING
description
: not set
metricKind
: The default metric kind isGAUGE
, but if you specify themetricKind
parameter of theTimeSeries
object, then the new metric has that kind. You can specify onlyGAUGE
andCUMULATIVE
kinds.valueType
: The value type is taken from the typed value of thePoint
being written. which must beBOOL
,INT64
,DOUBLE
, orDISTRIBUTION
. If you specify a value type in thevalueType
field of theTimeSeries
, then that type must match the type of thePoint
.unit
: not setdescription
:"Auto created custom metric."
.displayName
: not set
In a single timeSeries.create
call, you can
include multiple TimeSeries
objects that refer to the same nonexistent custom metric type. In that case,
the new type's fields are set in the same way, except that the labels in the new
metric descriptor consist of the union of all the labels in the
Metric
objects in all the time series in this call to create
.
Next step: See Writing metric data.
Creating metric descriptors manually
After you have collected the information you need for creating your custom
metric type, call the metricDescriptors.create
method, passing in
a MetricDescriptor
object.
See Modifying metric descriptors for more information.
It is usually an error to call
metricDescriptors.create
using the same type
name as an existing custom metric descriptor. However, if all the fields of the
new MetricDescriptor
object match the fields of the
existing descriptor exactly, then it is not an error but it has no effect.
In the following example, you create a gauge custom metric,
custom.googleapis.com/stores/daily_sales
. The metric has a single dimension
label, store_id
.
Protocol
To create a metric descriptor, use the
metricDescriptors.create
method.
You can execute this method by using the API Explorer widget on the
method's reference page. See API Explorer for more
information.
The following are the sample parameters to
metricDescriptors.create
:
- name (URL):
projects/[PROJECT_ID]
Request body: supply a
MetricDescriptor
object such as the following:{ "name": "", "description": "Daily sales records from all branch stores.", "displayName": "Sales", "type": "custom.googleapis.com/stores/sales", "metricKind": "CUMULATIVE", "valueType": "DOUBLE", "unit": "{USD}", "labels": [ { "key": "store_id", "valueType": "STRING", "description": "The ID of the store." }, ], }
Supply these values to the fields in the widget, using your project's ID
in place of [PROJECT_ID
]:
Click the Execute button to run the method.
When creating a new custom metric, the name
field in the
MetricDescriptor
is ignored and can be omitted. The create
method returns
the new metric descriptor with the name
field filled in, which in this
example would be the following:
"name": "projects/[PROJECT_ID]/metricDescriptors/custom.googleapis.com/stores/daily_sales"
If, for example, you want to get a metric's descriptor, you use this name.
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Next step: See Writing metric data.
Deleting metrics
To delete a custom metric, delete its metric descriptor. You cannot delete the time series data from your custom metric, but deleting the metric descriptor renders the data inaccessible. The data expires and is deleted according to the data retention policy.
To delete your custom metric descriptor, call the
metricDescriptors.delete
method.
Protocol
To delete a metric descriptor, use the
metricDescriptors.delete
method.
You can execute this method by using the API Explorer widget on the
method's reference page. See API Explorer for more
information.
You can only delete custom metric types.
To delete the stores/daily_sales
custom metric created in
Creating metric descriptors manually:
- Go to the
reference page for
metricDescriptors.delete
: Supply the name of the metric descriptor to the API Explorer widget:
name:
projects/[PROJECT_ID]/metricDescriptors/custom.googleapis.com/stores/daily_sales
Click the Execute button.
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Writing metric data
You can write data only to custom metric types. To write your data,
use the timeSeries.create
method.
If the time series already exists, this
method appends a new data point to the existing time series. If the
time series does not exist, this method creates it and appends the data.
You write data points by passing a list of TimeSeries
objects to
timeSeries.create
.
The maximum list size is 200
and each object in the list must specify a different time series:
- Each time series is identified by the
metric
andresource
fields of theTimeSeries
object. These fields represent the metric type of the data and the monitored resource from which the data was collected. - Omit the fields
metricKind
andvalueType
; they are ignored when writing data points. Each
TimeSeries
object must contain only a singlePoint
object:- The point's value and time interval must be consistent with the metric
type's definition. For information on time intervals for different
metric kinds, see
TimeInterval
. - The point's time interval must be later than any point already in the time series.
- The end time of the interval must not be more than 25 hours in the past or more than five minutes in the future.
- The point's value and time interval must be consistent with the metric
type's definition. For information on time intervals for different
metric kinds, see
If you want to write more than one point to the same time series, then use a separate call to the
timeSeries.create
method for each point. Don't write data to a single time series faster than one point each 10 seconds. If you are adding data points to different time series, then there is no rate limitation.
Protocol
To write metric data, use the timeSeries.create
method.
You can execute this method by using the API Explorer widget on the
method's reference page. See API Explorer for more
information.
To write a point to the stores/daily_sales
custom metric created in the
Creating metric descriptors manually:
- Go to the reference page for
timeSeries.create
. - Supply the parameters below to the API Explorer widget.
- Click the Execute button.
Use the following sample parameters:
- name:
projects/[PROJECT_ID]
request body: include a list of
TimeSeries
objects. The following sample has only one time series in the list.{ "timeSeries": [ { "metric": { "type": "custom.googleapis.com/my_metric", "labels": { "my_label": "my_value" } }, "resource": { "type": "gce_instance", "labels": { "project_id": "[PROJECT_ID]", "instance_id": "1234567890123456789", "zone": "us-central1-f" } }, "points": [ { "interval": { "endTime": "2018-06-01T10:00:00-04:00" }, "value": { "doubleValue": 123.45 } } ] } ] }
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Modifying metric descriptors
You can also use the timeSeries.create
call to add new
labels to custom metrics. You can add labels to your custom metric descriptor
by including them with the time series data.
If the label you attempt to write is valid and does not cause the number of labels in the metric descriptor to exceed the label limit of 10, the label is added. The time series data is then written as though the label had been there from the beginning.
If you want to do more than add new labels, you must delete and recreate the metric descriptor. In this case, you lose all the time series data previously collected for the old metric descriptor. See Deleting metrics for more information.
You cannot rename a metric.