Custom metrics are metrics defined by users. Custom metrics use the same elements that the built-in Stackdriver Monitoring metrics use:
- A set of data points.
- Metric-type information, which tells you what the data points represent.
- Monitored-resource information, which tells you where the data points originated.
For an introduction to the structures that underlie both the built-in and custom metrics, see Metrics, time series, and resources. To browse the built-in metrics, see Metrics list. To browse the available monitored resources, see Resource list.
To use a custom metric, you must have a metric descriptor for your new metric type. Stackdriver Monitoring can create the metric descriptor for you automatically, or you can use the metricDescriptors.create API method to create it yourself.
To have Stackdriver Monitoring create the metric descriptor for you, you simply write time series data for your metric, and Stackdriver Monitoring creates a descriptor based on the data you are writing. There are limits to auto-creation, so it's helpful to know what information goes into a metric definition.
After you have a new custom metric descriptor, whether you or Monitoring created it, you can use the metric descriptor with the metric descriptor API methods and the time series API methods.
You can also create charts and alerts for your custom metric data.
Custom metric names
When you create a custom metric, you create a string identifier that
represents the
metric type. This string must be unique among the custom metrics in your
Google Cloud
project. The string must begin with a prefix that marks it as a user-defined
metric. For Monitoring, the most common prefix is
custom.googleapis.com/
, though external.googleapis.com/
is also used.
This is followed by either a simple name or a
path name that generally describes what you are collecting.
Path names can help you organize your custom metrics, but they are handled
just like simple names.
See Naming rules for more details.
Here are examples of the two kinds of identifiers for metric types:
custom.googleapis.com/cpu_utilization
custom.googleapis.com/instance/cpu/utilization
Your custom metrics also have resource names that are unique among all Google Cloud resources in all projects. The format of a resource name for a metric is:
projects/[PROJECT_ID]/metricDescriptors/[METRIC_TYPE]
If the previous custom metric examples were created in project my-project-id
,
then their resource names would be the following:
projects/my-project-id/metricDescriptors/custom.googleapis.com/cpu_utilization
projects/my-project-id/metricDescriptors/custom.googleapis.com/instance/cpu/utilization
Name or type? In the Monitoring API, metric type name fields are usually
labeled type
and resource name fields are usually labeled name
.
However, in conversation and some documentation,
the type name is often called the "metric name."
Defining your custom metric descriptor
To collect custom metric data, you must provide a MetricDescriptor object that specifies various information about the metric. Stackdriver Monitoring can create this for you automatically; see Auto-creation of metric descriptors for more information.
If you want to create the metric descriptor explicitly, here is the information you'll need:
Choose an identifier for the custom metric type, as described in the previous section.
Choose a project in which to define your custom metric and write its time series data.
The project you choose must also be a member of the Workspace where you monitor the metric. If you need the same metric in several projects, then make identical definitions of the metric in each project.
Custom metrics for AWS should be created in the AWS connector project for your AWS account.
Choose a display name and description of the metric. The display name is used in the Monitoring console.
Choose the metric's kind, value type, and (optionally) units. Not all value types and metric kinds are supported for custom metrics. See Kinds of metrics.
Choose the metric's labels—their names, value types, and descriptions.
For more help in making these choices, you can browse the built-in metrics and look at their time series data.
Creating metric descriptors manually includes code samples for manually creating a metric descriptor.
Choosing a monitored-resource type
When you write your data to a time series, you must indicate where the data came from. To do this, you choose a monitored-resource type that represents where your data comes from, and then use that to describe the specific origin. The monitored resource is not part of the metric type; instead, the time series to which you write data includes a reference to the metric type, which describes the data, and a reference to the monitored resource, which indicates where the data originated.
Consider the monitored resource before creating your metric descriptor. The monitored-resource type you use affects which labels you need to include in the metric descriptor. For example, if you use a VM instance monitored resource with your data, then you won't need a metric label to specify the instance. For more information, see Choosing a monitored-resource type.
Each of your metric's data points must be associated with a monitored resource object. Points from different monitored resource objects are held in different time series.
Monitored resources for custom metrics
You can use only the following monitored-resource types in your custom metrics.
aws_ec2_instance
: Amazon EC2 instance.dataflow_job
: Dataflow job.gce_instance
: Google Compute Engine instance.gke_container
: Kubernetes Engine container instance.generic_node
: User-specified computing node.generic_task
: User-defined task.global
: Use this resource when no other resource type is suitable. For most use cases,generic_node
orgeneric_task
are better choices thanglobal
.k8s_cluster
: Kubernetes cluster.k8s_container
: Kubernetes container.k8s_node
: Kubernetes node.k8s_pod
: Kubernetes pod.
A common practice is to use the monitored resource objects that represent the physical resources where your application code is running. This has several advantages:
- You get better performance compared with using a single resource type.
- You avoid out-of-order data caused by multiple processes writing to the same time series.
- Your custom metric data can be grouped with other metric data from the same resources.
global
versus generic resources
The generic_task
and generic_node
resource types are useful in situations
where none of the more specific resource types are appropriate.
The generic_task
type is useful for defining task-like resources such as
applications. The generic_node
type is useful for defining node-like
resources such as virtual machines.
Both generic_*
types
have several common labels you can use to define unique resource objects,
making it easy to use them in metric filters for aggregations and reductions.
In contrast, the global
resource type has only project_id
and location
labels. If you have many sources of metrics within a project, using the
same global
resource object can cause
collisions and over-writes of your metric data.
Creating metric descriptors
The data you collect for a custom metric must be associated with a descriptor for a custom metric type. You can create the metric descriptor yourself, or Monitoring can create it for you.
- Auto-creation of metric descriptors describes how Monitoring creates metric descriptors.
- Creating metric descriptors manually provides more information and code samples.
You can change an existing descriptor in limited ways; see Modifying metric descriptors for more information.
Auto-creation of metric descriptors
If you write metric data to a custom metric type that does not yet exist, then a new custom metric descriptor is created for you automatically. However, this new metric descriptor might not be exactly what you want; auto-creation of metric descriptors involves some assumptions and defaults.
Specifically, during a call to timeSeries.create, if a TimeSeries object includes a Metric object that specifies a nonexistent metric-type name, then Stackdriver Monitoring creates a new MetricDescriptor with the following fields:
type
: The type is copied from the Metric object'stype
field.name
: The name is created from the project ID in the method call and the value of thetype
in theMetric
object.labels
: The labels are those that appear in theMetric
object. Each label descriptor in the new metric descriptor has the following fields:key
: the label key in theMetric
object.valueType
:STRING
description
: not set
metricKind
: The default metric kind isGAUGE
, but if you specify themetricKind
parameter of the TimeSeries object, then the new metric has that kind. You can specify onlyGAUGE
andCUMULATIVE
kinds.valueType
: The value type is taken from the typed value of the Point being written. which must beBOOL
,INT64
,DOUBLE
, orDISTRIBUTION
. If you specify a value type in thevalueType
field of the TimeSeries, then that type must match the type of the Point.unit
: not setdescription
:"Auto created custom metric."
.displayName
: not set
In a single timeSeries.create call, you can include multiple TimeSeries
objects that refer to the same nonexistent custom metric type. In that case,
the new type's fields are set in the same way, except that the labels in the new
metric descriptor consist of the union of all the labels in the
Metric objects in all the time series in this call to create
.
Next step: See Writing metric data.
Creating metric descriptors manually
After you have collected the information you need for creating your custom metric type, call the metricDescriptors.create method, passing in a MetricDescriptor object. See Modifying metric descriptors for more information.
It is usually an error to call metricDescriptors.create using the same type name as an existing custom metric descriptor. However, if all the fields of the new MetricDescriptor object match the fields of the existing descriptor exactly, then it is not an error but it has no effect.
In the following example, you create a gauge custom metric,
custom.googleapis.com/stores/daily_sales
. The metric has a single dimension
label, store_id
.
Protocol
To create a metric descriptor, use the metricDescriptors.create method. You can execute this method by using the API Explorer widget on the method's reference page. See API Explorer for more information.
The following are the sample parameters to metricDescriptors.create:
- name (URL):
projects/[PROJECT_ID]
Request body: supply a MetricDescriptor object such as the following:
{ "name": "", "description": "Daily sales records from all branch stores.", "displayName": "Sales", "type": "custom.googleapis.com/stores/sales", "metricKind": "CUMULATIVE", "valueType": "DOUBLE", "unit": "{USD}", "labels": [ { "key": "store_id", "valueType": "STRING", "description": "The ID of the store." }, ], }
Supply these values to the fields in the widget, using your project's ID
in place of [PROJECT_ID
]:
Click the Execute button to run the method.
When creating a new custom metric, note that the name
field in the
MetricDescriptor
is ignored and can be omitted. The create
method returns
the new metric descriptor with the name
field filled in, which in this
example would be the following:
"name": "projects/[PROJECT_ID]/metricDescriptors/custom.googleapis.com/stores/daily_sales"
You use this name, for example, if you want to get a metric's descriptor.
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Next step: See Writing metric data.
Deleting metrics
To delete a custom metric, delete its metric descriptor. You cannot delete the time series data from your custom metric, but deleting the metric descriptor renders the data inaccessible. The data expires and is deleted according to the data retention policy.
To delete your custom metric descriptor, call the metricDescriptors.delete method.
Protocol
To delete a metric descriptor, use the metricDescriptors.delete method. You can execute this method by using the API Explorer widget on the method's reference page. See API Explorer for more information.
You can only delete custom metric types.
To delete the stores/daily_sales
custom metric created in
Creating metric descriptors manually:
- Go to the reference page for metricDescriptors.delete:
Supply the name of the metric descriptor to the API Explorer widget:
name:
projects/[PROJECT_ID]/metricDescriptors/custom.googleapis.com/stores/daily_sales
Click the Execute button.
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Writing metric data
You can write data only to custom metric types. To write your data, use the timeSeries.create method. If the time series already exists, this method appends a new data point to the existing time series. If the time series does not exist, this method creates it and appends the data.
You write data points by passing a list of TimeSeries objects to timeSeries.create. The maximum list size is 200 and each object in the list must specify a different time series:
- Each time series is identified by the
metric
andresource
fields of the TimeSeries object. These fields represent the metric type of the data and the monitored resource from which the the data was collected. - Omit the fields
metricKind
andvalueType
; they are ignored when writing data points. Each TimeSeries object must contain only a single Point object:
- The point's value and time interval must be consistent with the metric type's definition. For information on time intervals for different metric kinds, see Metric kinds.
- The point's time interval must be later than any point already in the time series.
- The end time of the interval must not be more than 24 hours in the past or more than five minutes in the future.
If you want to write more than one point to the same time series, then use a separate call to the timeSeries.create method for each point. Don't make the calls faster than one time per minute. If you are adding data points to different time series, then there is no rate limitation.
Protocol
To write metric data, use the timeSeries.create method. You can execute this method by using the API Explorer widget on the method's reference page. See API Explorer for more information.
To write a point to the stores/daily_sales
custom metric created in the
Creating metric descriptors manually:
- Go to the reference page for timeSeries.create.
- Supply the parameters below to the API Explorer widget.
- Click the Execute button.
Use the following sample parameters:
- name:
projects/[PROJECT_ID]
request body: include a list of TimeSeries objects. The following sample has only one time series in the list.
{ "timeSeries": [ { "metric": { "type": "custom.googleapis.com/my_metric", "labels": { "my_label": "my_value" } }, "resource": { "type": "gce_instance", "labels": { "project_id": "[PROJECT_ID]", "instance_id": "1234567890123456789", "zone": "us-central1-f" } }, "points": [ { "interval": { "endTime": "2018-06-01T10:00:00-04:00" }, "value": { "doubleValue": 123.45 } } ] } ] }
C#
Go
Java
Node.js
PHP
Python
Ruby
See Troubleshooting API calls if you have difficulty.
Modifying metric descriptors
You can also use the timeSeries.create call to add new labels to custom metrics. You can add labels to your custom metric descriptor by including them with the time series data.
If the label you attempt to write is valid and does not cause the number of labels in the metric descriptor to exceed the label limit of 10, the label is added. The time series data is then written as though the label had been there from the beginning.
If you want to do more than add new labels, you must delete and recreate the metric descriptor. In this case, you lose all the time series data previously collected for the old metric descriptor. See Deleting metrics for more information.
You cannot rename a metric.