Service Monitoring adds the following resources to the Monitoring API:
This page refers to these additions collectively as the SLO API and illustrates the primary uses. For a general overview of the structures in the SLO API, see Constructs in the API. Comprehensive reference material appears under Cloud Monitoring API v3.
You can use the SLO API to manage services and service-level objectives (SLOs). This page focuses on custom services and service-level indicators (SLIs). Services running on Anthos Service Mesh, Istio on Google Kubernetes Engine, and App Engine are automatically detected, and the SLIs for them are predefined.
You can also create SLIs by using the Google Cloud Console. For more information, see Creating an SLO.
Valid identifiers
Several methods in the SLO API use identifiers for different elements, particularly identifiers for project and services. These IDs adhere to the following rules:
- Length: 1–63 characters
- Characters: only lower-case letters, number, and hyphens
- Initial character: lower-case letter
- Terminal character: lower-case letter or a number, but not a hyphen
The regular expression for these rules is as follows:
[a-z](?:[-a-z0-9]{0,61}[a-z0-9])?
Examples using curl
This section describes the conventions and setup used for invoking the
SLO API by using the curl
tool. If you are using this API through a client
library, you can skip this section.
The examples on this page access the SLO API by using the
curl
tool to send HTTP requests to REST endpoints. Use the following
information about authentication and about invoking curl
to set the variables
used in the invocations.
Authentication
Create an environment variable to hold the ID of your scoping project of a metrics scope (or the Workspace's host project): These examples use
PROJECT_ID
:PROJECT_ID=my-test-service
Authenticate to the Cloud SDK:
gcloud auth login
To avoid having to specify your project ID with each command, set it as the default by using Cloud SDK:
gcloud config set project ${PROJECT_ID}
Create an authorization token and capture it in an environment variable. These examples call the variable
ACCESS_TOKEN
:ACCESS_TOKEN=`gcloud auth print-access-token`
You have to periodically refresh the access token. If commands that worked suddenly report that you are unauthenticated, re-issue this command.
To verify that you got an access token, echo the
ACCESS_TOKEN
variable:echo ${ACCESS_TOKEN} ya29.GluiBj8o....
Invoking curl
Each curl
invocation includes a set of arguments,
followed by the URL of a SLO API resource. The common arguments include
values specified by the PROJECT_ID
and ACCESS_TOKEN
environment variables.
You might also have to specify other arguments, for example, to specify the type
of the HTTP request (for example, -X DELETE
). The default request is GET
,
so the examples don't specify it.
Each curl
invocation has this general structure:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" <other_args> https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/<request>
For example, to list all the services in your project, issue the following GET
request:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services
This request returns an array of service descriptions, with entries like the following, an Istio service called “currencyservice”:
{ "services": [ { "name": "projects/[PROJECT_NUMBER]/services/[PROJECT_ID]-zone-us-central1-c-csm-main-default-currencyservice", "displayName": "[PROJECT_ID]/us-central1-c/csm-main/default/currencyservice", "clusterIstio": { "location": "us-central1-c", "clusterName": "csm-main", "serviceNamespace": "default", "serviceName": "currencyservice" }, "telemetry": { "resourceName": "//container.googleapis.com/projects/[PROJECT_ID]/zones/us-central1-c/clusters/csm-main/k8s/namespaces/default/services/currencyservice" } }, ... ] }
See Service
for more information about the structure of a service.
Managing services
The Service
resource acts as the root element for organizing
your services. Aspects of a particular service, like its SLOs for example, are
organized under the name of the service.
Services on Anthos Service Mesh, Istio on Google Kubernetes Engine, and App Engine are created automatically for you. You can do things like add SLOs to these services by using the Anthos Service Mesh console or the SLO API.
Services you create manually are called custom services. Custom services can be created, deleted, retrieved, and updated only by using the SLO API.
After you have identified or created a service, you can add SLOs to it. Managing SLOs covers commands to manipulate SLOs.
Service IDs
Each service has a fully qualified identifier called the resource name. This
identifier is stored in the name
field of the service description, for
example:
"name": "projects/[PROJECT_NUMBER]/services/[PROJECT_ID]-zone-us-central1-a-cloudrun-istio-system-cluster-local-gateway",
Embedded in the resource name is a shorter ID for the service, the
part of the resource name after the substring
projects/[PROJECT_NUMBER]/services/
If you created your own service with the resource name
projects/[PROJECT_NUMBER]/services/my-test-service
, the service ID is
my-test-service
.
For brevity and accuracy, many curl
examples assume the service ID is held
in the environment variable SERVICE_ID
so you can refer to it repeatedly.
Listing services
To retrieve information about all the services in your project, invoke
the services.list
method. To retrieve information about a
specific service by service ID, use the services.get
method:
Protocol
To list information about services by usingcurl
, invoke the
services.list
or
services.get
method by sending a GET
request to:
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services
to list all serviceshttps://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]
to get a specific service
For example, the following request retrieves information about
the service identified by the value stored in the environment variable
SERVICE_ID
:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}
Creating a service
If you are not using an environment where services are automatically created
(that is, Anthos Service Mesh, Istio on Google Kubernetes Engine, and App Engine), then you can create services by using the
services.create
method.
If you manually create services, you then have to manually add the SLOs and other service-monitoring artifacts to the service structure by using the SLO API. For an overview of these structures, see Constructs in the API.
To create a service, you must specify a display name for the service and
a field named custom
with an empty object.
You can optionally specify the service ID you want the service to have.
Protocol
To create the service by using curl
, send a POST
message to the
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services
endpoint:
If you want to provide an ID for the service, create a variable for the service ID:
SERVICE_ID=my-test-service
This value is supplied in the URL query parameter
service_id
.Create a variable to hold the request body that describes the service:
CREATE_SERVICE_POST_BODY=$(cat <<EOF { "displayName": "My Test Service", "custom": {} } EOF )
Post the request body to the endpoint, and specify the service ID as the value of the
service_id
query parameter:curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_SERVICE_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services?service_id=${SERVICE_ID}
Deleting a service
To delete a custom service, invoke the
services.delete
method and specify the service ID.
Protocol
To delete a service by usingcurl
, send a DELETE
request to the
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]
endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" -X DELETE https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}
Managing SLOs
SLOs are associated with services. You can create SLOs using the Anthos Service Mesh console for Anthos Service Mesh, Istio on Google Kubernetes Engine, and App Engine, or by using the SLO API. You must use the SLO API to create SLOs for custom services.
You can also use the SLO API to retrieve the SLOs associated with a service, and to delete SLOs from a service. You can have up to 500 SLOs for each service.
To manage SLOs for a service, you must have the service ID. SLOs are named based on the service they belong to. Service IDs describes how to recognize service IDs.
Listing SLOs
To retrieve information about all the SLOs associated with a service, invoke
the services.serviceLevelObjectives.list
method.
To retrieve information about a specific SLO by name, use the
services.serviceLevelObjectives.get
method:
Protocol
To list information about services by usingcurl
, invoke the
services.serviceLevelObjectives.list
or
services.serviceLevelObjectives.get
method by sending a GET
request to:
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives
to list all SLOshttps://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives/[SLO_ID]
to get a specific SLO
For example, the following request lists all SLOs associated with the
service ID stored in the environment variable SERVICE_ID
:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives
The following is an availability SLO retrieved from the “currencyservice” service:
{ "name": "projects/[PROJECT_NUMBER]/services/[PROJECT_ID]-zone-us-central1-c-csm-main-default-currencyservice/serviceLevelObjectives/3kavNVTtTMuzL7KcXAxqCQ", "displayName": "98% Availability in Calendar Week" "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.98, "calendarPeriod": "WEEK", },
This SLO is built on an availability SLI, it sets a target performance goal of 98 percent, and it measures compliance over a calendar week. For more information on availability SLIs, see Service-level indicators.
See ServiceLevelObjective
for more information about the
structure of SLOs.
Retrieving a specific SLO
Each SLO has a unique identifier within the service, consisting of the string following
serviceLevelObjectives
in the SLO's name
field. In the following example:
"name": "projects/[PROJECT_NUMBER]/services/[PROJECT_ID]-zone-us-central1-c-csm-main-default-currencyservice/serviceLevelObjectives/3kavNVTtTMuzL7KcXAxqCQ",
the SLO ID is the string 3kavNVTtTMuzL7KcXAxqCQ
.
To retrieve information about this particular SLO, request the SLO by ID.
Protocol
To get a specific SLO by usingcurl
, invoke the
services.serviceLevelObjectives.get
method by sending a GET
request to https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives/[SLO_ID]
:
SLO_ID=dhefHDM4TwSRZEKIV4ZYEg curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}
Creating an SLO
To create an SLO by using the SLO API, you must create a
ServiceLevelObjective
object and pass it to the
serviceLevelObjectives.create
method.
The structure of an SLO has many embedded structures, including
one for the value of the serviceLevelIndicator
field.
For Anthos Service Mesh, Istio on Google Kubernetes Engine, and App Engine services, the SLIs are pre-defined. You can use the Anthos Service Mesh console to create SLOs; all you have to do is specify the performance goals and compliance periods.
You can also define SLOs by using the SLO API.
For custom services, you must define SLOs by using the SLO API. To specify an SLO, you must create a value for the
serviceLevelIndicator
field, as well. See Creating a service-level indicator for some techniques.
Skeleton of a SLO
The basic skeleton for building the SLO is as follows:
{ "displayName": string, "serviceLevelIndicator": { object (ServiceLevelIndicator) }, "goal": number, // Union field period can be only one of the following: "rollingPeriod": string, "calendarPeriod": enum (CalendarPeriod) // End of list of possible types for union field period. }
You must specify the following:
- Display name: a description of the SLO
- A service-level indicator, which is one of the three types:
- The performance goal (a percentage)
- The compliance period, which one of two types:
- A rolling period of some length (in seconds)
- A calendar period
For more information about SLIs, performance goals, and compliance periods, see Concepts in service monitoring.
For Anthos Service Mesh, Istio on Google Kubernetes Engine, and App Engine services, the SLI type is the basic SLI.
For custom services, you have to create a request-based SLI or a windows-based SLI. See Creating a service-level indicator for some techniques.
Example
After you have the SLI, you can build the SLO. The following example defines an SLO for a service that uses a basic SLI. You might have several SLOs on a single SLI, for example, one for 95% availability and one for 99% availability. The following example is an SLO for 95% availability over a calendar week:
{ "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.95, "calendarPeriod": "WEEK", "displayName": "95% Availability in Calendar Week" }
This example specifies an SLO for 75% availability over a rolling 3-day period:
{ "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.75, "rollingPeriod": "259200s", "displayName": "75% Availability in Rolling 3 Days" }
Protocol
To create an SLO by usingcurl
, send a POST
message to the
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives
endpoint:
Create a variable for the service ID:
SERVICE_ID=custom:my-test-service
Create a variable to hold the request body, an instance of the
ServiceLevelObjective
object. This example uses one of the SLOs described previously:CREATE_SLO_POST_BODY=$(cat <<EOF { "serviceLevelIndicator": { "basicSli": { "availability": {}, "location": [ "us-central1-c" ] } }, "goal": 0.75, "rollingPeriod": "259200s", "displayName": "75% Availability in Rolling 3 Days" } EOF )
Post the request body to the endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_SLO_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives
Optionally, you can also specify a desired ID for the SLO as the value of the
service_level_objective_id
query parameter:SLO_ID=test-slo-id curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" --header "Content-Type: application/json" -X POST -d "${CREATE_SLO_POST_BODY}" https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives?service_level_objective_id=${SLO_ID}
Deleting an SLO
To delete an SLO, invoke the
services.serviceLevelObjectives.delete
method
and specify the ID of the SLO in your project:
Protocol
To delete an SLO by usingcurl
, send a DELETE
request to the
https://monitoring.googleapis.com/v3/projects/[PROJECT_ID]/services/[SERVICE_ID]/serviceLevelObjectives/[SLO_ID]
endpoint:
curl --http1.1 --header "Authorization: Bearer ${ACCESS_TOKEN}" -X DELETE https://monitoring.googleapis.com/v3/projects/${PROJECT_ID}/services/${SERVICE_ID}/serviceLevelObjectives/${SLO_ID}
Accessing SLO time series
SLO data is stored in time series, so you can use the
timeSeries.list
method to retrieve it.
However, this data isn't stored in standard metric types, so
you can't use the standard mechanism of specifying a metric-type
filter to the timeSeries.list
method.
Instead, SLO time series are retrieved by specifying another type of
filter, a time-series selector, to the timeSeries.list
method in
the filter
parameter.
See Retrieving SLO data for information on using these selectors.
You also use time-series selectors to set up alerting policies programmatically. See Alerting on your burn rate for more information.