Unattended project recommender
The unattended project recommender analyzes usage activity on projects in your organization and provides recommendations that help you discover, reclaim or remove unattended projects.
Overview
In fast-moving organizations, it's not uncommon for cloud resources, including entire projects, to occasionally be forgotten about. Such unattended resources can be difficult to identify and tend to result in unnecessary waste and security risks.
Unattended project recommender analyzes usage activity across all projects in your organization and provides you with the following features to help you discover, reclaim, and shut down unattended projects:
- Usage insights for every project (networking, API, project owner, service activity, and more).
- Recommendations to turn down projects having low usage activity.
- Recommendations to assign a new owner to projects that have high usage activity but no active owner.
Shutting down or reclaiming unattended projects can provide the following impact and benefits to your organization:
- Reduction in security risks (
SECURITY
) - Reduction in unnecessary spending (
COST
) - Reduction in carbon footprint associated with your workloads (
SUSTAINABILITY
)
How it works
The unattended project recommender analyzes project usage over the last 30 days. The following table lists the recommendations and insights that the unattended project recommender can generate based on whether the usage activity is low or high .
Project usage/type | Insight subtype | Recommendation | Recommendation subtype |
---|---|---|---|
Projects with:
|
PROJECT_ACTIVITY |
Review or delete the project. | CLEANUP_PROJECT |
Projects with:
|
PROJECT_ACTIVITY |
Reclaim the project by assigning a new owner. | RECLAIM_PROJECT |
All other non-Apps Script projects. | PROJECT_ACTIVITY |
- | - |
Recommendations and Insights are available at different levels of your Cloud Organization Hierarchy. Accessing those recommendations and insights requires an IAM role binding at the corresponding level:
- At a Project level: Project Owners can discover and act on insights and recommendations for the projects they have access to.
- At an Organizational level: Organization Administrators can discover and act on insights and recommendations for projects within a cloud organization.
- At a Billing Account level: Billing Administrators can discover and act on insights and recommendations for projects associated with a particular billing account.
Ranking the usage activity of a project
A project can be ranked to have a low or high usage activity, based on which recommendations are provided. The ranking of a project's usage activity is computed based on the content included in the project's utilization insight.
For projects that are identified to have low usage activity, the following conditions apply:
Usage classification | Condition |
---|---|
Organizations that have more than 50 projects |
|
Organizations that have less than 50 projects |
|
Recommendation priority and impacts
Unattended project recommendations contain the following fields that help prioritize and estimate the benefits of applying them:
priority
- A priority value between P1-P4, with P1 being the highest priority.primaryImpact
-SECURITY
. The count of high-priority security recommendations in an unattended project.additionalImpact(s)
-COST
andSUSTAINABILITY
. The cost ($) and sustainability emission (kgCO2e) reduction that would be realized if the project was removed.
Recommendation priority
The priority of the unattended project recommendation ranges from P1 to P4, with P1 being the highest priority.
The recommender begins by reviewing security recommendations in each unattended project. From there, it sets the priority of the unattended project recommendation equal to that of the highest priority recommendation found within the project. For example, if an unattended project has 10 security recommendations, eight P2 recommendations and two P1 recommendations, the unattended project recommendation would be assigned a priority value of P1.
The most frequently occurring security recommendations found in unattended projects are from the IAM Recommender. An unattended project recommendation with a priority of P1 likely means there is a publicly accessible cloud storage bucket in the unattended project (learn more about how IAM Recommender sets priorities).
Security impact
The primaryImpact
- SECURITY
field offers a
securityProjection
equal to the count of high-priority
(P2 and higher) security recommendations in the unattended project.
For example, when a project has a recommendation to replace the broad Editor
role with a more granular role, that recommendation has a
priority of P2
and gets counted in the priorityRecommendationCount
.
Cost Impact
The additionalImpact
- COST
field offers a
costProjection
that is an estimate of the cost savings if the
unattended project was removed. This is based on the costs incurred from
resources running in the unattended project over the last 30 days.
Sustainability impact
The additionalImpact
- SUSTAINABILITY
field offers
a sustainabilityProjection
that is an estimate of
carbon emissions contributed by the
resources running in the unattended project over the last 30 days. The estimate
is measured in kilograms of carbon dioxide equivalent (kgCO2e).
Contents of a project utilization insight
The contents of a project's utilization insight are field values that are used
to rank the usage activity of the project and generate CLEANUP_PROJECT
and/or RECLAIM_PROJECT
recommendations.
The following table lists the various fields that the recommender surfaces as
part of the PROJECT_ACTIVITY
insight:
Field | Description |
---|---|
Level of usage | |
usagePercentile |
Percentile of the usage level of this project compared with other
projects within the same organization.
If the percentile is -1, this means that the value is not computed. The value is not computed when organizations have less than 50 projects. |
API activity | |
activeServiceAccountDailyCount |
Daily average of service accounts with authentication activity under this project. |
apiClientDailyCount |
Daily average of distinct clients of API calls produced by this project. |
consumedApiDailyCount |
Daily average of API calls consumed by this project. |
datastoreApiDailyCount |
Daily average of Datastore API calls consumed by this project |
Networking activity | |
vpcEgressDailyBytes |
Daily average of VPC egress bytes out of this project. |
vpcIngressDailyBytes |
Daily average of VPC ingress bytes into this project |
Billing usage | |
hasBillingAccount |
Checks if the project has a billing account at the end of the observation |
serviceWithBillableUsage |
Names of the billable services used in the trailing 30 days. |
Cloud services usage | |
activeAppengineInstanceDailyCount : system/instance_count |
Daily average of the number of active App Engine instances under this project |
activeCloudsqlInstanceDailyCount |
Daily average of the number of active Cloud SQL instances under this project. |
activeGceInstanceDailyCount |
Daily average of the number of active Compute Engine instances under this project. |
bigqueryInflightJobDailyCount |
Daily average of inflight BigQuery jobs under this project. |
bigqueryInflightQueryDailyCount |
Daily average of inflight BigQuery query count under this project. |
bigqueryStorageDailyBytes |
Daily average of inflight BigQuery storage under this project. |
bigqueryTableDailyCount |
Daily average of BigQuery table count under this project. |
gcsObjectDailyCount |
Daily average of number of Cloud Storage objects under this project. |
gcsRequestDailyCount |
Daily average of number of Cloud Storage API under this project. |
gcsStorageDailyBytes |
Daily average of storage bytes used by Cloud Storage under this project. |
User activity | |
numActiveUserOwners |
The number of active user type project owners at the end of the observation period. Active means that the user account is not disabled and there was some activity identified under this project during the observation period. |
owners |
List of project owners and their activities |
Other fields | |
hasActiveOauthTokens |
Checks if the project has any active OAuth tokens used in the last 180 days. |
carbonFootprintDailyKgCO2 |
Daily average of carbon emissions contributed by this project based on the last 30 days of data available. |
costProjection |
Projected cost savings for the specified duration if the project was to be removed. A custom or list price is shown based on the billing account permissions. It is an estimate based on the last 30 days of available data and does not constitute a bill. |
severity |
The severity is set to the corresponding value of the
highest severity security insight found in the project. For example,
based on how
IAM Recommender sets severities
for their insights, a CRITICAL severity on the unattended
project insight likely means there is a publicly accessible cloud storage
bucket in the unattended project. |
Recommendations to delete a project
A project is recommended for deletion when it has a low usage in the last 30 days and no OAuth tokens used in the last 180 days.
Recommendations to assign a new owner to a project
It is recommended to assign a new owner to a project when it meets all conditions as listed below:
- All assigned owners are either inactive on the project for the last 90 days or no longer with the company.
- The project has a high usage activity.
Pricing
For more details on Active Assist pricing, refer to the recommender pricing.
Before you begin
Before you can view the insights and recommendations, you must do the following:
- You must enable the Recommender API.
You only need to enable the API on a single billing project. You can then use this
same billing project to examine recommendations and insights for other projects, the
entire organization or the billing account, using the
--billing-project
functionality of gcloud/API. - Make sure that you have one of these required roles assigned:
Description | Roles |
---|---|
View recommendations |
recommender.projectUtilViewer |
View and update recommendations |
recommender.projectUtilAdmin |
Opting-out of insights and recommendations in Transparency and Control Center |
dataprocessing.admin |
These roles provide a set of permissions that enable you to access the insights and recommendations. For more information about roles, refer to understanding roles and granting IAM permissions.
Granting these roles at the Organization level lets you access the recommendations and insights for the projects in the organization individually, or collectively (all unattended projects in the organization) using the Organization level API.
Granting these roles at the Billing Account level lets you access the recommendations and insights for projects that use the specified billing account.
For more information about opting out, refer to opting out of insights and recommendations.
Recommender ID
Here are the unattended project recommender ID and insight type:
google.resourcemanager.projectUtilization.Recommender
google.resourcemanager.projectUtilization.Insight
Required IAM permissions
To access the list of recommendations and insights, the following permissions are required:
recommender.resourcemanagerProjectUtilizationRecommendations.get
recommender.resourcemanagerProjectUtilizationRecommendations.list
recommender.resourcemanagerProjectUtilizationInsights.get
recommender.resourcemanagerProjectUtilizationInsights.list
To update recommendations and insights (for example, to dismiss a recommendation), the following permissions are required:
recommender.resourcemanagerProjectUtilizationRecommendations.update
recommender.resourcemanagerProjectUtilizationInsights.update
Viewing unattended project recommendations
This section describes how to check for unattended insights and recommendations
using gcloud
, and using curl
to send requests to the recommender API.
gcloud
To view recommendations and insights using gcloud
, follow the steps below.
For more information, see Using the API - Insights and Using the API - Recommendations.
Recommendations
To list unattended project recommendations for the project where you've enabled the Recommender API, run the following command:
gcloud recommender recommendations list \ --project=PROJECT_ID \ --location=global \ --recommender=google.resourcemanager.projectUtilization.Recommender
Replace the following:
PROJECT_ID
: The project ID.
To list unattended project recommendations in a different project while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
gcloud recommender recommendations list \ --billing-project=BILLING_PROJECT_ID \ --project=PROJECT_ID \ --location=global \ --recommender=google.resourcemanager.projectUtilization.Recommender
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).PROJECT_ID
: The ID of a different project for which you'd like to view the unattended project recommendations.
To list unattended project recommendations for the entire organization, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
gcloud recommender recommendations list \ --billing-project=BILLING_PROJECT_ID \ --organization=ORGANIZATION_ID \ --location=global \ --recommender=google.resourcemanager.projectUtilization.Recommender
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).ORGANIZATION_ID
: The ID of the Organization for which you'd like to view the unattended project recommendations.
To list unattended project recommendations that are using a specific Billing Account, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
gcloud recommender recommendations list \ --billing-project=BILLING_PROJECT_ID \ --billing-account=BILLING_ACCOUNT_ID \ --location=global \ --recommender=google.resourcemanager.projectUtilization.Recommender
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).BILLING_ACCOUNT_ID
: The ID of your Google Cloud Billing Account.
Insights
To list usage insights for a given project, run the following command:
gcloud recommender insights list \ --project=PROJECT_ID \ --location=global \ --insight-type=google.resourcemanager.projectUtilization.Insight
Replace the following:
PROJECT_ID
: The project ID.
To list usage insights for a different project, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
gcloud recommender insights list \ --billing-project=BILLING_PROJECT_ID \ --project=PROJECT_ID \ --location=global \ --insight-type=google.resourcemanager.projectUtilization.Insight
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).PROJECT_ID
: The ID of a different project for which you'd like to view the unattended recommender insights.
To list usage insights for the entire organization, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
gcloud recommender insights list \ --billing-project=BILLING_PROJECT_ID \ --organization=ORGANIZATION_ID \ --location=global \ --insight-type=google.resourcemanager.projectUtilization.Insight
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).ORGANIZATION_ID
: The ID of the Organization for which you'd like to view the usage insights.
To list usage insights that are using a specific Billing Account, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
gcloud recommender insights list \ --billing-project=BILLING_PROJECT_ID \ --billing-account=BILLING_ACCOUNT_ID \ --location=global \ --insight-type=google.resourcemanager.projectUtilization.Insight
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).BILLING_ACCOUNT_ID
: The ID of your Google Cloud Billing Account.
API
To view recommendations and insights, you can use curl
to send a request to the recommender APIs. To authorize requests from the command line,
you use OAuth access tokens. An OAuth access token is a string that grants temporary access to an API.
Recommendations
To list unattended project recommendations for the project where you've enabled the Recommender API, run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ "https://recommender.googleapis.com/v1/projects/PROJECT_ID/locations/global/\ recommenders/google.resourcemanager.projectUtilization.Recommender/recommendations"
Replace the following:
- PROJECT_ID`: The project ID.
To list unattended project recommendations in a different project while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: BILLING_PROJECT_ID" \ "https://recommender.googleapis.com/v1/projects/PROJECT_ID/locations/global/\ recommenders/google.resourcemanager.projectUtilization.Recommender/recommendations"
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).PROJECT_ID
: The ID of a different project for which you'd like to view the unattended project recommendations.
To list unattended project recommendations for the entire organization, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: BILLING_PROJECT_ID" \ "https://recommender.googleapis.com/v1/organizations/ORGANIZATION_ID/locations/global/\ recommenders/google.resourcemanager.projectUtilization.Recommender/recommendations"
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).ORGANIZATION_ID
: The ID of the Organization for which you'd like to view the unattended project recommendations.
To list unattended project recommendations that are using a specific Billing Account, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: BILLING_PROJECT_ID" \ "https://recommender.googleapis.com/v1/billingAccounts/BILLING_ACCOUNT_ID/locations/global/\ recommenders/google.resourcemanager.projectUtilization.Recommender/recommendations"
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).BILLING_ACCOUNT_ID
: The ID of your Google Cloud Billing Account.
Insights
To list usage insights for a given project, run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ "https://recommender.googleapis.com/v1/projects/PROJECT_ID/locations/global\ /insightTypes/google.resourcemanager.projectUtilization.Insight/insights"
Replace the following:
- PROJECT_ID`: The project ID.
To list usage insights for a different project, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: BILLING_PROJECT_ID/" \ "https://recommender.googleapis.com/v1/projects/PROJECT_ID//locations/global\ /insightTypes/google.resourcemanager.projectUtilization.Insight/insights"
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).PROJECT_ID
: The ID of a different project for which you'd like to view the unattended recommender insights.
To list usage insights for the entire organization, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID/" \ "https://recommender.googleapis.com/v1/organizations/ORGANIZATION_ID/locations/global/\ insightTypes/google.resourcemanager.projectUtilization.Insight/insights"
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).ORGANIZATION_ID
: The ID of the Organization for which you'd like to view the unattended project recommendations.
To list usage insights that are using a specific Billing Account, while using a project where you've enabled the Recommender API (as in the previous step), run the following command:
curl \ -H "Authorization: Bearer $(gcloud auth print-access-token)" \ -H "x-goog-user-project: PROJECT_ID/" \ "https://recommender.googleapis.com/v1/billingAccounts/BILLING_ACCOUNT_ID/locations/global/\ insightTypes/google.resourcemanager.projectUtilization.Insight/insights"
Replace the following:
BILLING_PROJECT_ID
: The ID of the project that you've enabled with the Recommender API (this would be the same PROJECT_ID as given in the previous step).BILLING_ACCOUNT_ID
: The ID of your Google Cloud Billing Account.
Configuring recommendation parameters
This section describes how to view and configure this recommender using
gcloud
and curl
.
The following table lists various parameters that can be configured:
Parameter | Description | Expected Values |
---|---|---|
minimum_observation_period |
Minimum observation period defines the number of days of Project activity to be analyzed, before a Recommendation is generated.
Setting this value is currently only supported at the Organization level. |
"P30D"[DEFAULT], "P60D", "P90D", "P180D", "P270D", "P365D" |
gcloud
To view and update recommender configurations using gcloud
, follow the steps below.
Viewing configurations
To view Unattended Project Recommender configurations for a given organization, run the following command from a project belonging to that organization which has the Recommender API enabled:
gcloud beta recommender recommender-config describe \ google.resourcemanager.projectUtilization.Recommender \ --organization=ORGANIZATION_ID \ --location=global
Replace the following:
ORGANIZATION_ID
: The organization ID.
Example Output:
etag: ETAG name: organizations/ORGANIZATION_ID/locations/global/recommenders/google.resourcemanager.projectUtilization.Recommender/config recommenderGenerationConfig: params: minimum_observation_period: P30D revisionId: DEFAULT updateTime: '2022-04-19T00:57:55Z'
Note the ETAG value:
ETAG
: This value will be used inrecommender-config update
.
Updating configurations
To update Unattended Project Recommender configurations for a given organization, pick a project belonging to that organization where you have enabled the Recommender API, create a
config.gcloud
file:{ "params": { "minimum_observation_period": DURATION } }
Replace the following:
DURATION
: One value from "P30D", "P60D", "P90D", "P180D", "P270D", "P365D"
Run the following command:
gcloud beta recommender recommender-config update \ google.resourcemanager.projectUtilization.Recommender \ --organization=ORGANIZATION_ID \ --location=global \ --config-file=config.gcloud \ --etag=ETAG
Replace the following:
ORGANIZATION_ID
: The organization ID.ETAG
: The etag value that was returned fromrecommender-config describe
.
API
To view and update recommender configurations, you can use curl
to send a request to the Recommender APIs. To authorize requests from the command line,
you use OAuth access tokens. An OAuth access token is a string that grants temporary access to an API.
Viewing configurations
To view Unattended Project Recommender configurations for a given organization, run the following command from a project belonging to that organization which has the Recommender API enabled:
curl -X GET \ -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ https://recommender.googleapis.com/v1beta1/organizations/ORGANIZATION_ID/locations/global/recommenders/google.resourcemanager.projectUtilization.Recommender/config
Replace the following:
PROJECT_ID
: The project ID.ORGANIZATION_ID
: The organization ID.
Example Output:
{ "name": "organizations/ORGANIZATION_ID/locations/global/recommenders/google.resourcemanager.projectUtilization.Recommender/config", "recommenderGenerationConfig": { "params": { "minimum_observation_period": "P60D" } }, "etag": ETAG, "updateTime": "2022-04-21T18:51:17.407255Z", "revisionId": "75c3b0d7" }
Note the ETAG value:
ETAG
: This value will be used incurl PATCH
.
Updating configurations
To update Unattended Project Recommender configurations for a given organization, pick a project belonging to that organization where you have enabled the Recommender API, create a
request.json
file:{ "name": "organizations/ORGANIZATION_ID/locations/global/recommenders/google.resourcemanager.projectUtilization.Recommender/config", "recommenderGenerationConfig": { "params": { "minimum_observation_period": DURATION } }, "etag": ETAG, }
Replace the following:
DURATION
: One value from "P30D", "P60D", "P90D", "P180D", "P270D", "P365D"ETAG
: The etag value that was returned fromCURL get
.
Run the following command line:
curl -X PATCH \ -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \ -H "x-goog-user-project: PROJECT_ID" \ -H "Content-Type: application/json; charset=utf-8" -d @request.json \ https://recommender.googleapis.com/v1beta1/organizations/ORGANIZATION_ID/locations/global/recommenders/google.resourcemanager.projectUtilization.Recommender/config
Replace the following:
PROJECT_ID
: The project ID.ORGANIZATION_ID
: The organization ID.
What's next
- Shutting down (deleting) projects
- Deleting projects in an organization
- Carbon Footprint reporting methodology
- Interactive tutorial (open in Cloud Shell)