Introducing Cloud Storage soft delete

Soft delete provides bucket-level protection against accidental or malicious deletion by retaining recently deleted objects for a retention period that you select—seven days by default, which can be increased up to 90 days or be disabled altogether. Using a self-service restore capability, you can recover from these unfortunate events after they occur.

Normally, deleting an object cannot be undone. However, when soft delete is enabled, deleted objects enter a soft-deleted state, from which the object can still be restored. This will occur regardless of the reason for the deletion—whether an object was deleted by the delete API, overwritten by the insert/copy/rewrite API, deleted through the UI, or deleted due to an object lifecycle management policy.

Soft-deleted objects are special non-readable objects that are hidden from object listings unless a specific option is specified. Soft-deleted objects will be permanently deleted after they have been retained for the specified soft delete retention period, and once objects enter a soft-deleted state that retention period cannot be overridden (there is no way to permanently delete a soft-deleted object prematurely). This provides a significant level of protection against both accidental and malicious deletion events.

Restores copy one or more soft-deleted objects back into the same bucket from which they were deleted, making them accessible again as live objects. You can perform a synchronous restore by providing a specific list of objects, or you can execute a long-running asynchronous operation that restores all objects deleted between two timestamps. As newly created objects, restored objects have new creation dates. When Autoclass is in use, all restored objects will restart life in Standard storage class. Otherwise, restored objects will be created in the same storage class as they were when they were deleted.

It is important to set the soft delete retention time long enough to be able to detect and complete a restore in the case of a regretted deletion event. At an approximate restoration rate of 10 million objects per hour, it can take four days to restore a billion objects so it is wise to increase the retention period beyond 7 days for very large buckets.

Soft delete also provides protection against bucket-level deletion events. In the case of a bucket-level deletion, first contact Google for assistance in recreating the deleted bucket. You can then use the restore capability to restore the objects in that bucket.

Soft delete is in a promotional period from launch in March 2024 until August 31, 2024, with no additional charge for storing the first seven days of soft deleted data. Starting on September 1, 2024, we will start billing usage to existing storage SKUs at existing prices for all the time objects spend in soft deleted state since they are deleted. Because soft delete retains data for seven days by default, be sure to disable soft delete before August 25, 2024 to avoid all billing impact.

Before September 1, 2024 there will be no billing impact from soft delete unless you perform a restore and/or you increase the soft delete retention times on your buckets. This is intended to give you a reasonable amount of time to assess the future billing impact of the soft delete feature and make informed decisions as to the best soft delete settings to use across your buckets based on your budget and business needs.

Since all soft delete usage will be billed to existing SKUs, any existing discounts will continue to apply to any charges deriving from the soft delete feature.

Please note that on September 1, 2024, the documented Object Lifecycle Management billing exception for deletes with only an age condition will no longer apply for buckets with soft delete enabled, consistent with object versioning.

The primary billing impact from soft delete will be additional billed monthly storage charges for the usage associated with soft deleted data. Once an object is soft-deleted, we will continue to bill usage against the existing storage SKUs based on location and storage class until the end of its soft-delete retention period. For example, a Standard storage class object in us-east4 would be billed the us-east4 Standard storage class SKU while it is a live object and then continue to be billed at that same rate against the same SKU after being soft-deleted, until it has completed its soft-delete retention period (seven days, by default).

As mentioned above, during the promotional period from launch until August 31, 2024, there will be no additional charge for storing the first seven days of soft deleted data.

The time objects spend in soft-deleted state will count towards any applicable minimum storage duration based on storage class and Autoclass status. This is to your advantage because it will be easier to meet the minimum storage durations and avoid early delete charges when soft delete is enabled. For example, the Nearline storage class has a 30 day minimum storage duration in buckets not using Autoclass. Deleting an object after 23 days without soft delete would incur seven days of early delete charges. With soft delete enabled for the default 7 days, the object would be charged 30 days of storage, including the soft-delete period, so no early delete charges would apply.

There are no per-GiB processing charges for performing a restore. This includes not charging any retrieval fees when restoring Nearline, Coldline, or Archive objects.

The primary charge for actions associated with restores are class A operations associated with the location type of your bucket. For synchronous restores where you provide a list of specific objects to restore, we will meter one Class A operation for every object restored. This will always be billed as a Standard Class A operation, regardless of the actual storage class of the object because we do not want to penalize restores of colder objects. For asynchronous restores where we first have to determine what objects to restore, we will also bill one Standard Class A operation per thousand objects scanned prior to starting the restore.

Since a restore creates a new live object with a new create date in your bucket, that new object will be billed normally once created by the restore process and all normal pricing and storage duration requirements will apply to the new object. We will continue to bill the soft-deleted versions of these restored objects, although there will typically only be a few days of overlap based on the seven-day default soft-delete retention period.

Google decided to make soft delete on by default because we believe it is a high value feature that most customers will benefit from and that in most cases will have only a modest billing impact. However, you may decide that the seven-day soft delete retention policy is not suitable for some or all your Cloud Storage buckets. Prior to August 25, 2024, you should disable soft delete on any buckets where you do not need this protection—for example on buckets containing a large amount of short-lived temporary data. Conversely, you can also choose to increase the seven-day retention period to as long as 90 days on buckets where more protection is desired for business-critical data.

In addition to the information below, you may want to review the blog post Managing Cloud Storage soft delete at scale for best practices and sample scripts to help you evaluate the suitability of soft delete for your buckets and to automate settings. This post also includes how to modify soft delete settings in Terraform templates.

We are enhancing our Cloud Monitoring storage metrics so that you can inspect the amount of current vs non-current vs soft-deleted bytes on any bucket.

For buckets where soft delete is already enabled, the easiest way to inspect the billing impact of the soft delete feature is to examine the storage/v2/total_bytes metric, which gives you the total size of all objects in the bucket, grouped by storage class and object type (live, noncurrent, soft-deleted) at the end of the last usage day. If you compare the percentage of soft-deleted bytes relative to the total you can get a pretty good estimate of the billing impact of soft delete on your monthly storage charges (provided that your deletions are performed at a relatively steady rate so they would be captured by this metric).

We are also adding a new storage/v2/deleted_bytes metric which provides a delta count of deleted bytes per bucket, grouped by storage class. Even if soft delete is disabled, you can compare your delete rate using this metric with the total_bytes metric to estimate the billing impact of soft delete on a particular bucket.

Examples:

• To calculate the absolute billing impact of soft delete, you can use the storage/v2/deleted_bytes metric which measures the number of bytes deleted over time. The absolute cost of soft delete can be calculated as follows: soft delete retention duration × deleted bytes × $/GiB mo price. For example, the cost of enabling a 7-day soft delete policy on a bucket in us-central1 in Standard storage at list price with 100,000 GB of deletions during the course of the year is 7 / 30.4375 × 100,000 × $0.02 = $459.96 (where 30.4375 is the average number of days per month).
• To calculate the relative cost of soft delete, you can combine the storage/v2/deleted_bytes metric with the storage/v2/total_byte_seconds metric: soft delete retention duration × deleted bytes / total bytes. Continuing from the above example and given 1,000,000 GB-months of storage over the entire course of the year, the relative cost of enabling soft delete in this case is: 7 / 30.4375 × 100,000 / 1,000,000 = ~2% impact.

For those of you using object versioning, please note that we are exposing the live vs non-current bytes and object counts as part of the new metrics which is new visibility that may be useful even for those of you who opt out of the soft delete feature.

Full information on the enhanced metrics will be available in the Storage Metrics documentation as soon as the new metrics are available.

The soft delete retention period can be adjusted for each bucket individually. To disable soft delete, change the retention period to zero. We will be providing sample scripts to help you evaluate the suitability of soft delete for all your buckets, and to automate updating the settings—even on millions of buckets. You may also want to consider customizing the soft delete settings in Terraform or KCC scripts and other bucket creation workflows so that new buckets are created with settings that meet your business needs. 

Finally, you can create organization policy constraints to enforce particular settings for soft delete for newly created buckets.