Manage Data Access audit log costs

Bigtable is typically used for large, high-volume workloads. As a result, if you don't manage the log volume, Bigtable can generate an extremely high number of DATA_READ and DATA_WRITE logs, leading to unexpectedly high log storage costs. If you use Data Access audit logging, you should take steps to manage the log volume.

When you follow the best practices for Bigtable authentication, most Data Access audit log activity is generated by service accounts. A service account is an account that an application uses to authenticate and make API calls to Google Cloud services such as Bigtable. Managing service account logs is the most important step to reduce log volume. You might want to also limit logs using other criteria.

You can enable Data Access audit logging for Bigtable in the following ways:

After you enable audit logging, take the following steps to restrict the volume of logs.

Identify service accounts

First, identify the service accounts that you don't need logs for. The list of service accounts that are not useful and should not be logged depends on your application and business needs.To get a list of service accounts that have Cloud Bigtable API (Data API) permissions, you can search IAM policies for your organization. You can also view them on the IAM Permissions Google Cloud console page on the Principals tab.

Set up log restrictions

Next, set up your log restrictions. There are two ways to manage your Bigtable log volume by limiting service account logs. You can either exempt service accounts using audit configuration, or you can exclude service account logs using logs exclusion filters. For each method, you can either use the Cloud Logging API or the Google Cloud console.

Exempt service accounts using audit configuration

Exempting service accounts using audit configuration is the recommended approach because it lets you prevent certain logs from being generated in the first place. For detailed instructions, see the following:

Exclude service accounts using exclusion filters

Exclusion filters let you specify logs to be excluded from ingestion into your logs buckets. In this approach, logs are discarded after they have been created, so they still impose a processing load on the Bigtable service components that serve your data. Because of this load, we recommend that you use audit configuration instead. For more information on setting up filters using the Google Cloud console and the API, see Create a sink.

Estimate Data Access audit log costs

Because Bigtable is typically used for large, high-volume workloads, it has the potential to generate an extremely high number of logs. Before you enable Data Access audit logging for Bigtable, you should estimate and understand the Cloud Audit Logs ingestion and storage costs that audit logging can incur each month.

Your Data Access audit logging costs are directly related to the number of Bigtable requests that you choose to log each month. The following table shows rough estimates of the Cloud Audit Logs costs that you can expect based on your average requests per second and the length of time that you store your logs, assuming that you log all data requests. See Calculating your costs for a detailed explanation of how these estimates are calculated.

Average requests per second Log retention time Approximate monthly cost
1,000 30 $1,197
1,000 90 $1,246
10,000 30 $12,195
10,000 90 $12,684
100,000 30 $122,177
100,000 90 $124,621

Calculate your costs

Start with the following assumptions:

  • The number of seconds in an average month is about 2,628,000.
  • The average audit size is around 1 kb.
  • You are not charged for the first 50 GiB of audit logs that are ingested per month, and after you pass that amount, you're charged $0.50/GiB.
  • Storage is free for 30 days; after that you're charged $0.01/GiB for storage.

The method described on this page provides a gross estimate based on all traffic. In production, you are encouraged to restrict service account logging.

Calculate your monthly log volume

First, estimate the average amount of logs your traffic will generate in an average month.

  1. Gather the average number of requests per second that your application sends to Bigtable over the course of a month.
    • If you use client-side metrics, you can use them to determine your average queries per second (QPS) for the last month.
    • If you prefer to use your instance's Monitoring page in the Google Cloud console, use it to determine the average values for Read requests and Write requests per second over the last month, then add those two values together.
  2. Multiply the requests per second by 2,628,000 to get the average requests per month.
  3. Divide that number by 10e6, or 1,000,000. The result is the estimated monthly log volume in GB that you might generate each month.
  4. Multiply the monthly log volume in GB by .93 to get the approximate monthly log volume in GiB.

Calculate your ingestion costs

  1. Subtract 50 GiB from the monthly log volume in GiB that you calculated. There is no charge for the first 50 GiB.
  2. Multiply the remainder by $0.50 to arrive at your estimated monthly ingestion costs.

Calculate your storage costs

  1. If you plan to let your logs expire after 30 days, your cost for storage is $0.00.
  2. If you store your logs for longer than 30 days, your storage costs can be estimated by multiplying the monthly log volume by $0.01. These costs start to incur after the first month.

Detailed example

5,000 request per second, logs retained for 90 days

In this example, suppose that your average number of requests per second is 5,000 and you plan to keep your logs for 90 days. Using the steps on this page, you calculate the following estimates:

  • Multiply 5,000 by 2,628,000 to arrive at 13,140,000,000 requests per month.
  • Divide 13,140,000,000 by 10e6 to arrive at roughly 13,140 GB of monthly log volume.
  • Convert that number to GiB by multiplying it by .93 to arrive at 12,220.
  • Subtract 50 GiB from your monthly log volume to get 12,170 GiB.
  • Multiply by $0.50 to get $6,085 in ingestion costs.
  • For the first month that your logs exist, the storage cost is $0.
  • The second month, the log storage cost is 12,170 multiplied by $0.01, or about $122.
  • Every month after the second month, the monthly storage cost is double that, or $244.
  • After the second month, your estimated Data Access audit logging costs would be around $6,329 per month.

Presented in equation form, this looks like (((((5,000 rps * 2,628,000 sec)/1,000,000) * .93) - 50 GiB) * $0.50) + $122 = $6,207.

In this example, your monthly Data Access logging costs are around $6,329 per month.

What's next