Quotas and limits

The following sections explain quotas and limits for Dataproc Metastore. To change a quota, request additional quota by using the Google Cloud console. Limits generally cannot be increased unless specifically noted.

Quotas and limits

Dataproc Metastore services

Item Quotas and limits Notes
Services per single region 20 services When a project violates Google Cloud service policy, its per-region service quota is significantly lowered.
Services per multi-region 5 services When a project violates Google Cloud service policy, its per-region service quota is significantly lowered.

Metadata storage for Dataproc Metastore services

Item Quotas and limits Notes
Metadata storage per Developer tier service 10 GB
Metadata storage per Enterprise tier service 1 TB

Queries per second for Dataproc Metastore projects

Item Quotas and limits Notes
Queries per second (QPS) per project 20 QPS
Queries per second (QPS) per project per user 5 QPS
Maintenance queries per second (QPS) per project 20 QPS The following Hive Metastore requests consume this quota:
  • createTable
  • createDatabase
  • deleteTable
  • deleteDatabase
    		•

    Job timeout limit for Dataproc Metastore

    Item Quotas and limits Notes
    Job timeout limit for export/backup/restore 1 hour
    Job timeout limit for import 3 hours

    Quotas management

    Dataproc Metastore enforces quotas on resource usage for a variety of reasons. For example, quotas protect the community of Google Cloud users by preventing unforeseen spikes in usage. Quotas also help users who are exploring Google Cloud with the free tier to stay within their trial.

    All projects start with the same quotas, which you can change by requesting additional quota. Some quotas may increase automatically based on your use of a product.

    Permissions

    To view quotas or request quota increases, Identity and Access Management (IAM) principals need one of the following roles.

    Task Required role
    Check quotas for a project Project Owner or Editor or Quota Viewer
    Modify quotas, request additional quota Project Owner or Editor, Quota Administrator, or custom role with the serviceusage.quotas.update permission

    Check your quota

    Console

    1. In the Google Cloud console, go to the Quotas page.

      Go to the Quotas page

    2. To search for the quota that you want to update, use the Filter table. If you don't know the name of the quota, use the links on this page instead.

    Errors when exceeding your quota

    If you exceed a quota with a gcloud command, gcloud outputs a quota exceeded error message and returns with the exit code 1.

    If you exceed a quota with an API request, Google Cloud returns the following HTTP status code: HTTP 413 Request Entity Too Large.

    Request additional quota

    Request additional quota from the Quotas page in the Google Cloud console. Quota requests take 24 to 48 hours to process.

    Console

    1. In the Google Cloud console, go to the Quotas page.

      Go to the Quotas page

    2. On the Quotas page, select the quotas that you want to change.

    3. At the top of the page, click Edit quotas.

    4. Fill out your name, email, and phone number, and then click Next.

    5. Fill in your quota request, and then click Done.

    6. Submit your request.

    Resource availability

    Each quota represents a maximum number for a particular type of resource that you can create, provided that resource is available. It's important to note that quotas do not guarantee resource availability. Even if you have available quota, you won't be able to create a new resource if it is not available.

    For example, you might have sufficient quota to create a new regional, external IP address in the us-central1 region, but that would not be possible if there were no available external IP addresses in that region. Zonal resource availability can also affect your ability to create a new resource.

    Situations where resources are unavailable in an entire region are rare; however, resources within a zone can be depleted from time to time, typically without impact to the service level agreement (SLA) for the type of resource. For more information, review the relevant SLA for the resource.

    What's next