- 2.17.0 (latest)
- 2.16.0
- 2.15.0
- 2.14.0
- 2.13.0
- 2.12.0
- 2.11.0
- 2.10.0
- 2.9.0
- 2.8.0
- 2.7.0
- 2.6.0
- 2.5.0
- 2.4.0
- 2.3.0
- 2.2.1
- 2.1.0
- 2.0.0
- 1.44.0
- 1.43.0
- 1.42.3
- 1.41.1
- 1.40.0
- 1.39.0
- 1.38.0
- 1.37.1
- 1.36.2
- 1.35.1
- 1.34.0
- 1.33.0
- 1.32.0
- 1.31.2
- 1.30.0
- 1.29.0
- 1.28.1
- 1.27.0
- 1.26.0
- 1.25.0
- 1.24.1
- 1.23.0
- 1.22.0
- 1.21.0
- 1.20.0
- 1.19.0
- 1.18.0
- 1.17.0
Configuring Timeouts and Retries
When using object methods which invoke Google Cloud Storage API methods, you have several options for how the library handles timeouts and how it retries transient errors.
Configuring Timeouts
For a number of reasons, methods which invoke API methods may take longer than expected or desired. By default, such methods all time out after a default interval, 60.0 seconds. Rather than blocking your application code for that interval, you may choose to configure explicit timeouts in your code, using one of three forms:
- You can pass a single integer or float which functions as the timeout for the entire request. E.g.:
bucket = client.get_bucket(BUCKET_NAME, timeout=300.0) # five minutes
- You can also be passed as a two-tuple,
(connect_timeout, read_timeout)
, where theconnect_timeout
sets the maximum time required to establish the connection to the server, and theread_timeout
sets the maximum time to wait for a completed response. E.g.:
bucket = client.get_bucket(BUCKET_NAME, timeout=(3, 10))
- You can also pass
None
as the timeout value: in this case, the library will block indefinitely for a response. E.g.:
bucket = client.get_bucket(BUCKET_NAME, timeout=None)
NOTE: Depending on the retry strategy, a request may be repeated several times using the same timeout each time.
See also:
Configuring Retries
NOTE: For more background on retries, see also the GCS Retry Strategies Document
Methods which invoke API methods may fail for a number of reasons, some of which represent “transient” conditions, and thus can be retried automatically. The library tries to provide a sensible default retry policy for each method, base on its semantics:
For API requests which are always idempotent, the library uses its
DEFAULT_RETRY
policy, which retries any API request which returns a “transient” error.For API requests which are idempotent only if the blob has the same “generation”, the library uses its
DEFAULT_RETRY_IF_GENERATION_SPECIFIED
policy, which retries API requests which returns a “transient” error, but only if the original request includes anifGenerationMatch
header.For API requests which are idempotent only if the bucket or blob has the same “metageneration”, the library uses its
DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED
policy, which retries API requests which returns a “transient” error, but only if the original request includes anifMetagenerationMatch
header.For API requests which are idempotent only if the bucket or blob has the same “etag”, the library uses its
DEFAULT_RETRY_IF_ETAG_IN_JSON
policy, which retries API requests which returns a “transient” error, but only if the original request includes anETAG
in its payload.For those API requests which are never idempotent, the library passes
retry=None
by default, suppressing any retries.
Rather than using one of the default policies, you may choose to configure an explicit policy in your code.
- You can pass
None
as a retry policy to disable retries. E.g.:
bucket = client.get_bucket(BUCKET_NAME, retry=None)
- You can pass an instance of
google.api_core.retry.Retry
to enable retries; the passed object will define retriable response codes and errors, as well as configuring backoff and retry interval options. E.g.:
from google.api_core import exceptions
from google.api_core.retry import Retry
_MY_RETRIABLE_TYPES = [
exceptions.TooManyRequests, # 429
exceptions.InternalServerError, # 500
exceptions.BadGateway, # 502
exceptions.ServiceUnavailable, # 503
]
def is_retryable(exc):
return isinstance(exc, _MY_RETRIABLE_TYPES)
my_retry_policy = Retry(predicate=is_retryable)
bucket = client.get_bucket(BUCKET_NAME, retry=my_retry_policy)
- You can pass an instance of
google.cloud.storage.retry.ConditionalRetryPolicy
, which wraps aRetryPolicy
, activating it only if certain conditions are met. This class exists to provide safe defaults for RPC calls that are not technically safe to retry normally (due to potential data duplication or other side-effects) but become safe to retry if a condition such as if_metageneration_match is set. E.g.:
from google.api_core.retry import Retry
from google.cloud.storage.retry import ConditionalRetryPolicy
from google.cloud.storage.retry import is_etag_in_json
def is_retryable(exc):
... # as above
my_retry_policy = Retry(predicate=is_retryable)
my_cond_policy = ConditionalRetryPolicy(
my_retry_policy, conditional_predicate=is_etag_in_json)
bucket = client.get_bucket(BUCKET_NAME, retry=my_cond_policy)
Retry Module API
class google.cloud.storage.retry.ConditionalRetryPolicy(retry_policy, conditional_predicate, required_kwargs)
Bases: object
A class for use when an API call is only conditionally safe to retry.
This class is intended for use in inspecting the API call parameters of an
API call to verify that any flags necessary to make the API call idempotent
(such as specifying an if_generation_match
or related flag) are present.
It can be used in place of a retry.Retry
object, in which case
_http.Connection.api_request
will pass the requested api call keyword
arguments into the conditional_predicate
and return the retry_policy
if the conditions are met.
Parameters
retry_policy (class:google.api_core.retry.Retry) – A retry object defining timeouts, persistence and which exceptions to retry.
conditional_predicate (callable) – A callable that accepts exactly the number of arguments in
required_kwargs
, in order, and returns True if the arguments have sufficient data to determine that the call is safe to retry (idempotent).required_kwargs (list(str)) – A list of keyword argument keys that will be extracted from the API call and passed into the
conditional predicate
in order.
google.cloud.storage.retry.DEFAULT_RETRY( = <google.api_core.retry.Retry object )
The default retry object.
This retry setting will retry all _RETRYABLE_TYPES and any status codes from _ADDITIONAL_RETRYABLE_STATUS_CODES.
To modify the default retry behavior, create a new retry object modeled after
this one by calling it a with_XXX
method. For example, to create a copy of
DEFAULT_RETRY with a deadline of 30 seconds, pass
retry=DEFAULT_RETRY.with_deadline(30)
. See google-api-core reference
(https://googleapis.dev/python/google-api-core/latest/retry.html) for details.
google.cloud.storage.retry.DEFAULT_RETRY_IF_ETAG_IN_JSON( = <google.cloud.storage.retry.ConditionalRetryPolicy object )
Conditional wrapper for the default retry object.
This retry setting will retry all _RETRYABLE_TYPES and any status codes from
_ADDITIONAL_RETRYABLE_STATUS_CODES, but only if the request included an
ETAG
entry in its payload.
google.cloud.storage.retry.DEFAULT_RETRY_IF_GENERATION_SPECIFIED( = <google.cloud.storage.retry.ConditionalRetryPolicy object )
Conditional wrapper for the default retry object.
This retry setting will retry all _RETRYABLE_TYPES and any status codes from
_ADDITIONAL_RETRYABLE_STATUS_CODES, but only if the request included an
ifGenerationMatch
header.
google.cloud.storage.retry.DEFAULT_RETRY_IF_METAGENERATION_SPECIFIED( = <google.cloud.storage.retry.ConditionalRetryPolicy object )
Conditional wrapper for the default retry object.
This retry setting will retry all _RETRYABLE_TYPES and any status codes from
_ADDITIONAL_RETRYABLE_STATUS_CODES, but only if the request included an
ifMetagenerationMatch
header.
google.cloud.storage.retry.is_etag_in_json(data)
Return True if an etag is contained in the JSON body.
Indended for use on calls with relatively short JSON payloads.
google.cloud.storage.retry.is_generation_specified(query_params)
Return True if generation or if_generation_match is specified.
google.cloud.storage.retry.is_metageneration_specified(query_params)
Return True if if_metageneration_match is specified.