Add Retry Strategy to manual libraries.

@crwilcox

Feature request to track work done to fix retry strategy issues in Python libraries.

Issues that have a workaround will be deduped against this bug as @crwilcox, @tritone and I organize the effort.

This might be possible with the gcs client currently via

session = requests.Session()
session.mount("https://storage.googleapis.com", requests.adapters.HTTPAdapter(max_retries=5))
client = Client(project=env.GOOGLE_PROJECT, _http=session)

Haven't tested though

@micimize

@micimize This is giving me errors (below) on deployment. Any idea why?

I'm assuming Client here is really:
from google.cloud import storage
storage.Client(...) ?

Code snippet:

from google.cloud import storage
import requests
session = requests.Session()
session.mount("https://storage.googleapis.com", requests.adapters.HTTPAdapter(max_retries=5))
client = storage.Client(project=PROJECT_ID, _http=session)

Error when attempting blob.download_as_string():

ERROR: (gcloud.functions.deploy) OperationError: code=3, message=Function failed on loading user code. Error message: 401 GET https://storage.googleapis.com/download/storage/v1/b/<redacted file name>/o/secrets?alt=media: ('Request failed with status code', 401, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PARTIAL_CONTENT: 206>)

@dylancaponi

@dylancaponi does it work in the same environment without the _http argument? 401 means unauthorized, so you probably need to "bind the credentials" like the docs say the default does:

If not passed, an _http object is created that is bound to the credentials for the current object

They broke the monorepo up, but I found where they set credentials for reference:

        if self._http_internal is None:
            self._http_internal = google.auth.transport.requests.AuthorizedSession(
                self._credentials,
                refresh_timeout=_CREDENTIALS_REFRESH_TIMEOUT,
            )

Since these retry-able errors happen quite frequently, I'd really like for this to be a feature sooner than later.
Thanks for your efforts!

@micimize

@micimize Yes, the code runs when we don't pass _http. That is how our normal (non-retrying) connection is established.

builtins.storage_client = storage.Client(project='<redacted>')

However, that means it's not passing the session that defines 5 retries which I really want to do. Do you know what the proper approach to that is?

@dylancaponi

@dylancaponi I don't, but what I would do is keep digging through the source until you find google.auth.transport.requests.AuthorizedSession. You basically want to extract the auth logic they're using and figure out how to combine it with the requests.adapters.HTTPAdapter(max_retries=5) solution, and then pass the resulting session in as _http

Thanks! I think I want to do something like:

storage_client = storage.Client()
adapter = requests.adapters.HTTPAdapter(max_retries=5)
storage_client._http.mount("https://", adapter)
storage_client._http._auth_request.session.mount("https://", adapter)

as mentioned here:
googleapis/python-bigquery#59 (comment)

After 8 days of testing, this function did not terminate once due to connectionResetError!

Adding to production code and will report back.

@micimize

We implemented this in production and have seen zero errors after 5 days.

Previously we were seeing ~10 errors a day.

This solved our issues. Thanks @micimize & @pietrodn

def add_client_retries(client):
    """ Retry connection to prevent connectionResetError
    """
    adapter = requests.adapters.HTTPAdapter(max_retries=5)
    client._http.mount("https://", adapter)
    client._http._auth_request.session.mount("https://", adapter)
    return client

in main.py:

from google.cloud import storage
from google.cloud import bigquery
storage_client = add_client_retries(storage.Client())
bq_client = add_client_retries(bigquery.Client())

Also, remember to only initialize these once outside of your main() function and reuse them in main().

The solution above worked for a month or two but we are now experiencing this issue again ...
How is everyone else faring?

This is now resolved for storage. Retries are supported by default, documentation work for configuring them is pending (for now, see docstrings).

@andrewsg

@andrewsg : What release version of google-cloud-storage is retries supported by default?

Navigation Menu

Repository navigation

Add Retry Strategy to manual libraries. #338

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add Retry Strategy to manual libraries. #338

Description

Activity

micimize commented on Jan 21, 2020

dylancaponi commented on Mar 3, 2020

micimize commented on Mar 3, 2020

micimize commented on Mar 3, 2020

RunOrVeith commented on May 13, 2020

dylancaponi commented on Jul 9, 2020

micimize commented on Jul 9, 2020

dylancaponi commented on Jul 9, 2020

dylancaponi commented on Jul 21, 2020

dylancaponi commented on Dec 4, 2020

andrewsg commented on Dec 11, 2020

jackkwok commented on May 17, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions