AuthMetadataPlugin: gRPC requires it to not block; causes timeouts to not be respected #351

The gRPC documentation for AuthMetadataPlugin.__call__ states: "Implementations of this method must not block." Unfortunately, google.auth.transport.grpc.AuthMetadataPlugin violates this requirement. I believe this causes all gRPC calls using one instance of AuthMetadataPlugin to be blocked as long as token refreshing is happening. This may be causing occasional errors where the timeout we pass to Google Cloud API calls is not being respected, since the calls are all blocked waiting on the OAuth token to be refreshed.

Possible fix

It seems likely this plugin needs to do OAuth token refreshing using a thread pool, like the grpc._auth.GoogleCallCredentials implementation does. This will have the side benefit of not having multiple threads "race" to refresh the credentials quite as much.

A quick-and-dirty "fix" that would reduce the impact of this error would be to add a timeout on the token refreshing requests. This is probably a good idea anyway. However, this would at least ensure the "impact" of slow/blocked token refreshes is minimized. On our case, we have seen processes that may have been "stuck" for ~15 minutes due to this issue.

Blocking call path

When using google.api_core.grpc_helpers.create_channel, which seems to be what is used by Google's generated GAPIC clients:

1. gRPC calls google.auth.transport.grpc.AuthMetadataPlugin.__call__(...)
2. Calls google.auth.transport.grpc.AuthMetadataPlugin._get_authorization_headers(...)
3. Calls self._credentials.before_request(...) which is google.auth.credentials.Credentials.before_request
4. If the credentials have expired, it calls self.refresh(...). In the case of service accounts, this is a google.oauth2.service_account.Credentials object.
5. Calls _client.jwt_grant(...) which calls _token_endpoint_request(...)
6. Calls request(...) which is an instance of google.auth.transport.requests.Request
7. This calls in to the requests API, which eventually makes a blocking HTTP request.