v2.42.0 appears to break custom scopes for impersonated credentials in Vertex AI Pipelines #1854

The Problem

Upgrading from 2.41.1 to 2.42.0+ causes 403: Request had insufficient authentication scopes when accessing Google Sheets API from Vertex AI Pipelines. Same code, same service account, same IAM permissions.

Error:

google.api_core.exceptions.Forbidden: 403 POST https://sheets.googleapis.com/v4/spreadsheets/[ID]/values:batchUpdate:
Request had insufficient authentication scopes.

Context:

• Oct 28, 2024: v2.42.0 released (includes PR fix: Read scopes from ADC json for impersoanted cred #1820: "fix: Read scopes from ADC json for impersonated cred")
• Nov 1, 2024: Docker build installed 2.41.1 → pipeline succeeded
• Nov 2, 2024: Docker build installed 2.42.1 → 403 scope error

Only known change between builds was google-auth version (unpinned as google-auth>=2.0.0).

In addition:

• SA has Editor permission on the target Google Sheet
• SA able to successfully authenticate to other GCP APIs
• IAM audit logs show no permission changes between working and broken builds

Environment

Platform: Vertex AI Pipelines (KFP v2.14.5), Python 3.12 in Docker container

Authentication: Impersonated service account credentials via Vertex AI runtime (ADC from GCE metadata service)

Code:

from google.auth import default

credentials, _ = default(scopes=[
    'https://www.googleapis.com/auth/drive.readonly',
    'https://www.googleapis.com/auth/spreadsheets'
])

Works in 2.41.1, fails for us in 2.42.

Workaround

google-auth>=2.0.0,<2.42.0

Questions

• Is this now intended behaviour?
• If so, what's the recommended approach for Vertex AI Pipelines needing custom API scopes?