google.cloud.storage allows access but gcsfs does not

Question

I would like to connect to a bucket on Google Cloud storage using gcsfs. So far I have only been using the native google.cloud module but it turns out I actually need file like objects for a certain application so had to switch over.

My GOOGLE_APPLICATION_CREDENTIALS environment variable points to a JSON file on my local filesystem. Using google.cloud, I am able to access a bucket in cloud storage with no issues. However using gcsfs I am not. Here's some code to show what I mean:

from google.cloud import storage
import gcsfs
import google.auth

storage_client = storage.Client()
bucket = storage_client.bucket(...)
blob = bucket.blob(f'data/audio_wav/test.wav')
bts = blob.download_as_bytes()

credentials, _ = google.auth.default()
fs = gcsfs.GCSFileSystem(project=..., token=credentials)
folders = fs.ls('data') #throws error

As commented, the very last line throws what appears to be an authentication error:

google.auth.exceptions.RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', {'error': 'invalid_scope', 'error_description': 'Invalid OAuth scope or ID token audience provided.'})

I would think that if google.cloud.storage accepts my credentials, then so should gcsfs. Why do I get the above error, and how can I fix it?

Alternatively, is there a way to open a file like object using google.cloud.storage and not gcsfs?

JSON file

{
  "type": "service_account",
  "project_id": ...,
  "private_key_id": ....,
  "private_key": ...,
  "client_email": "[email protected]",
  "client_id": "113487991001810308893",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/ai-dj-884%40ai-dj-76527.iam.gserviceaccount.com"
}

Answer 1

As @John Hanley suggested in comments modify the mode of authentication (i.e., “token”) from "token=credentials" to "token=None", to use your default gcloud credentials or, attempt to get credentials from the Google metadata service, or fall back to anonymous access.

Refer to Credentials for more information.

To open a file using ‘“google.cloud.storage’” instead of "gcsfs" you can use the below method,

#create storage client  
storage_client = storage.Client('json-file-path’')  
#get bucket with name  
bucket = storage_client.get_bucket('bucket-name')  
#get bucket data as blob  
blob = bucket.get_blob('file-name’)

Answer 2

The error is with the scope of your credentials.

As @John Hanley mentioned you can use the 'googleapis.com/auth/cloud-platform' scope, this will not give you any scope errors, but your permissions are the only thing preventing the service account from accessing Google services.

The snippet below uses the GCS Readonly scope so will not work if you want to write back to GCS.

from google.cloud import storage
import gcsfs
import google.auth

credentials = google.auth.default()
scoped_credentials = credentials.with_scopes(['https://www.googleapis.com/auth/devstorage.read_only']) # This is what you are missing
fs = gcsfs.GCSFileSystem(project=..., token=scoped_credentials)
folders = fs.ls('data')