'running gsutil throws incorrect permission issue on: [/home/.config/gcloud/configurations]

I have a jar running in google dataproc cluster , which fails while trying to run gsutil copy command.

    Ensure your have the correct permissions on: [/home/.config/gcloud/configurations].
  Could not create directory [/home/.config/gcloud/configurations]: Permission denied.

Please verify that you have permissions to write to the parent directory.
22/03/11 13:22:17 ERROR org.apache.spark.deploy.yarn.ApplicationMaster: User class threw exception: org.apache.spark.sql.AnalysisException: Path does not exist: gs://us-aml-cdm-stream-stage/TRANS_SESSN_STREAM_TEMP/DatePartition=2022-02-22;
org.apache.spark.sql.AnalysisException: Path does not exist: 

However, I did notice that the path exist and gsutil copy command runs on terminal within the same cluster..

The service account seems to be active:

    gsutil version: 4.68
checksum: 0dbe8e7xxxxxxdef94b73330e09852a (OK)
boto version: 2.49.0
python version: 2.7.16 (default, Oct 10 2019, 22:02:15) [GCC 8.3.0]
OS: Linux 5.10.0-0.bpo.9-amd64
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /etc/boto.cfg
gsutil path: /usr/lib/google-cloud-sdk/platform/gsutil_py2/gsutil
compiled crcmod: False
installed via package manager: False
editable install: False


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source