Cloud Storage - metadata part is too large

Question

We have photo uploading functionality in the app which works for a lot of people, but some people get the following error:

"{
    "responseData": "Metadata part is too large."
}
"

when trying to upload a file.

I use Google Cloud Storage using FormData() to upload a file through POST using signed policy document.

This is how I issue the policy using nodejs latest cloud storage library (5.18.1):

        const gcsClient = new Storage({
            projectId: EnvConfig.gcpProjectId,
            credentials: getGoogleCloudStorageCredentials ()
        });

        const policyOpts = {
            equals: ['$Content-Type', contentType],
            expires: getTodaysDateSecondsAdded(900),
            contentLengthRange: {
                min: 0,
                max: 20000000 // 20MB
            }
        };

        const [{ base64, signature, string }] = await gcsClient
            .bucket(EnvConfig.gcpUploadBucket)
            .file(fileName)
            .getSignedPolicy(policyOpts);

and this is how it gets uploaded:

    const data = new FormData();

    data.append('policy', policy64);
    data.append('signature', policySignature);
    data.append('file', {uri: sourceUrl, type: contentType});
    data.append('key', fileName);
    data.append('GoogleAccessId', googleAccessId);
    data.append('Content-Type', contentType);

    const CancelToken = axios.CancelToken;
    const source = CancelToken.source();


    try {
          await axios.post(uploadUrl, data);
    } catch (e) {

      if (axios.isCancel(e)) {
        throw new Error(`axios upload image failed due to timeout`);
      }

      throw e;
    }

It is confusing to me what metadata part is being referenced to? Are we talking about the object metadata that's part of cloud storage concept? If yes, I have no control over it, I don't add any metadata anywhere? Or are we talking about the image metadata itself?

Is this just a bad error and it actually means that file size is too large?

According to this:

"There is a maximum combined size limit for all custom metadata keys and values of 8 KiB per object." - https://cloud.google.com/storage/quotas

So my current hypothesis is that somewhere a magic object metadata is generated that is over 8KB, is there some undocumented part of logic in cloud storage that automatically pulls custom metadata from POST request?

I saw bunch of potential duplicates but there's no answer out there:

Uploading File to GCS - Metadata part is too large

Is there a limit to the length of metadata values in Google Cloud Storage?

Answer 1

I have figured out after months of debugging,

basically when attaching file to form data, you also need to specify name: otherwise it will fail for bigger files >4MB+.

before (kept failing):

data.append('file', { uri: assetUri, type: contentType});

after:

data.append('file', { uri: assetUri, type: contentType, name: 'dummy.jpg' });

If you read react-native source code, you'll see the "name" will force emiting filename= in the multipart/formdata request

https://issuetracker.google.com/issues/225060183

Answer 2

The problem is how you are passing in the file:

    data.append('file', {uri: sourceUrl, type: contentType});

This is not actually setting a file as you intended. I believe the file is being passed as regular form data instead of a separate part in the multipart/form-data upload, and so is exceeding a limit. This format of call does not appear in the FormData doc you linked.

Instead, try actually creating a File object to pass:

    data.append('file', f /*an instance of File or Blob*/);