Copying a file to a Docker image from another Docker image: Re-uploads this file to Docker Hub every time as if it was new

Question

I have a dockerfile that I copy in a large (2.7GB) grid file for mapping. I created a second docker image (hosted on docker hub) that hosts this file so when in development, I am not uploading that file every time I rebuild the docker. That part of the image is cached on my computer. All of that works fine.

The issue I run into is when I am pushing my updated image to the docker hub, it treats this file in the image as if it was new every time. This makes the image take about ~15 minutes to upload to the hub. I'm asking to see if anyone has a solution to this. The way I am currently adding in the file is..

COPY --from=dockerimage:latest /file.grd /path/to/place/file/in/image

Note: The latest image has never changed. I do not edit the file or do anything with the directory is in after copying it.

Thanks in advance!