Docker build is so slow, how can I speed it up?

Question

The steps "[internal] load metadata for docker.io..." and "[internal] load build context" take soooooo long every time that I build my docker image that it's painful.

Below is an output of a recent build, and the above two steps mentioned (*) have taken almost 4 minutes combined. Is this normal?

 => [internal] load build definition from Dockerfile                                                             0.1s
 => => transferring dockerfile: 287B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.1s
 => => transferring context: 2B                                                                                    0.0s
 *=> [internal] load metadata for docker.io/library/python:3.8                                                     81.5s
 => [auth] library/python:pull token for registry-1.docker.io                                                      0.0s
 *=> [internal] load build context                                                                                167.5s
 => => transferring context: 3.59MB                                                                              167.4s
 => [1/6] FROM docker.io/library/python:3.8@sha256:53cb5152064a7e7b485ad42704ea63c5155b264c82e7f17de99d3aa28e4f89  0.0s
 => CACHED [2/6] WORKDIR /code                                                                                     0.0s
 => CACHED [3/6] COPY requirements.txt /code                                                                       0.0s
 => CACHED [4/6] RUN pip install --upgrade pip                                                                     0.0s
 => [5/6] RUN pip install -r requirements.txt                                                                     40.7s
 => [6/6] COPY . .                                                                                                 8.7s
 => exporting to image                                                                                             4.1s
 => => exporting layers                                                                                            4.1s
 => => writing image sha256:25a9f036db87ec77ef64e2dfae9cffe2973a947887face86d989cdaa60169216                       0.0s

Before I upgraded to Windows 11, I'm sure this part of the build would take several seconds. Especially once the image from Docker Hub was already downloaded onto my local machine. However, even after I've pulled down the image from Docker Hub (as in this case) it will still take several minutes re-pulling the same image.

Machine: Windows 11 Docker version: 4.5.1 Using WSL2 Ubuntu 20.04

Can anybody help provide any suggestions on why the Docker build takes so long?

Thanks in advance