Slow performance to next hop from wireguard vpn gateway in AWS

Question

[ mac machine (pppoe gw) ]
   |
   | 100MB download in (~30s)
   |
[ vpn host (wireguard and nginx) AZ A ]


[ mac machine (pppoe gw) ]
   |
   | 
   |
[ vpn host (wireguard and nginx) AZ A ]
   |
   | 100MB download in (~70s)
   |
[nginx host AZ A]




[ vpn host (wireguard and nginx) AZ A ]
   |
   | Measured perf between vpn host and nginx host ~178MB/s
   |
[nginx host AZ A]

I was surprised to see a doubling in time when going to another hop rather than just to the gateway. All tests are using wget using the internal wireguard ips (so over the wireguard protocol.)

I've tried playing around with the MTU's, linux network stack tuning and mss clamping but nothing has magically worked to speed up the next hop.

dpkg -l | grep wireguard
ii  wireguard                         1.0.20200513-1~20.04              all          fast, modern, secure kernel VPN tunnel (metapackage)
ii  wireguard-dkms                    1.0.20201112-1~20.04.1            all          fast, modern, secure kernel VPN tunnel (DKMS version)
ii  wireguard-tools                   1.0.20200513-1~20.04.2            amd64        fast, modern, secure kernel VPN tunnel (userland utilities)