How do I prevent access to website by IP on kestrel?

Question

I have a aspnet core 6.0 web app running on port 443 using kestrel on Ubuntu 20.04. The certificate and origin pulls are set up and all work!

However, I can still access my site by going to https://<my server ip>.

Is there a way I can prevent access this way?

My appsettings.json:

"Kestrel": {
"HttpsDefaults": {
  "ClientCertificateMode": "RequireCertificate"
},
"Endpoints": {
  "Https": {
    "Url": "https://<my domain>:443",
    "Certificate": {
      "Path": "/etc/ssl/certificate.pfx",
      "Password": "",
      "AllowInvalid": false
    }
  }
}

}