host.docker.internal will not resolve in docker-compose

Question

So I have a .NET solution with two projects. These work fine when running with dotnet run, but I'm having issues with my docker compose. When adding env variables with url paths to talk between containers, i generally use something like host.docker.internal to resolve the path to the other container, but for some reason that doesn't resolve and just gets used as, for instance, https://host.docker.internal:49833/connect/authorize instead of https://localhost:49833/connect/authorize.

This doesn't make sense to me as i literally have another project on my machine that runs with this setup just fine. what am i missing?

project 1 has a dockerfile like this:

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app

# Copy csproj and restore as distinct layers
COPY ["RecipeManagement/src/RecipeManagement/RecipeManagement.csproj", "./RecipeManagement/src/RecipeManagement/"]
#COPY ["SharedKernel/SharedKernel.csproj", "./SharedKernel/"]
RUN dotnet restore "./RecipeManagement/src/RecipeManagement/RecipeManagement.csproj"

# Copy everything else and build
COPY . ./
RUN dotnet build "RecipeManagement/src/RecipeManagement/RecipeManagement.csproj" -c Release -o /app/build

FROM build-env AS publish
RUN dotnet publish "RecipeManagement/src/RecipeManagement/RecipeManagement.csproj" -c Release -o /app/out

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY --from=publish /app/out .

ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080

ENTRYPOINT ["dotnet", "/app/RecipeManagement.dll"]

project 2 has a dockerfile like this:

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build-env
WORKDIR /app

RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
RUN apt install -y nodejs

# Copy csproj and restore as distinct layers
COPY ["AuthServerWithDomain/AuthServerWithDomain.csproj", "./AuthServerWithDomain/"]
#COPY ["SharedKernel/SharedKernel.csproj", "./SharedKernel/"]
RUN dotnet restore "./AuthServerWithDomain/AuthServerWithDomain.csproj"

# Copy everything else and build
COPY . ./
RUN dotnet build "AuthServerWithDomain/AuthServerWithDomain.csproj" -c Release -o /app/build

FROM build-env AS publish
RUN dotnet publish "AuthServerWithDomain/AuthServerWithDomain.csproj" -c Release -o /app/out

# Build runtime image
FROM mcr.microsoft.com/dotnet/aspnet:6.0
WORKDIR /app
COPY --from=publish /app/out .

ENV ASPNETCORE_URLS=http://+:8080
EXPOSE 8080

ENTRYPOINT ["dotnet", "/app/AuthServerWithDomain.dll"]

and the compose looks like this:

version: '3.7'

services:
  recipemanagement-db:
    image: postgres
    restart: always
    ports:
      - '63230:5432'
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres
      POSTGRES_DB: dev_recipemanagement
    volumes:
      - recipemanagement-data:/var/lib/postgresql/data

  recipemanagement-api:
    build:
      context: .
      dockerfile: RecipeManagement/src/RecipeManagement/Dockerfile
    ports:
      - "63231:8080"
    environment:
      ASPNETCORE_ENVIRONMENT: "Development"
      ASPNETCORE_URLS: https://+:8080;
      ASPNETCORE_Kestrel__Certificates__Default__Path: "/https/aspnetappcert.pfx"
      ASPNETCORE_Kestrel__Certificates__Default__Password: "password"
      DB_CONNECTION_STRING: "Host=recipemanagement-db;Port=5432;Database=dev_recipemanagement;Username=postgres;Password=postgres"
      "AUTH_AUDIENCE": "recipe_management"
      "AUTH_AUTHORITY": "https://host.docker.internal:49833"
#      ^^^^^^^^^^^ not sure why this and the below don't resolve *************************************************************************
      "AUTH_AUTHORIZATION_URL": "https://host.docker.internal:49833/connect/authorize"
      "AUTH_TOKEN_URL": "https://host.docker.internal:49833/connect/token"
      "AUTH_CLIENT_ID": "recipe_management.swagger"
      "AUTH_CLIENT_SECRET": "974d6f71-d41b-4601-9a7a-a33081f80687"
      "RMQ_HOST": "localhost:TBDRMQPORT"
      "RMQ_VIRTUAL_HOST": "/"
      "RMQ_USERNAME": "guest"
      "RMQ_PASSWORD": "guest"

    volumes:
      - ~/.aspnet/https:/https:ro
      
  recipemanagement-authserver:
    build:
      context: .
      dockerfile: AuthServerWithDomain/Dockerfile
    ports:
      - "49833:8080"
    environment:
      "ASPNETCORE_ENVIRONMENT": "Development"
      ASPNETCORE_URLS: "https://+:8080;"
      ASPNETCORE_Kestrel__Certificates__Default__Path: "/https/aspnetappcert.pfx"
      ASPNETCORE_Kestrel__Certificates__Default__Password: "password"

    volumes:
      - ~/.aspnet/https:/https:ro
        
volumes:
  recipemanagement-data:

Answer 1

See @DavidMaze comment. Best to search on Networking in Compose as the starting point.

(The Dockerfile's should be irrelevant to your question.) There are a couple of options, with trial and error to get the exact behavior you want. I'd encourage you to find better explanations on the following suggestions:

In your compose file, at the 'service' level, you can add extra_hosts

my-service:
  extra_hosts:
    host.docker.internal:host-gateway
    #host.docker.internal:127.0.0.1 for linux

But when using compose, a better option is to have docker create a network specific to your containers with docker network create --driver bridge my_recipe_ntwk

and then at the top level of your compose:

services:
volumes:
networks:
  my-private-ntwk:
    external:
      name: my_recipe_ntwk

and then at the 'service' level

my-service-1:
  networks:
    - my-private-ntwk

my-service-2:
  networks:
    - my-private-ntwk

For postgres running on the host, see this answer and there should be a few other similar answers.

I believe if all services are configured on the same bridge network, you actually use the service name, e.g. my-service-1:8080 and don't inclue the https. I might be wrong. (In my case, my containers connect to Postgres on the host itself, and that requries 'extra_hosts' and it's own special workaround.) You are hosting a Postgres container, so I believe your connection host and port, when using the bridge network, would simply be my-private-ntwk:5432