Grafana Container does not pick up changes from env file even after updated compose file and updated values in inspect command

Question

System Information

docker compose version
Docker Compose version v2.5.0

Design

I am using a Makefile that uses the multiple compose file logic using the -f flag in docker compose alongside the config command to build a docker-compose.yml. The Makefile can build the compose file, run it as well as bring it down and remove the file

Structure

├── conf
│   ├── grafana
│   │   ├── config
│   │   │   └── grafana.ini
|   |   |-- .env
├── docker-compose.base.yml
├── services
│   ├── docker-compose.grafana.yml

The conf directory has all the .env file where I pass the necessary admin credentials for Grafana

conf/grafana/.env

## Grafana Admin Credentials
GF_SECURITY_ADMIN_USER=admin
GF_SECURITY_ADMIN_PASSWORD=supersecretpass

docker-compose.base.yml

networks:
  internal:

services:
  grafana:
     env_file:
       - ./conf/grafana/.env
     volumes:
       - grafana:/var/lib/grafana
       - ./conf/grafana/config:/usr/local/etc/grafana

services/docker-compose.grafana.yml

services:
  grafana:
    image: grafana/grafana:8.4.5
    container_name: my-grafana
    environment:
      - GF_SERVER_ROOT_URL=/grafana
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - GF_PATHS_CONFIG=/usr/local/etc/grafana/grafana.ini
    logging:
      options:
        max-size: "1m"
    networks:
      - internal
    ports:
      - "3000:3000"
    security_opt:
      - "no-new-privileges:true"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro

Makefile

.PHONY: help build compose clean down run
.SILENT: help

SERVICES_DIR=services
COMPOSE_FILES:=-f docker-compose.base.yml

# STEP: 1 Add the service name here
define OPTIONS
    - grafana -
endef
export OPTIONS

ARGS=$(wordlist 2, $(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
$(eval $(ARGS):;@:)



ifeq (grafana, $(filter grafana,$(ARGS)))
    COMPOSE_FILES:=$(COMPOSE_FILES) -f $(SERVICES_DIR)/docker-compose.grafana.yml
endif

SERVICES:=$(filter-out ${OPTIONS},$(ARGS))

.PHONY: $(OPTIONS)

help:
    echo "refer to the README.md file"

build:
    make compose grafana

compose:
    docker compose $(COMPOSE_FILES) config > docker-compose.yml

clean:down
    rm -rf ./docker-compose.yml

down:
    docker compose down

run:build
    docker compose up -d $(SERVICES)

Problem Reproduction

1. I run the following:

    make run
   

   which will build the docker-compose.yml file in the root using the docker compose config command and using the -f services/docker-compose.grafana.yml command with the base file.

2. Once the container is up and reachable on localhost:3000 I check by entering the password and it works

3. Now I change the password in the conf/grafana/.env to supersecretpass2 and run the make run again

   This actually rewrites the docker-compose.yml file with the newly updated environment variables for grafana service and re-runs the docker compose up command again which should pick up the new configuration i.e., new password.

Problem

Even though the docker-compose.yml is updated and the CLI states that the container is recreated and restarted, upon entering the new password the Grafana UI does not pick up the adapted environment variable

Inspection

upon doing

docker inspect my-grafana

I can clearly see the

  "StdinOnce": false,
            "Env": [
                "GF_SECURITY_ADMIN_PASSWORD=supersecretpass2",
                "GF_SERVER_SERVE_FROM_SUB_PATH=true",
                "GF_SECURITY_ADMIN_USER=admin",
                "GF_PATHS_CONFIG=/usr/local/etc/grafana/grafana.ini",
                "GF_SERVER_ROOT_URL=/grafana",
                "PATH=/usr/share/grafana/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GF_PATHS_DATA=/var/lib/grafana",
                "GF_PATHS_HOME=/usr/share/grafana",
                "GF_PATHS_LOGS=/var/log/grafana",
                "GF_PATHS_PLUGINS=/var/lib/grafana/plugins",
                "GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
            ],

by executing:

docker compose exec -it grafana /bin/bash env

I can see that the updated env var of the password is passed into the container, however Grafana does not pick this changes up.

the UI mentions the password is invalid, but accepts the original password.

Repo for Bug Reproduction

In order to reproduce this bug I have the following repo