MinIO cluster on different hosts by Docker

Question

Many post, forums and video has been checked, but I didn't see any solution.
I would like to build a MinIO Cluster with 4 different hosts with docker-compose.
Is there any solution?

Recent error message:

API: SYSTEM()
Time: 13:26:13 UTC 11/23/2021
Error: Read failed. Insufficient number of disks online (*errors.errorString)
       5: cmd/prepare-storage.go:268:cmd.connectLoadInitFormats()
       4: cmd/prepare-storage.go:317:cmd.waitForFormatErasure()
       3: cmd/erasure-server-pool.go:91:cmd.newErasureServerPools()
       2: cmd/server-main.go:637:cmd.newObjectLayer()
       1: cmd/server-main.go:542:cmd.serverMain()
Waiting for a minimum of 2 disks to come online (elapsed 1m4s)

Unable to read 'format.json' from http://10.10.10.1:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

Unable to read 'format.json' from http://10.10.10.2:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

Unable to read 'format.json' from http://10.10.10.3:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

Unable to read 'format.json' from http://10.10.10.4:9000/data: Server expects 'storage' API version 'v41', instead found 'v41' - *rolling upgrade is not allowed* - please make sure all servers are running the same MinIO version (DEVELOPMENT.2021-11-09T03-21-45Z)

docker-compose.yml

version: '2'

services:
  minio1:
    image: 'bitnami/minio:latest'
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      - MINIO_ACCESS_KEY=user
      - MINIO_SECRET_KEY=pass123
      - MINIO_DISTRIBUTED_MODE_ENABLED=yes
      - MINIO_DISTRIBUTED_NODES=10.10.10.1,10.10.10.2,10.10.10.3,10.10.10.4
      - MINIO_SKIP_CLIENT=yes
    volumes:
      - data1-1:/data1
      - data1-2:/data2
volumes:
  data1-1:
  data1-2:

networks:
  custom:
    driver: bridge

Answer 1

This works for me (4 nodes cluster on 4 servers)

version: '3.7'

services:
  minio1: # rename on different nodes
    restart: always
    image: 'bitnami/minio:latest'
    container_name: minio1 # rename on different nodes
    hostname: minio1 # rename on different nodes
    ports:
      - '9000:9000'
      - '9001:9001'
    environment:
      - MINIO_ROOT_USER=minioadmin
      - MINIO_ROOT_PASSWORD=minioadmin
      - MINIO_DISTRIBUTED_MODE_ENABLED=yes
      - MINIO_DISTRIBUTED_NODES=minio1,minio2,minio3,minio4
      - MINIO_SKIP_CLIENT=yes
    extra_hosts:
      - "minio1:10.10.10.1"  # 1st node
      - "minio2:10.10.10.2"  # 2nd node
      - "minio3:10.10.10.3"  # 3rd node
      - "minio4:10.10.10.4"  # 4th node
    volumes:
      - /opt/docker/minio/data:/data
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:9000/minio/health/live" ]
      interval: 30s
      timeout: 20s
      retries: 3

Answer 2

Are you sure the API Gateway is deployed? The error message seems to indicate no routes are available.

If it is deployed, but something is wrong with your backend, the error is typically like so:

{"message":"Internal Server Error"}