Add and start/stop containers with docker-compose dynamically

Question

I have a docker-compose.yml file with some services, like web server, database and nginx. However, beside these services that are always up, I need a number of instances of some other service which I want to dynamically create then start and stop.

For example, I have a small service that listens on some port and returns some value back when requested. I need 300 instances of this service running as containers which can then start and stop.

The number is not fixed and I want to be able to have 0-300 instances to be up on demand via docker tools/stack.

How can I do this in a better way? Is swarm something that can help here?

Answer 1

Yes, that is something one can achieve with an orchestrator.

Nowadays, Kubernetes is the trend. However, it is complicated to set up and the learning curve is steep compared to Docker Swarm. If you're already familiar with Docker, using Docker Swarm is straightforward.

Suppose your docker-compose.yml looks like this:

services:
  webserver:
    ...
  database:
    ...
  nginx:
    ...
  small-service:
    deploy:
      replicas: 6
    volumes:
      - "/path/for/task/{{.Task.Slot}}:/data"

You have your webserver, database, nginx, and your small service. Note that you can decide how many replicas (or task, as it's called in Docker Swarm) of a service you want to have in the docker-compose.yml file. More options under deploy can be found here.

Another interesting feature, as you asked for, is mounting different volumes for different tasks. By using .Task.Slot in the path, each task will mount their data to different locations in the host machine. Please check this documentation to see which other options are available besides .Task.Slot.

To deploy this using Docker Swarm, simply run this command:

docker stack deploy --compose-file docker-compose.yml <your-stack-name>

Then, besides other services, you will have 6 replicas of your small-service running.

At some point, you might want to change the number of replicas to 100. To do that, use the scale function:

docker service scale small-service=100

Or if you want to stop that service, rm is the way to go.

docker service rm small-service