'How to properly use jenkinsfile "parallel" tag and docker-compose to test multiple python versions and not prematurely kill containers?

I have built a jenkinsfile that runs some tests on a repo I've got, but for multiple versions of python, and done in parallel using the "parallel" jenkinsfile tag.

This jenkinsfile makes use of a docker-compose file (corresponding to python 3.8) and then two docker-compose overrides (corresponding to python 3.9 and 3.10).

The problem I'm currently running into is that sometimes, one of the docker-compose down --remove-orphans calls seem to be trying to sometimes kill containers that are a part of one of the other parallel processes, i.e. I get a error while removing network: network <network name> id <id> has active endpoints error.

I'm wondering what is the proper way to deal with this issue. Thanks!

JENKINSFILE CODE:

pipeline {
    agent none

    stages {
        stage('Test Across Python Versions:') {
            parallel {
                stage('Python 3.8: Checkout and Create Environment Variables') {
                    agent {
                        node {
                            label 'MyBuilder'
                        }
                    }
                    stages('Testing'){
                        stage('Python 3.8: Build and Test'){
                            steps {
                                sh 'docker-compose -p py3_8 build'
                                sh 'docker-compose -p py3_8 run --rm tests'
                            }
                        }
                    }
                    post{
                        always {
                            sh "docker-compose -p py3_8 down --remove-orphans"
                            junit '**/junit*.xml'
                        }
                    }
                }
                stage('Python 3.9: Checkout and Create Environment Variables') {
                    agent {
                        node {
                            label 'MyBuilder'
                        }
                    }
                    stages('Testing'){
                        stage('Python 3.9: Build and Test'){
                            steps {
                                sh 'docker-compose -p py3_9 -f docker-compose.yaml -f docker-compose.python3_9.yaml build'
                                sh 'docker-compose -p py3_9 -f docker-compose.yaml -f docker-compose.python3_9.yaml run --rm tests'
                            }
                        }
                    }
                    post{
                        always {
                            sh "docker-compose -p py3_9 -f docker-compose.yaml -f docker-compose.python3_9.yaml down --remove-orphans"
                            junit '**/junit*.xml'
                        }
                    }
                }
                stage('Python 3.10: Checkout and Create Environment Variables') {
                    agent {
                        node {
                            label 'MyBuilder'
                        }
                    }
                    stages('Testing'){
                        stage('Python 3.10: Build and Test'){
                            steps {
                                sh 'docker-compose -p py3_10 -f docker-compose.yaml -f docker-compose.python3_10.yaml build'
                                sh 'docker-compose -p py3_10 -f docker-compose.yaml -f docker-compose.python3_10.yaml run --rm tests'
                            }
                        }
                    }
                    post{
                        always {
                            sh "docker-compose -p py3_10 -f docker-compose.yaml -f docker-compose.python3_10.yaml down --remove-orphans"
                            junit '**/junit*.xml'
                        }
                    }
                }
            }
        }
    }
}

DOCKER-COMPOSE CODE:

version: "3.4"

x-tests: &blah
  build:
    context: .
    dockerfile: Dockerfile
  image: my_repo:test-venv-3_8
  working_dir: /home
  volumes:
    - .:/home/

services:
  tests:
    <<: *blah
    container_name: tests
    entrypoint: python3 -m pytest -c pytest.ini

DOCKER-COMPOSE OVERRIDE CODE:

version: "3.4"

x-tests:
  build:
    context: .
    dockerfile: python3_9.Dockerfile
  image: my_repo:test-venv-3_9
  working_dir: /home
  volumes:
    - .:/home/


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source