'How to properly use jenkinsfile "parallel" tag and docker-compose to test multiple python versions and not prematurely kill containers?
I have built a jenkinsfile that runs some tests on a repo I've got, but for multiple versions of python, and done in parallel using the "parallel" jenkinsfile tag.
This jenkinsfile makes use of a docker-compose file (corresponding to python 3.8) and then two docker-compose overrides (corresponding to python 3.9 and 3.10).
The problem I'm currently running into is that sometimes, one of the docker-compose down --remove-orphans calls seem to be trying to sometimes kill containers that are a part of one of the other parallel processes, i.e. I get a error while removing network: network <network name> id <id> has active endpoints error.
I'm wondering what is the proper way to deal with this issue. Thanks!
JENKINSFILE CODE:
pipeline {
agent none
stages {
stage('Test Across Python Versions:') {
parallel {
stage('Python 3.8: Checkout and Create Environment Variables') {
agent {
node {
label 'MyBuilder'
}
}
stages('Testing'){
stage('Python 3.8: Build and Test'){
steps {
sh 'docker-compose -p py3_8 build'
sh 'docker-compose -p py3_8 run --rm tests'
}
}
}
post{
always {
sh "docker-compose -p py3_8 down --remove-orphans"
junit '**/junit*.xml'
}
}
}
stage('Python 3.9: Checkout and Create Environment Variables') {
agent {
node {
label 'MyBuilder'
}
}
stages('Testing'){
stage('Python 3.9: Build and Test'){
steps {
sh 'docker-compose -p py3_9 -f docker-compose.yaml -f docker-compose.python3_9.yaml build'
sh 'docker-compose -p py3_9 -f docker-compose.yaml -f docker-compose.python3_9.yaml run --rm tests'
}
}
}
post{
always {
sh "docker-compose -p py3_9 -f docker-compose.yaml -f docker-compose.python3_9.yaml down --remove-orphans"
junit '**/junit*.xml'
}
}
}
stage('Python 3.10: Checkout and Create Environment Variables') {
agent {
node {
label 'MyBuilder'
}
}
stages('Testing'){
stage('Python 3.10: Build and Test'){
steps {
sh 'docker-compose -p py3_10 -f docker-compose.yaml -f docker-compose.python3_10.yaml build'
sh 'docker-compose -p py3_10 -f docker-compose.yaml -f docker-compose.python3_10.yaml run --rm tests'
}
}
}
post{
always {
sh "docker-compose -p py3_10 -f docker-compose.yaml -f docker-compose.python3_10.yaml down --remove-orphans"
junit '**/junit*.xml'
}
}
}
}
}
}
}
DOCKER-COMPOSE CODE:
version: "3.4"
x-tests: &blah
build:
context: .
dockerfile: Dockerfile
image: my_repo:test-venv-3_8
working_dir: /home
volumes:
- .:/home/
services:
tests:
<<: *blah
container_name: tests
entrypoint: python3 -m pytest -c pytest.ini
DOCKER-COMPOSE OVERRIDE CODE:
version: "3.4"
x-tests:
build:
context: .
dockerfile: python3_9.Dockerfile
image: my_repo:test-venv-3_9
working_dir: /home
volumes:
- .:/home/
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
