How to run dynamic stages in paralell on jenkins with a separate kubernetes agent for each stage

Question

I tried combining things I have found on the syntax but this is as close as I can get. It creates multiple stages but says they have no steps.

I can get it to run a bunch of parallel steps on the same agent if I move the agent syntax down to where the "test" stage is defined but I want to spin up separate pods for each one so I can actually use the kubernetes cluster effectively and do my work parallel.

attached is an example Jenkinsfile for reference

def parallelStagesMap

def generateStage(job) {
    return {
        stage ("$job.key") {
            agent {
                kubernetes {
                    cloud 'kubernetes'
                    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""
                }
            }
            steps {
                sh """
                    do some important stuff
                """
            }
        }
    }
}

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
                script {
                    def map = [
                        "name" : "aparam",
                        "name2" : "aparam2"
                    ]
                    parallelStagesMap = map.collectEntries {
                        ["${it.key}" : generateStage(it)]
                    }
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    parallel parallelStagesMap 
                }
            }
        }
        stage('Release') {
            agent etc
            steps {
                etc
            }
        }
    }
}

Answer 1

As explained in this answer:

Dynamic parallel stages could be created only by using Scripted Pipelines. The API built-it Declarative Pipeline is not available (like agent).

So, you can't run dynamic stages in parallel on different agents.

To achieve what you want to do, a solution would be to trigger another pipeline that run on a new kube pod and wait for its completion before next steps.

Here is the Jenkinsfiles for more understanding:

• Main job Jenkinsfile:

def parallelJobsMap

def triggerJob(item) {
     return {
        build job: 'myChildJob', parameters: [string(name: 'MY_PARAM', value: "${item.value}")], wait: true
     }
}

pipeline {
    agent none
    stages {
        stage('Create List of Stages to run in Parallel') {
            steps {
               script {
                    def map = [
                        "name" : "aparam",
                        "name2" : "aparam2"
                     ]
                     parallelJobsMap = map.collectEntries {
                         ["${it.key}" : triggerJob(it)]
                     }
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    parallel parallelJobsMap
                }
            }
        }
        stage('Release') {
            agent any
            steps {
                 echo "Release stuff"
            }
        }
    }
}

• Child job Jenkinsfile:

pipeline {
    agent none
    parameters {
      string(
        name: 'MY_PARAM',
        description: 'My beautiful parameter',
        defaultValue: 'A default value',
        trim: true
      )
    }
    stages {
        stage ("Job") {
            agent {
                kubernetes {
                    cloud 'kubernetes'
                    yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: name
    image: image
    command:
    - sleep
    args:
    - infinity
"""
                }
            }
            steps {
                echo "Do some important stuff with the parameter " + params.MY_PARAM
            }
        }
    }
}