Parallel checkout in declarative Jenkins pipeline on multiple nodes

wl2776

I'm developing a declarative Jenkins pipeline for CI builds, triggered from Gitlab. What I have now:


// variable definitions

pipeline {
    agent none

    parameters {
            string(defaultValue: 'develop',
                description: 'Commit ID or branch name to build', 
                name: 'branch', 
                trim: false)
    }

    stages {
        stage('Checkout') {
            parallel {
                stage ('Windows') {
                    agent {
                        label 'build' && 'windows'
                    }

                    steps {
                        script {

                            def checkout_ext = [[$class: 'CleanCheckout'],
                                                [$class: 'CleanBeforeCheckout']] // calls git clean -fdx and git reset --hard

                            if (env.gitlabActionType == "MERGE"){   
                                  checkout_ext.add([$class: 'PreBuildMerge', 
                                                    options: [ mergeRemote: "origin",
                                                              mergeTarget: "${env.gitlabTargetBranch}"]])
                            }
                        }

                        checkout([
                                $class: 'GitSCM', 
                                branches: [[name: "${params.branch}"]],
                                userRemoteConfigs: [[ url: "${git_url}",                                          credentialsId: "${git_credentials_id}" ]],
                                extensions:   checkout_ext 
                        ])
                    }
                }

                stage('Linux') {
                    agent {
                        label 'build' && 'linux'
                    }
                    steps {
                        script {

                            def checkout_ext = [[$class: 'CleanCheckout'], 
                                                [$class: 'CleanBeforeCheckout']] // calls git clean -fdx and git reset --hard

                            if (env.gitlabActionType == "MERGE"){   
                               checkout_ext.add([$class: 'PreBuildMerge', 
                                                 options: [ mergeRemote: "origin",
                                                 mergeTarget: "${env.gitlabTargetBranch}"]])
                            }
                        }

                        checkout([
                                $class: 'GitSCM', 
                                branches: [[name: "${params.branch}"]],
                                userRemoteConfigs: [[ url: "${git_url}", credentialsId: "${git_credentials_id}"]],
                                extensions:   checkout_ext 
                        ])
                    }
                }
            }
        }
    }
}

Checkout stage is somewhat complex. If gitlabActionType is MERGE, then first try to merge into a target branch, to make sure that merge request does not break anything in it.

This code is the same for both OSes. I'd like to avoid code duplication, but cannot figure out correct syntax for that.

I have tried moving definition of checkout steps to the global variable, but have got syntax errors.


def checkout_step = {
    script {
   ...
    }
    checkout (... )
}

pipeline {
...
   stages {
        stage('Checkout') {
            parallel {
                stage ('Windows') {
                    agent {
                        label 'build' && 'windows'
                    }

                    steps {
                        checkout_step
                    }
                }
                stage ('Linux') {
                    agent {
                        label 'build' && 'linux'
                    }

                    steps {
                        checkout_step
                    }
                }
            }
        }
    }
}

If add steps, there's also an error:


def checkout_step = steps {
    script {
   ...
    }
    checkout (... )
}

pipeline {
...
   stages {
        stage('Checkout') {
            parallel {
                stage ('Windows') {
                    agent {
                        label 'build' && 'windows'
                    }

                    checkout_step

                }
                stage ('Linux') {
                    agent {
                        label 'build' && 'linux'
                    }

                    checkout_step

                }
            }
        }
    }
}
wl2776

Have found solution here


git_url = "[email protected]:group/repo.git"
git_credentials_id = 'aaaaaaa-bbbb-cccc-dddd-eefefefefef'


def checkout_tasks(os_labels) {
    tasks = [:]

    for (int i = 0; i < os_labels.size(); i++) {
        def os = os_labels[i]
        tasks["${os}"] = {
            node("build && ${os}"){

                def checkout_ext = [[$class: 'CleanCheckout'], [$class: 'CleanBeforeCheckout']] // calls git clean -fdx and git reset --hard

                if (env.gitlabActionType == "MERGE"){   
                    checkout_ext.add([
                            $class: 'PreBuildMerge', 
                            options: [ 
                            mergeRemote: "origin", 
                            mergeTarget: "${env.gitlabTargetBranch}" 
                            ]
                    ])
                         /* using this extension requires .gitconfig with section [user for Jenkins]
                            Example
                            [user]
                            email = jenkins@builder
                            name = Jenkins

                          */
                }

                checkout([
                        $class: 'GitSCM', 
                        branches: [[name: "${params.branch}"]],
                        userRemoteConfigs: [[
                            url: "${git_url}", 
                            credentialsId: "${git_credentials_id}"
                        ]],
                        extensions:  checkout_ext 
                ])

            }
        }
    }
    return tasks
}


pipeline {
    agent none

    parameters {
        string(defaultValue: 'develop',
               description: 'Commit ID or branch name to build', 
               name: 'branch', 
               trim: false)
    }

    stages {
        stage('Checkout') {
            steps {
                script {
                    def OSes = ["windows", "linux"]
                    parallel checkout_tasks(OSes)
                }
            }
        }
   }
}

It is important also to declare git_url and git_credentials_id without def, so that functions can read them.

More details in this question

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

Jenkins parallel declarative pipeline

Jenkins Declarative Pipeline parallel stages

Parallel items in Jenkins Declarative pipeline

parallel steps on different nodes on declarative jenkins pipeline cannot access variables defined in global scope

Dynamically defining parallel steps in declarative jenkins pipeline

Jenkins declarative pipeline parallel steps executors

How to lock multiple stages of declarative Jenkins pipeline?

Creating a sequential step in a jenkins declarative pipeline preceding a parallel stage

How to detect which parallel stage failed in a Jenkins declarative pipeline?

How to make parallel calls to the same function in stage in Jenkins Declarative Pipeline

Assigning variables in a parallel step using Declarative Pipeline steps in Jenkins

Parallel runs in declarative pipeline

Jenkins input on declarative pipeline

Jenkins Pipeline: is it possible to avoid multiple checkout?

Jenkins scripted pipeline or declarative pipeline

How to run multiple stages on the same node with declarative Jenkins pipeline?

Can I define multiple agent labels in a declarative Jenkins Pipeline?

How to select multiple JDK version in declarative pipeline Jenkins

running the same task on multiple agents in my Jenkins declarative pipeline

Dynamic Variable in Jenkins Declarative Pipeline

Using waitForQualityGate in a Jenkins declarative pipeline

Jenkins: Using XmlSlurper In Declarative Pipeline

Jenkins dynamic declarative pipeline parameters

Jenkins Declarative Pipeline Docker Registry

Referencing variable in declarative jenkins pipeline

Not serializable error for jenkins declarative pipeline

Jenkins declarative pipeline @tmp folders

Running non-matrix stages in parallel to a matrix in Jenkins declarative-pipeline

Checkout a tag in Jenkins pipeline