[git] Dynamically Fill Jenkins Choice Parameter With Git Branches In a Specified Repo

I have a parameterized Jenkins job which requires the input of a specific Git branch in a specific Git repo. Currently this parameter is a string parameter.

Is there any way to make this parameter a choice parameter and dynamically fill the drop down list with the Git branches? I don't want to require someone to maintain this choice parameter by manually configuring the drop down every time a new branch is created.

This question is related to git parameters branch jenkins choice

The answer is


Extended Choice Parameter plugin will allow you to read the choices from a file.

Of course, now you have another problem: how to make sure the file is up-to-date (that can be done with a post-commit hook) and propagated to all the users (that can be done by placing it on a shared file server). But there may be better solutions.


I am facing a similar problem here. Our users are migrating their jobs from freestyle to pipeline. They do not want Jenkinsfile stored in their repos(historical reason) and still want to use "Git Parameter" plugin

So we have to use use "Pipeline script" and develop a different plugin which works like "Git Parameter".

This new plugin does not integrate with SCM setting in the project. The plugin is at https://plugins.jenkins.io/list-git-branches-parameter

Hope it helps you as well


Its quite simple using the "Git Parameter Plug-in".

Add Name like "SELECT_BRANCH" ## Make sure for this variable as this would be used later. Then Parameter Type : Branch

Then reach out to SCM : Select : Git and branch specifier : ${SELECT_BRANCH}

To verify, execute below in shell in jenkins:

echo ${SELECT_BRANCH}

env.enter image description here

enter image description here


You may try this, This list dynamic branch names in dropdown w.r.t inputted Git Repo.

Jenkins Plugins required:

  1. Validating String Parameter
  2. Active Choices

OPTION 1: Jenkins File:

    properties([
        [$class: 'JobRestrictionProperty'], parameters([validatingString(defaultValue: 'https://github.com/kubernetes/kubernetes.git', description: 'Input Git Repo (https) Url', failedValidationMessage: 'Invalid Git Url. Retry again', name: 'GIT_REPO', regex: 'https://.*'), [$class: 'CascadeChoiceParameter', choiceType: 'PT_SINGLE_SELECT', description: 'Select Git Branch Name', filterLength: 1, filterable: false, name: 'BRANCH_NAME', randomName: 'choice-parameter-8292706885056518', referencedParameters: 'GIT_REPO', script: [$class: 'GroovyScript', fallbackScript: [classpath: [], sandbox: false, script: 'return[\'Error - Unable to retrive Branch name\']'], script: [classpath: [], sandbox: false, script: ''
                            'def GIT_REPO_SRC = GIT_REPO.tokenize(\'/\')
                            GIT_REPO_FULL = GIT_REPO_SRC[-2] + \'/\' + GIT_REPO_SRC[-1]
                            def GET_LIST = ("git ls-remote --heads [email protected]:${GIT_REPO_FULL}").execute()
                            GET_LIST.waitFor()
                            BRANCH_LIST = GET_LIST.in.text.readLines().collect {
                                it.split()[1].replaceAll("refs/heads/", "").replaceAll("refs/tags/", "").replaceAll("\\\\^\\\\{\\\\}", "")
                            }
                            return BRANCH_LIST ''
                            ']]]]), throttleJobProperty(categories: [], limitOneJobWithMatchingParams: false, maxConcurrentPerNode: 0, maxConcurrentTotal: 0, paramsToUseForLimit: '
                            ', throttleEnabled: false, throttleOption: '
                            project '), [$class: '
                            JobLocalConfiguration ', changeReasonComment: '
                            ']])
    try {
        node('master') {
            stage('Print Variables') {
                echo "Branch Name: ${BRANCH_NAME}"
            }
    }
    catch (e) {
        currentBuild.result = "FAILURE"
        print e.getMessage();
        print e.getStackTrace();
    }



OPTION 2: Jenkins UI

enter image description here

enter image description here



Sample Output:

enter image description here


Yes, I wrote a little groovy script which does the trick You should add a 'Dynamic Choice Parameter' to your job and customize the following groovy script to your needs :

#!/usr/bin/groovy

def gitURL = "git repo url"
def command = "git ls-remote --heads --tags ${gitURL}"

def proc = command.execute()
proc.waitFor()              

if ( proc.exitValue() != 0 ) {
   println "Error, ${proc.err.text}"
   System.exit(-1)
}

def text = proc.in.text
# put your version string match
def match = /<REGEX>/
def tags = []

text.eachMatch(match) { tags.push(it[1]) }
tags.unique()
tags.sort( { a, b ->
         def a1 = a.tokenize('._-')
         def b1 = b.tokenize('._-')
         try {
            for (i in 1..<[a1.size(), b1.size()].min()) { 
                 if (a1[i].toInteger() != b1[i].toInteger()) return a1[i].toInteger() <=> b1[i].toInteger()
            }
            return 1
         } catch (e) {
            return -1;
         }
} )
tags.reverse()

I my case the version string was in the following format X.X.X.X and could have user branches in the format X.X.X-username ,etc... So I had to write my own sort function. This was my first groovy script so if there are better ways of doing thing I would like to know.


I was able to achieve this result using the Jenkins Dynamic Parameter Plug-in. I used the Dynamic Choice Parameter option and, for the choices script, I used the following:

proc1 = ['/bin/bash', '-c', "/usr/bin/git ls-remote -h ssh://[email protected]/path/to/repo.git"].execute()
proc2 = ['/bin/bash', '-c', "awk '{print \$2}'"].execute()
proc3 = ['/bin/bash', '-c', "sed s%^refs/heads%origin%"].execute()

all = proc1 | proc2 | proc3
String result = all.text

String filename = "/tmp/branches.txt"
boolean success = new File(filename).write(result) 

def multiline = "cat /tmp/branches.txt".execute().text
def list = multiline.readLines()

For Me I use the input stage param:

  1. I start my pipeline by checking out the git project.
  2. I use a awk commade to generate a barnch.txt file with list of all branches
  3. In stage setps, i read the file and use it to generate a input choice params

When a user launch a pipeline, this one will be waiting him to choose on the list choice.

pipeline{
agent any

stages{

    stage('checkout scm') {
        steps {
            script{
                git credentialsId: '8bd8-419d-8af0-30960441fcd7', url: 'ssh://[email protected]:/usr/company/repositories/repo.git'
                sh 'git branch -r | awk \'{print $1}\' ORS=\'\\n\' >>branch.txt'
            }

        }
    }
     stage('get build Params User Input') {
        steps{
            script{

                liste = readFile 'branch.txt'
                echo "please click on the link here to chose the branch to build"
                env.BRANCH_SCOPE = input message: 'Please choose the branch to build ', ok: 'Validate!',
                        parameters: [choice(name: 'BRANCH_NAME', choices: "${liste}", description: 'Branch to build?')]


            }
        }
    } 
    stage("checkout the branch"){
        steps{
            echo "${env.BRANCH_SCOPE}"
            git  credentialsId: 'ea346a50-8bd8-419d-8af0-30960441fcd7', url: 'ssh://[email protected]/usr/company/repositories/repo.git'
            sh "git checkout -b build ${env.BRANCH_NAME}"
        }
    }
    stage(" exec maven build"){
        steps{
            withMaven(maven: 'M3', mavenSettingsConfig: 'mvn-setting-xml') {
               sh "mvn clean install "
            }
        }
    }
    stage("clean workwpace"){
        steps{
            cleanWs()
        }
    }
}

}

And then user will interact withim the build :

enter image description here

enter image description here


I tried a couple of answers mentioned in this link, but couldn't figure out how to tell Jenkins about the user-selected branch. As mentioned in my previous comment in above thread, I had left the branch selector field empty.

But, during further investigations, I found another way to do the same thing - https://wiki.jenkins-ci.org/display/JENKINS/Git+Parameter+Plugin I found this method was a lot simpler, and had less things to configure!

Here's what I configured -

  1. Installed the git parameter plugin
  2. Checked the 'This build is parameterized' and added a 'Git parameter'
  3. Added the following values: Git Parameter plugin config in the job

  4. Then in the git SCM section of the job I added the same value mentioned in the 'Name' section, as if it were an environment variable. (If you read the help for this git parameter plugin carefully, you will realize this) Branch Selector

After this I just ran the build, chose my branch(Jenkins checks out this branch before building) and it completed the build successfully, AND by choosing the branch that I had specified.


I'm aware to the fact that in the original question Jenkins pipeline was not mentioned, but if it is still applicable (using it), I find this solution easy to maintain and convenient.

This approach describe the settings required to compose a Jenkins pipeline that "polls" (list) dynamically all branches of a particular repository, which then lets the user run the pipeline with some specific branch when running a build of this job.

The assumptions here are:

  • The Jenkins server is 2.204.2 (hosted on Ubuntu 18.04)
  • The repository is hosted in a BitBucket.

First thing to do is to provide Jenkins credentials to connect (and "fetch") to the private repository in BitBucket. This can be done by creating an SSH key pair to "link" between the Jenkins (!!) user on the machine that hosts the Jenkins server and the (private) BitBucket repository.

  1. First thing is to create an SSH key to the Jenkins user (which is the user that runs the Jenkins server - it is most likely created by default upon the installation):

    guya@ubuntu_jenkins:~$ sudo su jenkins 
    [sudo] password for guya:
    jenkins@ubuntu_jenkins:/home/guya$ ssh-keygen
    

    The output should look similar to the following:

Generating public/private rsa key pair. Enter file in which to save the key
(/var/lib/jenkins/.ssh/id_rsa): Created directory '/var/lib/jenkins/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /var/lib/jenkins/.ssh/id_rsa. Your public key has been saved in /var/lib/jenkins/.ssh/id_rsa.pub. The key fingerprint is: SHA256:q6PfEthg+74QFwO+esLbOtKbwLG1dhtMLfxIVSN8fQY jenkins@ubuntu_jenkins The key's randomart image is: +---[RSA 2048]----+ | . .. o.E. | | . . .o... o | | . o.. o | | +.oo | | . ooX..S | |..+.Bo* . | |.++oo* o. | |..+*..*o | | .=+o==+. | +----[SHA256]-----+ jenkins@ubuntu_jenkins:/home/guya$

  1. Now the content of this SSH key needs to be set in the BitBucket repository as follows:
  • Create (add) an SSH key in the BitBucket repository by going to: Settings --> Access keys --> Add key.
  • Give the key Read permissions and copy the content of the PUBLIC key to the "body" of the key. The content of the key can be displayed by running: cat /var/lib/jenkins/.ssh/id_rsa.pub
  1. After that the SSH key was set in the BitBucket repository, we need to "tell" Jenkins to actually USE it when it tries to fetch (read in this case) the content of the repository. NOTE that by letting Jenkins know, actually means let user jenkins this "privilege".

This can be done by adding a new  SSH User name with private key to the Jenkins --> Credentials --> System --> Global Credentials --> Add credentials.

  • In the ID section put any descriptive name to the key.
  • In the Username section put the user name of the Jenkins server which is jenkins.
  • Tick the Private key section and paste the content of the PRIVATE key that was generated earlier by copy-paste the content of: ~/.ssh/id_rsa. This is the private key which start with the string:-----BEGIN RSA PRIVATE KEY----- and ends with the string: -----END RSA PRIVATE KEY-----. Note that this entire "block" should be copied-paste into the above section.
  1. Install the Git Parameter plugin that can be found in its official page here

  2. The very minimum pipeline that is required to list (dynamically) all the branches of a given repository is as follows:

    pipeline 
    { 
        agent any parameters
        { 
            gitParameter branchFilter: 'origin/(.*)', defaultValue: 'master', name: 'BRANCH', type: 'PT_BRANCH' 
        }
        stages 
        { 
           stage("list all branches") 
           { 
               steps 
               { 
                    git branch: "${params.BRANCH}", credentialsId: "SSH_user_name_with_private_key", url: "ssh://[email protected]:port/myRepository.git" 
               } 
          } 
       } 
    }
    

NOTES:

  • The defaultValue is set to master so that if no branches exist - it will be displayed in the "drop list" of the pipeline.
  • The credentialsId has the name of the credentials configured earlier.
  • In this case I used the SSH URL of the repository in the url parameter.
  • This answer assumes (and configured) that the git server is BitBucket. I assume that all the "administrative" settings done in the initial steps, have their equivalent settings in GitHub.

We can eliminate the unnecessary file read/write by using text. My complete solution is the following:

proc1 = ['/bin/bash', '-c', 
  "/usr/bin/git ls-remote --heads ssh://repo_url.git"].execute()
proc2 = ['/bin/bash', '-c', 
  "/usr/bin/awk ' { gsub(/refs\\/heads\\//, \"\"); print \$2 }' "].execute()
all = proc1 | proc2

choices = all.text
return choices.split().toList();

You can accomplish the same using the extended choice parameter plugin before mentioned by malenkiy_scot and a simple php script as follows(assuming you have somewhere a server to deploy php scripts that you can hit from the Jenkins machine)

<?php
chdir('/path/to/repo');
exec('git branch -r', $output);
print('branches='.str_replace('  origin/','',implode(',', $output)));
?>

or

<?php
exec('git ls-remote -h http://user:[email protected]', $output);
print('branches='.preg_replace('/[a-z0-9]*\trefs\/heads\//','',implode(',', $output)));
?>

With the first option you would need to clone the repo. With the second one you don't, but in both cases you need git installed in the server hosting your php script. Whit any of this options it gets fully dynamic, you don't need to build a list file. Simply put the URL to your script in the extended choice parameter "property file" field.


I have a new response for this case: the easy way to solve this is having you jenkinsFile from source code.

Then you chose: this job have a git parameter

enter image description here

And and the Pipeline setup, unchecked the "Lightweight checkout" checkbox, this will perform a really git checkout on job workspace.

enter image description here

After, the parameter will be autopopulate by your git branch


expanding on @malenkiy_scot's answer. I created a new jenkins job to build up the file that is used by Extended Choice Plugin.

you can do the following (I did it as execute shell steps in jenkins, but you could do it in a script):

git ls-remote [email protected]:my/repo.git |grep refs/heads/* >tmp.txt
sed -e 's/.*refs\/heads\///' tmp.txt > tmp2.txt
tr '\n' ',' < tmp2.txt > tmp3.txt
sed '1i\branches=' tmp3.txt > tmp4.txt
tr -d '\n'  < tmp4.txt > branches.txt

I then use the Artifact deployer plugin to push that file to a shared location, which is in a web url, then just use 'http://localhost/branches.txt' in the Extended Choice plugin as the url. works like a charm.


The following groovy script would be useful, if your job does not use "Source Code Management" directly (likewise "Git Parameter Plugin"), but still have access to a local (cloned) git repository:

import jenkins.model.Jenkins

def envVars = Jenkins.instance.getNodeProperties()[0].getEnvVars() 
def GIT_PROJECT_PATH = envVars.get('GIT_PROJECT_PATH') 
def gettags = "git ls-remote -t --heads origin".execute(null, new File(GIT_PROJECT_PATH))

return gettags.text.readLines()
         .collect { it.split()[1].replaceAll('\\^\\{\\}', '').replaceAll('refs/\\w+/', '')  }
         .unique()

See full explanation here: https://stackoverflow.com/a/37810768/658497


Examples related to git

Does the target directory for a git clone have to match the repo name? Git fatal: protocol 'https' is not supported Git is not working after macOS Update (xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools) git clone: Authentication failed for <URL> destination path already exists and is not an empty directory SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 GitLab remote: HTTP Basic: Access denied and fatal Authentication How can I switch to another branch in git? VS 2017 Git Local Commit DB.lock error on every commit How to remove an unpushed outgoing commit in Visual Studio?

Examples related to parameters

Stored procedure with default parameters AngularJS ui router passing data between states without URL C#: HttpClient with POST parameters HTTP Request in Swift with POST method In Swift how to call method with parameters on GCD main thread? How to pass parameters to maven build using pom.xml? Default Values to Stored Procedure in Oracle How do you run a .exe with parameters using vba's shell()? How to set table name in dynamic SQL query? How to pass parameters or arguments into a gradle task

Examples related to branch

Get git branch name in Jenkins Pipeline/Jenkinsfile Why do I have to "git push --set-upstream origin <branch>"? Your configuration specifies to merge with the <branch name> from the remote, but no such ref was fetched.? When does Git refresh the list of remote branches? Fix GitLab error: "you are not allowed to push code to protected branches on this project"? Git push: "fatal 'origin' does not appear to be a git repository - fatal Could not read from remote repository." Git: Merge a Remote branch locally git pull from master into the development branch Depend on a branch or tag using a git URL in a package.json? How can I copy the content of a branch to a new local branch?

Examples related to jenkins

Maven dependencies are failing with a 501 error Jenkins pipeline how to change to another folder Docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock groovy.lang.MissingPropertyException: No such property: jenkins for class: groovy.lang.Binding How to solve npm install throwing fsevents warning on non-MAC OS? Run bash command on jenkins pipeline Try-catch block in Jenkins pipeline script How to print a Groovy variable in Jenkins? Jenkins pipeline if else not working Error "The input device is not a TTY"

Examples related to choice

Dynamically Fill Jenkins Choice Parameter With Git Branches In a Specified Repo