How to stop an unstoppable zombie job on Jenkins without restarting the server

Question

Our Jenkins server has a job that has been running for three days  but is not doing anything  Clicking the little X in the corner does nothing  and the console output log doesn t show anything either  I ve checked on our build servers and the job doesn t actually seem to be running at all    Is there a way to tell jenkins that the job is  done   by editing some file or lock or something  Since we have a lot of jobs we don t really want to restart the server

User · Answer

Here is how I fixed this issue in version 2.100 with Blue Ocean

The only plugins I have installed are for bitbucket.
I only have a single node.

ssh into my Jenkins box
cd ~/.jenkins (where I keep jenkins)
cd job/<job_name>/branches/<problem_branch_name>/builds
rm -rf <build_number>

After this, you can optionally change the number in nextBuildNumber (I did this)

Finally, I restarted jenkins (brew services restart jenkins) This step will obviously be different depending how you manage and install Jenkins.

User · Answer

Recently I came across a node agent which had one executor occupied for days by a build  X  of a pipeline job  although that jobs page claimed build  X  did not exist anymore  discarded after 10 subsequent builds      as configured in the pipeline job   Verified that on disk  build  X  was really gone   The solution  it was the agent node which wrongly reported that the occupied executor was busy running build  X   Interrupting that executor s thread has immediately released it   def executor   Jenkins instance getNode  NODENAME   computer executors find       it isBusy    amp  amp  it name contains  JOBNAME      println executor  name if  executor  isBusy    executor interrupt     Other answers considered    The answer from  cheffe  did not work  see next point  and update below   The answers with Thread getAllStackTraces    no matching thread  The answer from  levente-holl   and all answers with getBuildByNumber    did not apply as the build wasn t really there anymore  The answer from  austinfromboston  that came close to my needs  but it would also have nuked any other builds running at the moment    Update  I experienced again a similar situation  where a Executor was occupied for days by a  still existing  finished pipeline build  This code snippet was the only working solution

User · Answer

I guess it is too late to answer but my help some people    Install the monitoring plugin   http   wiki jenkins-ci org display JENKINS Monitoring  Go to jenkinsUrl monitoring nodes Go to the Threads section at the bottom Click on the details button on the left of the master Sort by User time  ms  Then look at the name of the thread  you will have the name and number of the build Kill it   I don t have enough reputation to post images sorry   Hope it can help

User · Answer

I had also the same problem and fix it via Jenkins Console   Go to  Manage Jenkins     Script Console  and run a script    Jenkins  instance getItemByFullName  JobName            getBuildByNumber JobNumber           finish hudson model Result ABORTED  new java io IOException  Aborting build        You ll have just specify your JobName and JobNumber

User · Answer

Have had the same problem happen to me twice now  the only fix sofa has been to restart the tomcat server and restart the build

User · Answer

You can just copy the job and delete the old one  If it doesn t matter that you lost the old build logs

User · Answer

I had same issue at the last half hour     Was not able to delete a zombie build running in my multi-branch pipeline  Even Server restarts by UI or even from commandline via sudo service jenkins restart did block the execution    The build was not stoppable    It always reapeared      Used Version  Jenkins ver 2 150 2   I was very annoyed  but    when looking into the log of the build I found something intersting at the end of the log     The red marked parts are the  frustrating parts     As you can see I always wanted to Abort the build from UI but it did not work     But there is a hyperlink with text Click here to forcibly terminate running steps    first green one  Now I pressed the link     After the link execution a message about Still paused appeared with another Link Click here to forcibily kill entire build  second green one  After pressing this link also the build finally was hard killed     So this seems to work without any special plugins  except the multibranch-pipeline build plugin itself

User · Answer

Once I encounterred a build which could not be stopped by the  Script Console   Finally I solved the problem with these steps   ssh onto the jenkins server cd to  jenkins jobs  lt job-name gt  builds  rm -rf  lt build-number gt  restart jenkins

User · Answer

None of these solutions worked for me  I had to reboot the machine the server was installed on  The unkillable job is now gone

User · Answer

Enter the blue-ocean UI  Try to stop the job from there

User · Answer

Using the Script console at https   my-jenkins script  import hudson model Job import org jenkinsci plugins workflow job WorkflowRun  Collection lt Job gt  jobs   Jenkins instance getItem  My-Folder   getAllJobs   for  int i   0  i  lt  jobs size    i        def job   jobs i    for  int j   0  j  lt  job builds size    j          WorkflowRun build   job builds j      if  build isBuilding            println  Stopping  job   build number          build setResult Result FAILURE

User · Answer

Build-timeout Plugin can come handy for such cases  It will kill the job automatically if it takes too long

User · Answer

If you have an unstoppable Pipeline job  try the following    Abort the job by clicking the red X next to the build progress bar Click on  Pause resume  on the build to pause Click on  Pause resume  again to resume the build     Jenkins will realize that the job should be terminated and stops the build

User · Answer

Go to  Manage Jenkins     Script Console  to run a script on your server to interrupt the hanging thread    You can get all the live threads with Thread getAllStackTraces   and interrupt the one that s hanging   Thread getAllStackTraces   keySet   each       t - gt  if  t getName     YOUR THREAD NAME        t interrupt           UPDATE   The above solution using threads may not work on more recent Jenkins versions  To interrupt frozen pipelines refer to this solution  by alexandru-bantiuc  instead and run   Jenkins instance getItemByFullName  JobName                    getBuildByNumber JobNumber                   finish                          hudson model Result ABORTED                          new java io IOException  Aborting build

User · Answer

A utility I wrote called jkillthread can be used to stop any thread in any Java process  so long as you can log in to the machine running the service under the same account

User · Answer

Without having to use the script console or additional plugins  you can simply abort a build by entering  stop   term  or  kill after the build URL in your browser  Quoting verbatim from the above link   Pipeline jobs can by stopped by sending an HTTP POST request to URL endpoints of a build    lt BUILD ID URL gt  stop - aborts a Pipeline   lt BUILD ID URL gt  term - forcibly terminates a build  should only be used if stop does not work   lt BUILD ID URL gt  kill - hard kill a pipeline  This is the most destructive way to stop a pipeline and should only be used as a last resort

User · Answer

Had this same issue but there was not stack thread  We deleted the job by using this snippet in the Jenkins Console  Replace jobname and buil dnumber with yours   def jobname    Main FolderName BuildDefinition  def buildnum   6 Jenkins instance getItemByFullName jobname  getBuildByNumber buildnum  delete

User · Answer

This works for me everytime   Thread getAllStackTraces   keySet   each     if  it name contains  YOUR JOBNAME          println  Stopping  it name    it stop       Thanks to funql org

User · Answer

I had many zombi-jobs  so I used the following script   for int x   1000  x  lt  1813  x   x   1        Jenkins  instance getItemByFullName  JOBNAME BRANCH        getBuildByNumber x       finish hudson model Result ABORTED  new java io IOException  Aborting build

User · Answer

VERY SIMPLE SOLUTION The reason I was seeing this issue was incorrect http link on the page instead of https that should stop the job  All you need to do is to edit onclick attribute in html page  by following  Open up a console log of the job  pipeline  that got hang Click whatever is available to kill the job  x icon   quot Click here to forcibly terminate running steps quot  etc  to get  quot Click here to forcibly kill entire build quot  link visible  it s NOT gonna be clickable at the moment  Open the browser s console  use any one of three for chrome  F12  ctrl   shift   i  menu- gt more tools- gt developer tools  Locate  quot Click here to forcibly kill entire build quot  link manually or using  quot select an element in the page quot  button of the console Double click on onclick attribute to edit its value Append s to http to have https Press enter to submit the changes Click  quot Click here to forcibly kill entire build quot  link  Use screenshot for reference

User · Answer

I ve looked at the Jenkins source and it appears that what I m trying to do is impossible  because stopping a job appears to be done via a Thread interrupt  I have no idea why the job is hanging though    Edit    Possible reasons for unstoppable jobs    if Jenkins is stuck in an infinite loop  it can never be aborted  if Jenkins is doing a network or file I O within the Java VM  such as lengthy file copy or SVN update   it cannot be aborted

User · Answer

In case you got a Multibranch Pipeline-job  and you are a Jenkins-admin   use in the Jenkins Script Console this script   Jenkins instance  getItemByFullName   lt JOB NAME gt     getBranch   lt BRANCH NAME gt     getBuildByNumber  lt BUILD NUMBER gt    finish hudson model Result ABORTED  new java io IOException  Aborting build       From https   issues jenkins-ci org browse JENKINS-43020  If you aren t sure what the full name  path  of the job is  you may use the following snippet to list the full name of all items     Jenkins instance getAllItems AbstractItem class  each       println it fullName         From https   support cloudbees com hc en-us articles 226941767-Groovy-to-list-all-jobs

User · Answer

I usually use jenkins-cli in such cases  You can download the jar from a page http   your-jenkins-host PORT cli   Then run  java -jar jenkins-cli jar delete-builds name of job to delete hanging job number   Auxiliary info   You may also pass a range of builds like 350 400  General help available by running  java -jar jenkins-cli jar help   Context command help for delete-builds by  java -jar jenkins-cli jar delete-builds

User · Answer

The first proposed solution is pretty close  If you use stop   instead of interrupt   it even kills runaway threads  that run endlessly in a groovy system script  This will kill any build  that runs for a job  Here is the code   Thread getAllStackTraces   keySet   each         if  it name contains  YOUR JOBNAME              println  Stopping  it name        it stop

User · Answer

Alexandru Bantiuc s answer worked well for me to stop the build  but my executors were still showing up as busy   I was able clear the busy executor status using the following  server name pattern    your-servers- 1-5   jenkins model Jenkins instance getComputers   each   computer - gt    if  computer getName   find server name pattern         println computer getName       execList   computer getExecutors             for  exec in execList           busyState   exec isBusy       busy      idle        println  --    exec getDisplayName     busyState       if  exec isBusy              exec interrupt

User · Answer

I use the Monitoring Plugin for this task  After the installation of the plugin   Go to Manage Jenkins   Monitoring of Hudson Jenkins master Expand the Details of Threads  the small blue link on the right side Search for the Job Name that is hung  The Thread s name will start like this  Executor  2 for master   executing  lt your-job-name gt    lt build-number gt  Click the red  round button on the very right in the table of the line your desired job has

User · Answer

The top answer almost worked for me  but I had one major problem  I had a very large number   100  of zombie jobs due to a particularly poorly-timed Jenkins restart  so manually finding the job name and build number of each and every zombie job and then manually killing them was infeasible   Here s how I automatically found and killed the zombie jobs   Jenkins instance getItemByFullName multibranchPipelineProjectName  getItems   each   repository- gt    repository getItems   each   branch- gt      branch builds each   build- gt        if  build getResult   equals null             build doKill                         This script loops over all builds of all jobs and uses getResult   equals null  to determine whether or not the job has finished   A build that s in the queue but not yet started will not be iterated over  since that build won t be in job builds   and a build that s finished already will return something other than null for build getResult     A legitimately running job will also have a build result of null  so make sure you have no running jobs that you don t want to kill before running this   The multiple nested loops are mainly necessary to discover every branch PR for every repository in a Multibranch Pipeline project  if you re not using Multibranch Pipelines you can just loop over all your jobs directly with something like Jenkins instance getItems   each

[jenkins] How to stop an unstoppable zombie job on Jenkins without restarting the server?

Examples related to jenkins