Pods stuck in Terminating status

Question

I tried to delete a ReplicationController with 12 pods and I could see that some of the pods are stuck in Terminating status    My Kubernetes cluster consists of one control plane node and three worker nodes installed on Ubuntu virtual machines    What could be the reason for this issue   NAME        READY     STATUS        RESTARTS   AGE pod-186o2   1 1       Terminating   0          2h pod-4b6qc   1 1       Terminating   0          2h pod-8xl86   1 1       Terminating   0          1h pod-d6htc   1 1       Terminating   0          1h pod-vlzov   1 1       Terminating   0          1h

User · Answer

Delete the finalizers block from resource  pod deployment ds etc     yaml    finalizers        foregroundDeletion

User · Answer

you can use awk   kubectl get pods --all-namespaces   awk   if   4   quot Terminating quot   print  quot oc delete pod  quot   2  quot  -n  quot   1  quot  --force --grace-period 0  quot       sh

User · Answer

Force delete the pod   kubectl delete pod --grace-period 0 --force --namespace  lt NAMESPACE gt   lt PODNAME gt    The --force flag is mandatory

User · Answer

I found this command more straightforward   for p in   kubectl get pods   grep Terminating   awk   print  1     do kubectl delete pod  p --grace-period 0 --force done   It will delete all pods in Terminating status in default namespace

User · Answer

please try below command   kubectl patch pod -p    quot metadata quot    quot finalizers quot  null

User · Answer

Before doing a force deletion i would first do some checks  1- node state  get the node name where your node is running  you can see this with the following command   quot kubectl -n YOUR NAMESPACE describe pod YOUR PODNAME quot   Under the  quot Node quot  label you will see the node name  With that you can do  kubectl describe node NODE NAME  Check the  quot conditions quot  field if you see anything strange  If this is fine then you can move to the step  redo   quot kubectl -n YOUR NAMESPACE describe pod YOUR PODNAME quot   Check the reason why it is hanging  you can find this under the  quot Events quot  section  I say this because you might need to take preliminary actions before force deleting the pod  force deleting the pod only deletes the pod itself not the underlying resource  a stuck docker container for example

User · Answer

In my case the --force option didn t quite work  I could still see the pod   It was stuck in Terminating Unknown mode  So after running   kubectl delete pods  lt pod gt  -n redis --grace-period 0 --force   I ran  kubectl patch pod  lt pod gt  -p    metadata    finalizers  null

User · Answer

If --grace-period 0 is not working then you can do   kubectl delete pods  lt pod gt  --grace-period 0 --force

User · Answer

I d not recommend force deleting pods unless container already exited   Verify kubelet logs to see what is causing the issue  quot journalctl -u kubelet quot  Verify docker logs  journalctl -u docker service Check if pod s volume mount points still exist and if anyone holds lock on it  Verify if host is out of memory or disk

User · Answer

I stumbled upon this recently to free up resource in my cluster  here is the command to delete them all    kubectl get pods --all-namespaces   grep Terminating   while read line  do  pod name   echo  line   awk   print  2     name space   echo  line   awk    print  1      kubectl delete pods  pod name -n  name space --grace-period 0 --force   done   hope this help someone who read this

User · Answer

You can use following command to delete the POD forcefully   kubectl delete pod  lt PODNAME gt  --grace-period 0 --force --namespace  lt NAMESPACE gt

User · Answer

Practical answer -- you can always delete a terminating pod by running   kubectl delete pod NAME --grace-period 0   Historical answer -- There was an issue in version 1 1 where sometimes pods get stranded in the Terminating state if their nodes are uncleanly removed from the cluster

User · Answer

The original question is  What could be the reason for this issue   and the answer is discussed at https   github com kubernetes kubernetes issues 51835  amp  https   github com kubernetes kubernetes issues 65569  amp   see https   www bountysource com issues 33241128-unable-to-remove-a-stopped-container-device-or-resource-busy  Its caused by docker mount leaking into some other namespace    You can logon to pod host to investigate   minikube ssh docker container ps   grep  lt id gt  docker container stop  lt id gt

User · Answer

I stumbled upon this recently when removing rook ceph namespace - it got stuck in Terminating state   The only thing that helped was removing kubernetes finalizer by directly calling k8s api with curl as suggested here    kubectl get namespace rook-ceph -o json  gt  tmp json delete kubernetes finalizer in tmp json  leave empty array  finalizers       run kubectl proxy in another terminal for auth purposes and run following curl request to returned port curl -k -H  Content-Type  application json  -X PUT --data-binary  tmp json 127 0 0 1 8001 k8s clusters c-mzplp api v1 namespaces rook-ceph finalize namespace is gone   Detailed rook ceph teardown here

[kubernetes] Pods stuck in Terminating status

Examples related to kubernetes