My kubernetes pods keep crashing with CrashLoopBackOff but I can t find any log

Question

This is what I keep getting    root centos-master     kubectl get pods NAME               READY     STATUS             RESTARTS   AGE nfs-server-h6nw8   1 1       Running            0          1h nfs-web-07rxz      0 1       CrashLoopBackOff   8          16m nfs-web-fdr9h      0 1       CrashLoopBackOff   8          16m   Below is output from  describe pods  kubectl describe pods  Events    FirstSeen LastSeen    Count   From                SubobjectPath       Type        Reason      Message   --------- --------    -----   ----                -------------       --------    ------      -------   16m       16m     1    default-scheduler                      Normal      Scheduled   Successfully assigned nfs-web-fdr9h to centos-minion-2   16m       16m     1    kubelet centos-minion-2    spec containers web     Normal      Created     Created container with docker id 495fcbb06836   16m       16m     1    kubelet centos-minion-2    spec containers web     Normal      Started     Started container with docker id 495fcbb06836   16m       16m     1    kubelet centos-minion-2    spec containers web     Normal      Started     Started container with docker id d56f34ae4e8f   16m       16m     1    kubelet centos-minion-2    spec containers web     Normal      Created     Created container with docker id d56f34ae4e8f   16m       16m     2    kubelet centos-minion-2                Warning     FailedSync  Error syncing pod  skipping  failed to  StartContainer  for  web  with CrashLoopBackOff   Back-off 10s restarting failed container web pod nfs-web-fdr9h default 461c937d-d870-11e6-98de-005056040cc2     I have two pods  nfs-web-07rxz  nfs-web-fdr9h  but if I do  kubectl logs nfs-web-07rxz  or with  -p  option I don t see any log in both pods    root centos-master     kubectl logs nfs-web-07rxz -p  root centos-master     kubectl logs nfs-web-07rxz   This is my replicationController yaml file  replicationController yaml file  apiVersion  v1 kind  ReplicationController metadata    name  nfs-web spec    replicas  2   selector      role  web-frontend   template      metadata        labels          role  web-frontend     spec        containers        - name  web         image  eso-cmbu-docker artifactory eng vmware com demo-container demo-version3 0         ports            - name  web             containerPort  80         securityContext            privileged  true   My Docker image was made from this simple docker file   FROM ubuntu RUN apt-get update RUN apt-get install -y nginx RUN apt-get install -y nfs-common   I am running my kubernetes cluster on CentOs-1611  kube version    root centos-master     kubectl version Client Version  version Info Major  1   Minor  3   GitVersion  v1 3 0   GitCommit  86dc49aa137175378ac7fba7751c3d3e7f18e5fc   GitTreeState  clean   BuildDate  2016-12-15T16 57 18Z   GoVersion  go1 6 3   Compiler  gc   Platform  linux amd64   Server Version  version Info Major  1   Minor  3   GitVersion  v1 3 0   GitCommit  86dc49aa137175378ac7fba7751c3d3e7f18e5fc   GitTreeState  clean   BuildDate  2016-12-15T16 57 18Z   GoVersion  go1 6 3   Compiler  gc   Platform  linux amd64     If I run the docker image by  docker run  I was able to run the image without any issue  only through kubernetes I got the crash    Can someone help me out  how can I debug without seeing any log

User · Answer

I had the need to keep a pod running for subsequent kubectl exec calls and as the comments above pointed out my pod was getting killed by my k8s cluster because it had completed running all its tasks  I managed to keep my pod running by simply kicking the pod with a command that would not stop automatically as in   kubectl run YOUR POD NAME -n YOUR NAMESPACE --image SOME PUBLIC IMAGE latest --command tailf  dev null

User · Answer

My pod kept crashing and I was unable to find the cause  Luckily there is a space where kubernetes saves all the events that occurred before my pod crashed     List Events sorted by timestamp   To see these events run the command   kubectl get events --sort-by  metadata creationTimestamp   make sure to add a --namespace mynamespace argument to the command if needed  The events shown in the output of the command showed my why my pod kept crashing

User · Answer

I had similar issue but got solved when I corrected my zookeeper yaml file which had the service name mismatch with file deployment s container names  It got resolved by making them same  apiVersion  v1 kind  Service metadata    name  zk1   namespace  nbd-mlbpoc-lab   labels      app  zk-1 spec    ports    - name  client     port  2181     protocol  TCP   - name  follower     port  2888     protocol  TCP   - name  leader     port  3888     protocol  TCP   selector      app  zk-1 --- kind  Deployment apiVersion  extensions v1beta1 metadata    name  zk-deployment   namespace  nbd-mlbpoc-lab spec    template      metadata        labels          app  zk-1     spec        containers        - name  zk1         image  digitalwonderland zookeeper         ports          - containerPort  2181         env          - name  ZOOKEEPER ID           value   quot 1 quot          - name  ZOOKEEPER SERVER 1           value  zk1

User · Answer

Try rerunning the pod and running  kubectl get pods --watch  to watch the status of the pod as it progresses  In my case  I would only see the end result   CrashLoopBackOff   but the docker container ran fine locally  So I watched the pods using the above command  and I saw the container briefly progress into an OOMKilled state  which meant to me that it required more memory

User · Answer

I observed the same issue  and added the command and args block in yaml file  I am copying sample of my yaml file for reference  apiVersion  v1     kind  Pod     metadata        labels          run  ubuntu       name  ubuntu       namespace  default     spec        containers        - image  gcr io ow hellokubernetes ubuntu         imagePullPolicy  Never         name  ubuntu         resources            requests              cpu  100m         command    quot  bin sh quot           args    quot -c quot    quot while true  do echo hello  sleep 10 done quot         dnsPolicy  ClusterFirst       enableServiceLinks  true

User · Answer

I solved this problem I increased memory resource    resources            limits              cpu  1             memory  1Gi           requests              cpu  100m         memory  250Mi

User · Answer

If you have an application that takes slower to bootstrap  it could be related to the initial values of the readiness liveness probes  I solved my problem by increasing the value of initialDelaySeconds to 120s as my SpringBoot application deals with a lot of initialization  The documentation does not mention the default 0  https   kubernetes io docs api-reference v1 9  probe-v1-core   service    livenessProbe      httpGet        path   health local       scheme  HTTP       port  8888     initialDelaySeconds  120     periodSeconds  5     timeoutSeconds  5     failureThreshold  10   readinessProbe      httpGet        path   admin health       scheme  HTTP       port  8642     initialDelaySeconds  150     periodSeconds  5     timeoutSeconds  5     failureThreshold  10   A very good explanation about those values is given by What is the default value of initialDelaySeconds       The health or readiness check algorithm works like          wait for initialDelaySeconds   perform check and wait timeoutSeconds for a timeout   if the number of continued successes is greater than successThreshold return success   if the number of continued failures is greater than failureThreshold return failure otherwise wait periodSeconds and start a new check      In my case  my application can now bootstrap in a very clear way  so that I know I will not get periodic crashloopbackoff because sometimes it would be on the limit of those rates

User · Answer

From This page  the container dies after running everything correctly but crashes because all the commands ended  Either you make your services run on the foreground  or you create a keep alive script  By doing so  Kubernetes will show that your application is running  We have to note that in the Docker environment  this problem is not encountered  It is only Kubernetes that wants a running app   Update  an example    Here s how to avoid CrashLoopBackOff  when launching a Netshoot container   kubectl run netshoot --image nicolaka netshoot -- sleep infinity

User · Answer

In my case the problem was what Steve S  mentioned      The pod is crashing because it starts up then immediately exits  thus Kubernetes restarts and the cycle continues     Namely I had a Java application whose main threw an exception  and something overrode the default uncaught exception handler so that nothing was logged   The solution was to put the body of main into try         catch and print out the exception  Thus I could find out what was wrong and fix it    Another cause could be something in the app calling System exit  you could use a custom SecurityManager with an overridden checkExit to prevent  or log the caller of  exit  see https   stackoverflow com a 5401319 204205

User · Answer

As  Sukumar commented  you need to have your Dockerfile have a Command to run or have your ReplicationController specify a command    The pod is crashing because it starts up then immediately exits  thus Kubernetes restarts and the cycle continues

User · Answer

In my case  the issue was a misconstrued list of command-line arguments  I was doing this in my deployment file      args    -  quot --foo 10 quot    -  quot --bar 100 quot   Instead of the correct approach      args    -  quot --foo quot    -  quot 10 quot    -  quot --bar quot    -  quot 100 quot

User · Answer

i solved this problem by removing space between quotes and command value inside of array  this is happened because container exited after started and no executable command present which to be run inside of container    sh    -c    echo Hello Kubernetes   amp  amp  sleep 3600

User · Answer

I had same issue and now I finally resolved it  I am not using docker-compose file  I just added this line in my Docker file and it worked   ENV CI true   Reference  https   github com GoogleContainerTools skaffold issues 3882

User · Answer

In my case this error was specific to the hello-world docker image  I used the nginx image instead of the hello-world image and the error was resolved

User · Answer

In your yaml file  add command and args lines       containers        - name  api         image  localhost 5000 image-name          command     sleep            args     infinity          Works for me

User · Answer

kubectl -n  lt namespace-name gt  describe pod  lt pod name gt   kubectl -n  lt namespace-name gt  logs -p   lt pod name gt

User · Answer

Whilst troubleshooting the same issue I found no logs when using kubeclt logs  lt pod id gt   Therefore I ssh ed in to the node instance to try to run the container using plain docker  To my surprise this failed also    When entering the container with    docker exec -it faulty latest  bin sh   and poking around I found that it wasn t the latest version   A faulty version of the docker image was already available on the instance   When I removed the faulty latest instance with   docker rmi faulty latest   everything started to work

[kubernetes] My kubernetes pods keep crashing with "CrashLoopBackOff" but I can't find any log

Examples related to kubernetes