High CPU Utilization in java application - why

Question

I have a Java Application  web-based  that at times shows very high CPU Utilization  almost 90   for several hours  Linux TOP command shows this  On application restart  the problem goes away   So to investigate   I take Thread Dump to find what threads are doing  Several Threads are found in  RUNNABLE  state  some in few other states  On taking repeated Thread Dumps  i do see some threads that are always present in  RUNNABLE  state  So  they appear to be the culprit   But I am unable to tell for sure  which Thread is hogging the CPU or has gone into a infinite loop  thereby causing high CPU util    Logs don t necessarily help  as the offending code may not be logging anything   How do I investigate - What part of the application or what-thread is causing High CPU Utilization  - Any other ideas

User · Answer

During these peak CPU times  what is the user load like   You say this is a web based application  so the culprits that come to mind is memory utilization issues   If you store a lot of stuff in the session  for instance  and the session count gets high enough  the app server will start thrashing about   This is also a case where the GC might make matters worse depending on the scheme you are using   More information about the app and the server configuration would be helpful in pointing towards more debugging ideas

User · Answer

If a profiler is not applicable in your setup  you may try to identify the thread following steps in this post   Basically  there are three steps    run top -H and get PID of the thread with highest CPU  convert the PID to hex  look for thread with the matching HEX PID in your thread dump

User · Answer

You may be victim of a garbage collection problem   When your application requires memory and it s getting low on what it s configured to use the garbage collector will run often which consume a lot of CPU cycles  If it can t collect anything your memory will stay low so it will be run again and again  When you redeploy your application the memory is cleared and the garbage collection won t happen more than required so the CPU utilization stays low until it s full again   You should check that there is no possible memory leak in your application and that it s well configured for memory  check the -Xmx parameter  see What does Java option -Xmx stand for    Also  what are you using as web framework  JSF relies a lot on sessions and consumes a lot of memory  consider being stateless at most

User · Answer

Flame graphs can be helpful in identifying the execution paths that are consuming the most CPU time  In short  the following are the steps to generate flame graphs yum -y install perf  wget https   github com jvm-profiling-tools async-profiler releases download v1 8 3 async-profiler-1 8 3-linux-x64 tar gz  tar -xvf async-profiler-1 8 3-linux-x64 tar gz chmod -R 777 async-profiler-1 8 3-linux-x64 cd async-profiler-1 8 3-linux-x64  echo 1  gt   proc sys kernel perf event paranoid echo 0  gt   proc sys kernel kptr restrict  JAVA PID  pgrep java     profiler sh -d 30  JAVA PID -f flame-graph svg  flame-graph svg can be opened using browsers as well  and in short  the width of the element in stack trace specifies the number of thread dumps that contain the execution flow relatively  There are few other approaches to generating them  By introducing -XX  PreserveFramePointer as the JVM options as described here Using async-profiler with -XX  UnlockDiagnosticVMOptions -XX  DebugNonSafepoints as described here  But using async-profiler without providing any options though not very accurate  can be leveraged with no changes to the running Java process with low CPU overhead to the process  Their wiki provides details on how to leverage it  And more about flame graphs can be found here

User · Answer

Your first approach should be to find all references to Thread sleep and check that    Sleeping is the right thing to do - you should use some sort of wait mechanism if possible - perhaps careful use of a BlockingQueue would help  If sleeping is the right thing to do  are you sleeping for the right amount of time - this is often a very difficult question to answer    The most common mistake in multi-threaded design is to believe that all you need to do when waiting for something to happen is to check for it and sleep for a while in a tight loop  This is rarely an effective solution - you should always try to wait for the occurrence   The second most common issue is to loop without sleeping  This is even worse and is a little less easy to track down

User · Answer

In the thread dump you can find the Line Number as below   for the main thread which is currently running      main   1 prio 5 os prio 0 tid 0x0000000002120800 nid 0x13f4 runnable  0x0000000001d9f000     java lang Thread State    RUNNABLE       at java io FileOutputStream writeBytes Native Method      at java io FileOutputStream write FileOutputStream java 313      at com rana samples   HighCPUUtilization main HighCPUUtilization java 17

User · Answer

You did not assign the  linux  to the question but you mentioned  Linux top   And thus this might be helpful   Use the small Linux tool threadcpu to identify the most cpu using threads  It calls jstack to get the thread name  And with  sort -n  in pipe you get the list of threads ordered by cpu usage   More details can be found here  http   www tuxad com blog archives 2018 10 01 threadcpu - show cpu usage of threads index html  And if you still need more details then create a thread dump or run strace on the thread

[java] High CPU Utilization in java application - why?

Examples related to java

Examples related to multithreading

Examples related to performance

Examples related to web-applications

Examples related to cpu-usage