I configured tomcat to work with a different external open source.
However, after the tomcat is running for a few minutes I get:
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b3f02f]) and a value of type [org.apache.axis.MessageContext] (value [org.apache.axis.MessageContext@5dbd4e]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
What could cause it?
Where do I have to look? Could it be datapooling on Tomcat?
And what does it mean Threads in Tomcat?
EDITED
Here is my full trace. The application seems to reloads its context while it's still running - and I don't know why!
Mar 13, 2011 10:56:12 PM org.apache.catalina.core.StandardContext reload
INFO: Reloading this Context has started
Mar 13, 2011 10:56:12 PM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 1 instance(s) to be deallocated
Mar 13, 2011 10:56:13 PM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 1 instance(s) to be deallocated
Mar 13, 2011 10:56:14 PM org.apache.catalina.core.StandardWrapper unload
INFO: Waiting for 1 instance(s) to be deallocated
Mar 13, 2011 10:56:14 PM org.apache.catalina.core.ApplicationContext log
INFO: Closing Spring root WebApplicationContext
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/MyProject] registered the JBDC driver [com.mysql.jdbc.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesJdbc
SEVERE: The web application [/MyProject] registered the JBDC driver [oracle.jdbc.driver.OracleDriver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [NioSocketAcceptor-1] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [NioProcessor-1] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [NioProcessor-4] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [bitronix-disk-force-batcher] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [bitronix-scheduler] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] is still processing a request that has yet to finish. This is very likely to create a memory leak. You can control the time allowed for requests to finish by using the unloadDelay attribute of the standard Context implementation.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [NioProcessor-7] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/MyProject] appears to have started a thread named [NioProcessor-2] but has failed to stop it. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b5a8e1]) and a value of type [org.mvel2.debug.DebuggerContext] (value [org.mvel2.debug.DebuggerContext@16259fd]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@84b0b4]) and a value of type [com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl] (value [com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl@16d2cfa]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [null] (value [com.sun.faces.util.Util$1@16bbac9]) and a value of type [java.util.HashMap] (value [{com.sun.faces.patternCache={ = }}]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b3f02f]) and a value of type [org.apache.axis.MessageContext] (value [org.apache.axis.MessageContext@5dbd4e]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [org.apache.axis.utils.XMLUtils.ThreadLocalDocumentBuilder] (value [org.apache.axis.utils.XMLUtils$ThreadLocalDocumentBuilder@84b0b4]) and a value of type [com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl] (value [com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl@378584]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [org.springframework.core.NamedThreadLocal] (value [Transactional resources]) and a value of type [java.util.HashMap] (value [{org.hibernate.impl.SessionFactoryImpl@ccc27b=org.springframework.orm.hibernate3.SessionHolder@4f6ada}]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
Mar 13, 2011 10:56:15 PM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap
SEVERE: The web application [/MyProject] created a ThreadLocal with key of type [null] (value [com.sun.faces.application.ApplicationAssociate$1@1f01fcf]) and a value of type [com.sun.faces.application.ApplicationAssociate] (value [com.sun.faces.application.ApplicationAssociate@1b85528]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak.
2011-03-13 22:57:27,734 ERROR ( ContextLoader.java:220) - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'transactionManager' defined in class path resource [applicationContext-hibernate.xml]: Cannot resolve reference to bean 'sessionFactory' while setting bean property 'sessionFactory'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'sessionFactory' defined in class path resource [applicationContext-hibernate.xml]: Invocation of init method failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:328)
at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:106)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyPropertyValues(AbstractAutowireCapableBeanFactory.java:1325)
The key "Transactional Resources" looks like you are talking to the database without a proper transaction. Make sure transaction management is configured properly and no invocation path to the DAO exists that doesn't run under a @Transactional annotation. This can easily happen when you configured transaction management on the Controller level but are invoking DAOs in a timer or are using @PostConstruct annotations. I wrote it up here http://georgovassilis.blogspot.nl/2014/01/tomcat-spring-and-memory-leaks-when.html
Edit: It looks like this is (also?) a bug with spring-data-jpa which has been fixed with v1.4.3. I looked it up in the spring-data-jpa sources of LockModeRepositoryPostProcessor which sets the "Transactional Resources" key. In 1.4.3 it also clears the key again.
Sometimes this has to do with configuration changes. When we upgraded from Tomncat 6.0.14 to 6.0.26, we had seen something similar. here is the solution http://www.skill-guru.com/blog/2010/08/22/tomcat-6-0-26-shutdown-reports-a-web-application-created-a-threadlocal-threadlocal-has-been-forcibly-removed/
I added the following to @PreDestroy method in my CDI @ApplicationScoped bean, and when I shutdown TomEE 1.6.0 (tomcat7.0.39, as of today), it clears the thread locals.
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package pf;
import java.lang.ref.WeakReference;
import java.lang.reflect.Array;
import java.lang.reflect.Field;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
/**
*
* @author Administrator
*
* google-gson issue # 402: Memory Leak in web application; comment # 25
* https://code.google.com/p/google-gson/issues/detail?id=402
*/
public class ThreadLocalImmolater {
final Logger logger = LoggerFactory.getLogger(ThreadLocalImmolater.class);
Boolean debug;
public ThreadLocalImmolater() {
debug = true;
}
public Integer immolate() {
int count = 0;
try {
final Field threadLocalsField = Thread.class.getDeclaredField("threadLocals");
threadLocalsField.setAccessible(true);
final Field inheritableThreadLocalsField = Thread.class.getDeclaredField("inheritableThreadLocals");
inheritableThreadLocalsField.setAccessible(true);
for (final Thread thread : Thread.getAllStackTraces().keySet()) {
count += clear(threadLocalsField.get(thread));
count += clear(inheritableThreadLocalsField.get(thread));
}
logger.info("immolated " + count + " values in ThreadLocals");
} catch (Exception e) {
throw new Error("ThreadLocalImmolater.immolate()", e);
}
return count;
}
private int clear(final Object threadLocalMap) throws Exception {
if (threadLocalMap == null)
return 0;
int count = 0;
final Field tableField = threadLocalMap.getClass().getDeclaredField("table");
tableField.setAccessible(true);
final Object table = tableField.get(threadLocalMap);
for (int i = 0, length = Array.getLength(table); i < length; ++i) {
final Object entry = Array.get(table, i);
if (entry != null) {
final Object threadLocal = ((WeakReference)entry).get();
if (threadLocal != null) {
log(i, threadLocal);
Array.set(table, i, null);
++count;
}
}
}
return count;
}
private void log(int i, final Object threadLocal) {
if (!debug) {
return;
}
if (threadLocal.getClass() != null &&
threadLocal.getClass().getEnclosingClass() != null &&
threadLocal.getClass().getEnclosingClass().getName() != null) {
logger.info("threadLocalMap(" + i + "): " +
threadLocal.getClass().getEnclosingClass().getName());
}
else if (threadLocal.getClass() != null &&
threadLocal.getClass().getName() != null) {
logger.info("threadLocalMap(" + i + "): " + threadLocal.getClass().getName());
}
else {
logger.info("threadLocalMap(" + i + "): cannot identify threadlocal class name");
}
}
}
This problem appears when we are using any third party solution, without using the handlers for the cleanup activitiy. For me this was happening for EhCache. We were using EhCache in our project for caching. And often we used to see following error in the logs
SEVERE: The web application [/products] appears to have started a thread named [products_default_cache_configuration] but has failed to stop it. This is very likely to create a memory leak.
Aug 07, 2017 11:08:36 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/products] appears to have started a thread named [Statistics Thread-products_default_cache_configuration-1] but has failed to stop it. This is very likely to create a memory leak.
And we often noticed tomcat failing for OutOfMemory error during development where we used to do backend changes and deploy the application multiple times for reflecting our changes.
This is the fix we did
<listener>
<listener-class>
net.sf.ehcache.constructs.web.ShutdownListener
</listener-class>
</listener>
So point I am trying to make is check the documentation of the third party libraries which you are using. They should be providing some mechanisms to clean up the threads during shutdown. Which you need to use in your application. No need to re-invent the wheel unless its not provided by them. The worst case is to provide your own implementation.
Reference for EHCache Shutdown http://www.ehcache.org/documentation/2.8/operations/shutdown.html
Source: Stackoverflow.com