try {
// CompareRecord record = new CompareRecord();
Connection conn = new CompareRecord().getConection("eliteddaprd","eliteddaprd","192.168.14.104","1521");
ResultSet res = null;
if (conn != null){
Statement stmt = conn.createStatement();
res = stmt.executeQuery("select rowindx,ADDRLINE1 from dedupinitial order by rowindx");
}
Map<Integer,String> adddressMap = new LinkedHashMap<Integer, String>();
if (res != null){
System.out.println("result set is not null ");
while(res.next()){
adddressMap.put(res.getInt(1),res.getString(2));
}
}
System.out.println("address Map size =========> "+adddressMap.size());
Iterator it = adddressMap.entrySet().iterator();
int count = 0;
int min = 0;
while (it.hasNext()){
Map.Entry pairs = (Map.Entry)it.next();
Pattern p = Pattern.compile("[,\\s]+");
Integer outerkey = (Integer)pairs.getKey();
String outerValue = (String)pairs.getValue();
//System.out.println("outer Value ======> "+outerValue);
String[] outerresult = p.split(outerValue);
Map.Entry pairs2 = null;
count++;
List<Integer> dupList = new ArrayList<Integer>();
Iterator innerit = adddressMap.entrySet().iterator();
boolean first = true;
while (innerit.hasNext()){
//System.out.println("count value ===> "+count);
int totmatch = 0;
if(first){
if(count == adddressMap.size()){
break;
}
for(int i=0;i<=count;i++){
pairs2 = (Map.Entry)innerit.next();
}
first = false;
}
else{
pairs2 = (Map.Entry)innerit.next();
}
Integer innterKey = (Integer)pairs2.getKey();
String innerValue = (String)pairs2.getValue();
//System.out.println("innrer value "+innerValue);
String[] innerresult = p.split(innerValue);
for(int j=0;j<outerresult.length;j++){
for(int k=0;k<innerresult.length;k++){
if(outerresult[j].equalsIgnoreCase(innerresult[k])){
//System.out.println(outerresult[j]+" Match With "+innerresult[k]);
totmatch++;
break;
}
}
}
min = Math.min(outerresult.length, innerresult.length);
if(min != 0 && ((totmatch*100)/min) > 50) {
//System.out.println("maching inner key =========> "+innterKey);
dupList.add(innterKey);
}
}
//System.out.println("Duplilcate List Sisze ===================> "+dupList.size()+" "+outerkey);
}
System.out.println("End =========> "+new Date());
}
catch (Exception e) {
e.printStackTrace();
}
Here ResultSet have processed around 500000 records, but it will give me error like:
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:508)
at java.util.LinkedHashMap.addEntry(LinkedHashMap.java:406)
at java.util.HashMap.put(HashMap.java:431)
at spite.CompareRecord.main(CompareRecord.java:91)
I know this error comes because of VM memory, but don't know how to increase it in Eclipse?
What do I do if I have to process even more than 500,000 records?
This question is related to
java
eclipse
exception-handling
I am not pro in Java but your problem can be solved by "blockingqueue" if you use it wisely.
Try to retrieve a chunk of records first, process them, and iterate the process until you complete your processing. This may help you to get rid of the OutOfMemory Exceptions
.
Sometimes the exception will not stop after you increase the memory in eclipse ini file. then try below option
Go to Window >> Preferences >> MyEclipse >> Java Enterprise Project >> Web Project >> Deployment Tab Under -> Under Library Deployment Policies UnCheck -> Jars from User Libraries
You may want to close open projects in your Project Explorer. You can do this by right-clicking an open project, and selecting "Close Project".
This is what worked for me after I noticed that I had to periodically increase the heap size in my_directory\eclipse\eclipse.ini.
Hope this helps! Peace!
Please make sure your code is fine. I too got stuck in this problem once and tried the solution accepted here but in vain. So I wrote my code again. Apparently I was using a custom array list and adding the values from an array. I tried changing the ArrayList to accept the primitive values only and it worked.
How to give your program more memory when running from Eclipse
Go to Run / Run Configurations. Select the run configuration for your program. Click on the tab "Arguments". In the "Program arguments" area, add a -Xmx
argument, for example -Xmx2048m
to give your program a max. of 2048 MB (2 GB) memory.
How to prevent this memory problem
Re-write your program in such a way that it does not need to store so much data in memory. I haven't looked at your code in detail, but it looks like you're storing a lot of data in a HashMap
; more than fits in memory when you have a lot of records.
To increase the Heap size in eclipse change the eclipse.ini file.
refer to
http://wiki.eclipse.org/FAQ_How_do_I_increase_the_heap_size_available_to_Eclipse%3F
In eclipse.ini file, make below entries
-Xms512m
-Xmx2048m
-XX:MaxPermSize=1024m
You can increase the size of the memory through the use of commandline arguments.
See this link.
eclipse -vmargs -Xmx1024m
Edit: Also see see this excellent question
The accepted solution of modifying a Run Configuration wasn't ideal for me as I have a few different run configurations and could easily forget to do this when adding further ones in future. Also I wanted the setting to apply whenever running anything, e.g. when running JUnit tests by right-clicking and selecting "Run As" -> "JUnit Test".
The above can be achieved by modifying the JRE/JDK settings instead:-
-Xms512m -Xmx4G -XX:MaxPermSize=256M
In the Eclipse download folder make the entries in the eclipse.ini
file :
--launcher.XXMaxPermSize
512M
-vmargs
-Dosgi.requiredJavaVersion=1.5
-Xms512m
-Xmx1024m
or what ever values you want.
What to do if i have to processed even more that 500000 records ?
There are a few ways, to increase the java heap size for your app where a few have suggested already. Your app need to remove the elements from your adddressMap as your app add new element into it and so you won't encounter oom if there are more records coming in. Look for producer-consumer if you are interested.
See http://blog.headius.com/2009/01/my-favorite-hotspot-jvm-flags.html
-Xms and -Xmx set the minimum and maximum sizes for the heap. Touted as a feature, Hotspot puts a cap on heap size to prevent it from blowing out your system. So once you figure out the max memory your app needs, you cap it to keep rogue code from impacting other apps. Use these flags like -Xmx512M, where the M stands for MB. If you don't include it, you're specifying bytes. Several flags use this format. You can also get a minor startup perf boost by setting minimum higher, since it doesn't have to grow the heap right away.
-XX:MaxPermSize=###M sets the maximum "permanent generation" size. Hotspot is unusual in that several types of data get stored in the "permanent generation", a separate area of the heap that is only rarely (or never) garbage-collected. The list of perm-gen hosted data is a little fuzzy, but it generally contains things like class metadata, bytecode, interned strings, and so on (and this certainly varies across Hotspot versions). Because this generation is rarely or never collected, you may need to increase its size (or turn on perm-gen sweeping with a couple other flags). In JRuby especially we generate a lot of adapter bytecode, which usually demands more perm gen space.
Go to "Window -> Preferences -> General -> C/C++ -> Code analysis" and disable "Syntax and Semantics Errors -> Abstract class cannot be instantiated"
In Eclipse goto Run->Run Configuration find the Name of the class you have been running, select it, click the Target tab then in "Additional Emulator Command Line Options" add:
-Xms512M -Xmx1524M
then click apply.
Source: Stackoverflow.com