Java Answers Forum - trying to improve code to avoid java.lang.OutOfMemory error

Articles |
News |
Weblogs |
Books |
Forums

Artima Forums | Articles | Weblogs | Java Answers | News

Sponsored Link •

Java Answers Forum
trying to improve code to avoid java.lang.OutOfMemory error

5 replies on 1 page. Most recent reply: Aug 4, 2008 11:26 AM by James Watson

Welcome Guest
Sign In

Back to Topic List

Reply to this Topic

Search Forum

Threaded View


Previous Topic		Next Topic

Flat View: This topic has 5 replies on 1 page

crystal smith

Posts: 2
Nickname: bdyarem
Registered: Jul, 2008

trying to improve code to avoid java.lang.OutOfMemory error

Posted: Jul 29, 2008 3:57 PM

Hello,

I am a newcomer to the forums here and Java novice. I have searched around quite a bit and will continue to do so, but thought I'd put this question to the user community in hopes someone can point out an obvious mistake I may be making...

I have a java class that executes a series of file loads, and as these file loads are executed the program uses a large amount of heap memory, eventually causing a failure with the java.lang.OutOfMemoryError. I can increase the JVM memory with -Xmx1024m (about the largest I can go on the machine running the program), and in most cases this will allow the program to run to completion. However, when given a large input, 1024m is still proving to be insufficient, and I'm pretty sure my code could use a few obvious improvements to allow for less memory consumption.

Using JProbe profiler, it appears my program consumes (and doesn't release) a large amount of memory during the following statement, which writes to a CSV file. ps2 is a PrintStream object that I am declaring with the autoflush = true option (which isn't seeming to help). Basically this nasty code is writing my CSV formatted file, and this statement can be iterated in upwards of 40K times. I've also tried explicitly nulling the ps2 object and forcing a garbage collection with gc() after all rows have been written, but isn't helping either. Any suggestions?

for(x=0;x < myQuery.getResultSize();x++)
{
ps2.println("\"" + myInfoObject.getFiles().myFilePathMethod() + "\"" + "," +
"\"" + ((IFile)myInfoObject.getFiles().get(0)).getName() + "\"" + "," +
"\"" + ((IFile)myInfoObject.getFiles().get(0)).getSize() + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN1") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN2") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN3") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN4") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN5") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN6") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN7") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN8") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN9") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN10") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN11") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN12") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN13") + "\"" + "," +
"\"" + ((IInfoObject)myQuery.get(x)).properties().getProp erty("COLUMN14") + "\"");

}

Kondwani Mkandawire

Posts: 530
Nickname: spike
Registered: Aug, 2004

Re: trying to improve code to avoid java.lang.OutOfMemory error

Posted: Jul 30, 2008 12:44 AM

PLEASE USE THE JAVA TAGS AND PROPER INDENTITION (SHOWN ON YOUR RIGHT-HAND SIDE WHEN POSTING.

Depending on how many times you are looping, try using a StringBuffer for concatenation as opposed to a String with the + operator.

E.g.

instead of:

Str str = "\"" + someString +"\"" + anotherString;

You do:

StringBuffer buf = new StringBuffer();
buf.append("\"").append(someString).append("\"").append(anotherString);

then go:

buf.toString();

when you want to use it.

It is much more efficient, not sure how drastically this will reduce your memory consumption though (but give that a shot).

Kondwani Mkandawire

Posts: 530
Nickname: spike
Registered: Aug, 2004

Re: trying to improve code to avoid java.lang.OutOfMemory error

Posted: Jul 30, 2008 12:51 AM

If you google StringBuffer vs String Java it leads you to this article, which will explain that there is actually a significant difference in Performance:

http://www.javaworld.com/javaworld/jw-03-2000/jw-0324-javaperf.html

Hope this helps.

crystal smith

Posts: 2
Nickname: bdyarem
Registered: Jul, 2008

Re: trying to improve code to avoid java.lang.OutOfMemory error

Posted: Jul 31, 2008 9:43 AM

Thank you very much for your suggestions. I seem to have corrected the memory consumption issue, and I'm afraid the solution was quite simple: 1) re-use 1 printStream object throughout the code, and more importantly 2) null out my query result objects after I am finished with each. 2) freed up a ton of memory.

Thanks again....

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: trying to improve code to avoid java.lang.OutOfMemory error

Posted: Aug 4, 2008 11:19 AM

> PLEASE USE THE JAVA TAGS AND PROPER INDENTITION (SHOWN ON
> YOUR RIGHT-HAND SIDE WHEN POSTING.
>
> Depending on how many times you are looping, try using a
> StringBuffer for concatenation as opposed to a String with
> the + operator.
>
> E.g.
>
> instead of:
>
>

> Str str = "\"" + someString +"\"" + anotherString;
>

>
> You do:
>
>

> StringBuffer buf = new StringBuffer();
> buf.append("\"").append(someString).append("\"").append(ano
> therString);
>

>
> then go:
>
> buf.toString();
>
> when you want to use it.
>
> It is much more efficient, not sure how drastically this
> will reduce your memory consumption though (but give that
> a shot).

Actually in modern versions of Javac (1.4+, maybe even in 1.3), the two ways of concatenating Strings compile to basically the same bytecodes. The compiler uses a StringBuffer under the hood. In 1.5 and beyond, the + version may even be faster as the compiler will use a StringBuilder instead of a StringBuffer.

Even if this were an old version of Java where a new String was created on each concat, this should not cause an OOME because the Strings should be available for GC as soon as the next concat call completes.

James Watson

Posts: 2024
Nickname: watson
Registered: Sep, 2005

Re: trying to improve code to avoid java.lang.OutOfMemory error

Posted: Aug 4, 2008 11:26 AM

> Thank you very much for your suggestions. I seem to have
> corrected the memory consumption issue, and I'm afraid the
> solution was quite simple: 1) re-use 1 printStream object
> throughout the code, and more importantly 2) null out my
> query result objects after I am finished with each. 2)
> freed up a ton of memory.
>
> Thanks again....

One thing that you can do with the code you posted is change it to something like this:

for(x=0;x < myQuery.getResultSize();x++)
{
ps2.print("\"");
ps2.print(myInfoObject.getFiles().myFilePathMethod());
ps2.print("\",\"");
ps2.print(((IFile)myInfoObject.getFiles().get(0)).getName());
 
// ... the rest
 
ps2.println();
}

That way you avoid the cost of allocating an unnecessary large character array in memory.

You should also look at http://ostermiller.org/utils/CSV.html. It's a good library and writes proper CSV files for a number of target parsers.

Flat View: This topic has 5 replies on 1 page

Previous Topic

Next Topic


	Web Artima.com