java heap size out of memory error

Hi All,

I was wondering if there is a better way to handle a large for loop's processing. Currently I have a for loop which processes 50k records. The way we get the 50k candidate records to process is before the for loop via database query. This query is a time consuming process and sometimes if the records pulled exceed 50k we run into the following error
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

What is the best way to deal with this should we request more memory on the app server or should we resort to improvisations in the code.
How to determine what heap size to specify in the app server configurations.

If we need to counter this in the code what would be the best way to do it. Is there anyway we can do the query pull in a staggered fashion, get the records in series of chunks one after the other and pass it on to the for loop process.

Please advise

Thanks,

What kind of records are they and what do you plan on doing with them?

These records are a collection of Hibernate POJO objects. We only fetch the column values which are absolutely necessary for processing.

You need to be more specific. What is the purpose of the records, or in this case, these columns? What processing do you need to do?

And why do you need to have them all in memory at the same time?

sorry for the partial info.

Stephan:
The database query to pull the records is kind of a feeder service functionality. Inside the for loop these records are further processed for all sort of java business logic calculations and the output stored in a database table for each such record retrieved.

Paul:
yes thats the question I meant to ask initially. Can I stagger the records retrieved that i fetch upfront currently. I don't need them all together but since earlier the volumes were low and we could retrieve them all at once we fetched everything in one go.

manish ahuja wrote:Paul:
yes thats the question I meant to ask initially. Can I stagger the records retrieved that i fetch upfront currently. I don't need them all together but since earlier the volumes were low and we could retrieve them all at once we fetched everything in one go.

I would say you should process the records one at a time, rather than loading them all into memory first. That's provided your application logic allows that, of course. Whether you can depends on a lot of things, none of which we can see from here.

We could but the additional database round-trip to fetch each record in this case will have a significant impact on the performance.
I was thinking on the lines of pagination where initially we only get the total record count to process and then divide them by say 10 to get discreet chunk sizes like first 10000, 10001-20000, 20001-30000 so on to process.

The records represented by individual chunks will then be pulled and passed on to the for loop and once this for loop ends we fetch the next set of records to process.
My only concern is how we can maintain the point to get the next set of records.

I may be totally wrong here, as this is not my area. But this sounds like a situation where you'd use a cursor (or at least you'd use what I believe a cursor to be).

A cursor is like a pointer into a query. it lets the database get the results and then you use the cursor to iterate through them. I don't know what the performance hit would be, but i imagine you could possibly set it up, have one thread getting the records into a buffer, and then have other threads processing them off. your getter would make sure there is always work for the processors to do by getting the next record each time the count in the buffer goes below a certain threshold (assuming there are any left to get).

manish ahuja wrote:We could but the additional database round-trip to fetch each record in this case will have a significant impact on the performance.

What's this extra round-trip you're talking about?

Normally you would do something like this:
ResultSet rs = // the result of a query while (rs.next()) { // process the row you just read }

There's aren't any "round-trips" in that code.

But you say you're using Hibernate? Probably Hibernate provides some advanced and convoluted way to do that simple thing, but I'm no Hibernate expert. If it doesn't, you might want to ask whether Hibernate is providing a net benefit to your design.

I think I didn't explain the context clearly. To stay with your example


ResultSet rs = //number of results from the query less than 50K works fine
while(rs.next())
{
// processing goes here
}



ResultSet rs = // number of results from the query is greater than 50K and fails with java heap out of memory error

while(rs.next())
{
// processing goes here
}

Now what i want to do is split the rs database query records into chunk sizes if greater than 50k and pull the records in smaller volumes from the database to work around the heap memory error.

paginate the resultset (retrieve a chunk of records) and for each pagination execute the for loop for the records retrieved.

Then you are building the ResultSet in such a way that all of its records get loaded into memory at the same time. Try to avoid doing that.

Pagination would be good option. You can store the number of records retrieved each time. That way each time you can select the number of rows such that the count is <= the number of records retrieved each time.

Agree there would be performance issues of hitting the dB multiple times for obtaining the ResulSet, but atleast better than coming across OutOfMemoryException.

Hi Manish ,

If you are using Oracle as your Database , try googling with rownum .

Paul Clapham wrote:Then you are building the ResultSet in such a way that all of its records get loaded into memory at the same time. Try to avoid doing that.

Sir, can you please elaborate your point.

What i think is that all the information will be retrieved and get stored in the ResultSet object. So, is it that the size of this object becomes very large, after getting more than 50K records, and hence heap is going outofmemory?
And if so then will running GC little earlier then this will help in solving the problem.

Please reply.

Thanks.