6 replies to this topic

[tontonskie]

how to solve OutOfMemoryException?

i'm creating a program that reads a log file with 40,000 + lines of string . . . then i'm putting it on an array . . . when the starts to read the contents of the log file it replies an error . . . and that's the outofmemory exception . . . .


can anyone give me an algorithm to solve this. . .

the program shows every line of the log file at the same time i'm putting it on the array . . . i cannot remove the contents of the array because i will use it to arrange the contents of the file . . .

[lavaCode]

You can always allocate more memory to Java.

[eafkuor]

Can you read the first 50 lines and arrange those, then read the next 50 and arrange those, and so on? I have no idea if this could work because you didn't tell us anything about what your program wants to do :D

I don't like the idea of allocating more memory.. what if that isn't enough too? What if the log becomes 80000 lines?

[mnirahd]

Hi,

This suggests that you need to allocate enough memory to hold the bytes from the file. The best way of doing it is

1. Open File
2. Calculate File Size
3. Allocate Bytes, read file: convert Bytes to String.

I hope this helps!

Munir

[Liars_paradox]

tontonskie said:

how to solve OutOfMemoryException?

i'm creating a program that reads a log file with 40,000 + lines of string . . . then i'm putting it on an array . . . when the starts to read the contents of the log file it replies an error . . . and that's the outofmemory exception . . . .


can anyone give me an algorithm to solve this. . .

the program shows every line of the log file at the same time i'm putting it on the array . . . i cannot remove the contents of the array because i will use it to arrange the contents of the file . . .

The java.lang.OutOfMemoryException is just the JVM's way of telling you that you don't have enough room to do what you're doing. I would say that you're probably placing more data into the array than what you can store into it, but you didn't mention anything like an array index out of bounds exception or anything like that.

It's possible that your machine can't handle an array of the size that you want to have your array. Here's what I found when I researched this online, "There is no explicit maximum, but the String API won't support Strings longer than Integer.MAX_VALUE (in excess of two billion) characters. Since characters are two bytes in size, a maximal String would require more than 4GB of storage". (source)

A worst-case scenario is that your array of strings which contains 40,000 members, will have every one of its members take up 4GB in memory space. If that were to happen, then you would have a data structure that takes up 156.25 TB in memory. But, the reality is that the JVM won't let you do this because it itself is but only so large and will only take up so much space in your real OS.

Since, your problem could potentially take up a lot of memory space, it seems that you have to process the data within the array before you finish reading your input file and storing them somewhere else. Doing this would avoid taking up too much space in RAM memory, but could be a challenge if you've never worked with threads in Java.

Assuming that you're not some programming student who doesn't know how to finish his assignment in Data Structures class, then I hope that you're familiar with the Producer-Consumer problem. This is something that I learned in college in my Operating Systems class. You will need to create different threads which fill your memory buffer and then create another set of threads to take the data outside of the data buffer.

I've never done this before in Java, so I don't have an example to give to you. However, I'm sure that there's tons of online articles for you to research in order for you to study. Another solution could be that you set the JVM variable MaxPermSize to a size that is large enough for what you're trying to accomplish. (source) But, that might be possible if you're array is just simply going to be too big for your computer.

[lethalwire]

I like the idea that eafkuor has, it seems he has the correct general idea.
Quite a few helpful hints come up when you Google "reading large files in java"

This link might be a feasible approach to your problem.
Handling large data files efficiently with Java

If it's not, like I stated before, there are quite a few approaches to this problem.

[josep]

The problem might be that you are trying to read data, create an array and write to an array all at the same time which is not a good idea. The best way would be to read one line of data at a time, save it to an object and store that in a collection. The other thing you should avoid is to create objects inside a loop, especially if you are dealing with large amounts of data, this might hamper the jvm because some objects might be unnecessarily occupying memory and the jvm can only garbage collect the eligible objects after the loop has finished executing.