No replies to this topic

[sadronmeldir]

Hello all,

I'm trying to make a Java application that will allow for efficient searches through large amounts of data. The original format for the data is tens of millions of lines of text, each containing a channel ID, time stamp, user ID and text message of up to 255 characters. The logs are split across thousands of files, each containing up to 10 MB on information.

The intended purpose of the program I'm designing is to filter through this information by date, channel, user or keyword from the text in an efficient manner.

Unfortunately, I've never worked on a project of this scope before. Does anyone have any recommendation on how to index this information or some other way to search and parse through the data efficiently?