Jump to content


Check out our Community Blogs

Register and join over 40,000 other developers!


Recent Status Updates

View All Updates

Photo
- - - - -

[C++] reading directly into stringstream from file

string stream

  • Please log in to reply
3 replies to this topic

#1 Flying Dutchman

Flying Dutchman

    CC Leader

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1090 posts
  • Location:::1
  • Programming Language:C++, Python

Posted 05 April 2011 - 05:49 AM

I'm parsing text files and some can be really huge (200k+ lines) so reading line by line directly from file can slow the process down.

Best I have so far is to get the file size, create a char buffer, read into that buffer and then create stringstream object from that buffer. But that doubles the amount of memory used.

I'm looking for a way to read directly into stringstream if it's possible, or any other means that might speed up (file reading) and save memory as well.
  • 0

The roots of education are bitter, but the fruit is sweet.


#2 gregwarner

gregwarner

    Obi Wan of Programming

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1586 posts
  • Location:Arkansas
  • Programming Language:C, Java, C++, C#, PHP, Transact-SQL

Posted 06 April 2011 - 10:56 AM

Here's how I'd do it:

Use the write(const char* s , streamsize n) method of stringstream in order to write the characters you're reading from the file directly into your stringstream's internal stringbuf. That way, you would not need to use double the memory to hold the whole file. You can't avoid having a char * buffer entirely, so I would make a char * of size 1024 bytes, or whatever the blocksize of your filesystem is, and continue reading one block at a time and writing it to the stringstream until the entire file is read into memory. You'll have to open your stringstream object with both modes ios_base::in and ios_base::out for this to work.

It's been many years since I've coded file IO in C++, so please tell me if that doesn't work.
  • 0

ti-99-sig.png
Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
– Douglas Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid


#3 Flying Dutchman

Flying Dutchman

    CC Leader

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1090 posts
  • Location:::1
  • Programming Language:C++, Python

Posted 06 April 2011 - 02:29 PM

Here's how I did it
while (!file.eof()) {
    file.read(buffer, size);
    std::istringstream iss(buffer);
    while (!iss.eof()) {
        //...
    }
}
Since the code is still in development I haven't really fully profiled it but so far it works great. Do you think implementing a dynamic buffer size, size according to filesystem's block size, would increase performance?

Edited by Flying Dutchman, 06 April 2011 - 02:34 PM.
typos

  • 0

The roots of education are bitter, but the fruit is sweet.


#4 gregwarner

gregwarner

    Obi Wan of Programming

  • Expert Member
  • PipPipPipPipPipPipPip
  • 1586 posts
  • Location:Arkansas
  • Programming Language:C, Java, C++, C#, PHP, Transact-SQL

Posted 07 April 2011 - 05:09 AM

No, probably not. But I'm not familiar with the performance statistics of C++'s stringstream. Sorry, I'm probably not the best person to answer your question.
  • 0

ti-99-sig.png
Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.
– Douglas Hofstadter, Gödel, Escher, Bach: An Eternal Golden Braid






Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download