5 replies to this topic

[DoktorD1313]

Hey guys,

Here's the next project on my list..

What I need to accomplish is writing a program that is able to read both numerical and text data from .CSV files, store it possibly in several arrays, and then manipulate the data with a few forumlas.

I've already written a program to do this, however it's just an Excel file with macros to import the data and manipulate it within the spread sheet. The problem is that there are ~60,000+ lines of data (multiple sets of data perline) that need to be manipulated and the Excel macros just take an incredible amount of time to execute.

I've also written the program in FORTRAN that accomplishes this fairly quickly and efficiently, but once I got to the point of printing out the finished product, trying to format text size and layout to send to a modern printer proved to be a fruitless effort. Besides, there are other aspects to this program that I would like to incorperate that FORTRAN is just not capable of executing.


I know FORTRAN well, VB to an extent, and have dabbled with C/C#.

I'm open to suggestions as to what language might be able to accomplish this in the most efficient manner. I don't mind learning another programming language to do what I need.


For those of you that are interested in what EXACTLY I'm trying to do, I'm writing a Suggested Reorder report writer. The computer system exports several .CSV files with the item description/information and the monthly sales history for me. Then my program comes in and needs to be able to access only certain information pertaining to reordering out of these three CSV files. This involves retaining integers (item numbers), character strings (descriptions), and real numbers (decimal costs) each from seperate files and combining them into one record for each item.

It's a pain, but it needs to be done. I've already done it once, but it's on the very edge of Excel's limits and could be done much more cleanly via a command line program or something of that nature.



Any suggestions would be much appreciated since I'm not well informed of the strengths of each programming language out there!

[WingedPanther]

I would start by loading the data into a database. Microsoft SQL Server has tools for loading data into the database from csv file very quickly. Most of your calculations can be done as SQL statements. Then it's just a matter of reporting on the data, either using Access or a data-aware programming language such as Delphi, Lazarus, or any .NET.

[DoktorD1313]

After looking into it, that seems like a good route to go with. Thanks for the input, it'll give me a good place to start.

[fayyazlodhi]

C# has a pretty nice interface to read csv's in case you are interested

Building a Simple CSV Parser in C# | Switch on the Code
A Fast CSV Reader - CodeProject

But there is no reason why you shouldn't try out with loading into a db first unless there is good reason not doing so.

Edited by fayyazlodhi, 07 July 2011 - 03:11 PM.
adding another link

[DoktorD1313]

The only reason why I would resort strictly to a coded program written in C# or the like is if I wouldn't be able grasp the concepts of using and integrating databases into Visual Studio.

I don't have ANY experience with SQL, only programming languages like C#, FORTRAN, and VB. My previous program(s) was/were written using the meat and potatoes such as IF/THEN/ELSE statements, loops, case statements, and multiple arrays to temporarily store the data.

It seems like SQL would be a relief to use since I'm dealing with multiple data types in a single data table. I just need to take a while to figure out using databases. (I've only had a college class on FORTRAN.. The rest is self taught knowledge, which may have been misinterpreted and/or incomplete in the grand scheme of things :rolleyes: )

[ReignInChaos]

I agree with WingedPanther about using a database to interact with your data. Depending on the size of you data set, the data structures may eat up too much memory and the program may not be too stable. Once your manipulations are done, you can use an interface like JDBC, or my favorite, embedded SQL w/ C, to format your data nicely. Java is very clunky when it comes to formatting and printing data.