Fetching Data from a Form Generated Website

[pclark2]

I am working on a project that requires me to access course data from various university websites (just basic class schedule stuff). Most of these sites however have the data on a dynamically created page requiring post data from a form to show the desired information, and in some cases even require a login. I had been working in php to parse the course data from the websites where it is all on a nice static page, but do not know how to get it from these other sites.

My plan currently is to write a program to run on Ubuntu Linux to go out and fetch the data, but I am not sure which language would be best or even how I would do this. I have seen some programs control browsers before in Windows to get data, but I don't know if that is the best way or not or if that is even possible on Linux.

I don't know too much about software programming so I will need to learn a lot for this. So any ideas on what language to use or what functions to call or anything? Thanks for any help that can be given!

[neerajatcodecall]

i read ur problem. i m sorry i dnt know how to implement the same. buti have been also looking for the same thing. i want to access a site from which i can fetch the data and store it in my database. i m looking for a java program on windows. if u find pl tell me or mail me at neeraj.sehgal@lntinfotech.com.

thanx in advance!

[ablaye]

Why don't you use PHP/MySQL?

[migliris]

I'm having the same kind of problem: I want to fetch data from web pages that require JavaScript to be enabled on the browser. When my program fetches the URL, the webpage says "This page requires scripting to be enabled", whereas the same URL in my Firefox shows the web page with the prices and everything.

So my question: is there a way to enable scripting for a program (something with InputStream and OutputStream ) (I'm writing in Java by the way). It is likely that there isn't, to avoid denial-of-service hacking, and in this case, is there a way to make Java interact with Firefox so that Firefox fetches the page and retrieves it to the Java program?

This project really means a lot to me, so thanks for any advice (even if I have to change my programming language)

miguel

[migliris]

neeraj, you may want to check out this website:
Reading from and Writing to a URLConnection (The Java™ Tutorials > Custom Networking > Working with URLs)

[neerajatcodecall]

hi migliris!

thanx for the reply. i found the site useful and have done the task.
thanx again.