Simple Spider for Link Validation
I've made a simple 2 part shell script.
The first part uses wget to extract url links
from the codecall index page. The tricky part was getting
the regular expressions correct as the file was filtered
through grep and gawk.
The second part also uses wget in spider mode as it
attempts to visit each link and report the result.
Here is the script:
Here the script is made executable and kicked off.
(it took over a minute to complete)
Here is the link.txt file that the first part creates.
Be sure to glance at the index file source and see how
well the filters work.
Code:http://codecall.net/index.html http://forum.codecall.net http://blog.codecall.net http://codecall.net/component/ionfiles/ http://wiki.codecall.net http://forum.codecall.net/tutorials/ http://codecall.net/resources.html http://freelance.codecall.net http://forum.codecall.net/tutorials/ http://forum.codecall.net http://freelance.codecall.net http://forum.codecall.net/register.php http://forum.codecall.net/blog.php?b=440 http://forum.codecall.net/blog.php?b=440 http://forum.codecall.net/blog.php?b=438 http://forum.codecall.net/blog.php?b=438 http://forum.codecall.net/blog.php?b=437 http://forum.codecall.net/blog.php?b=437 http://forum.codecall.net/external.php?type=RSS2 http://twitter.com/CodeCall http://www.facebook.com/people/Code-Call/1159055559 http://forum.codecall.net/showthread.php?t=21258 http://forum.codecall.net/showthread.php?t=21258 http://forum.codecall.net/showthread.php?t=21242 http://forum.codecall.net/showthread.php?t=21242 http://forum.codecall.net/showthread.php?t=21230 http://forum.codecall.net/showthread.php?t=21230 http://forum.codecall.net/showthread.php?t=21227 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21178 http://forum.codecall.net/forumdisplay.php?f=33 http://forum.codecall.net/showthread.php?t=21300 http://forum.codecall.net/forumdisplay.php?f=12 http://forum.codecall.net/showthread.php?t=21158 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21298 http://forum.codecall.net/forumdisplay.php?f=36 http://forum.codecall.net/showthread.php?t=21288 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21294 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21281 http://forum.codecall.net/forumdisplay.php?f=12 http://forum.codecall.net/showthread.php?t=21063 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=19383 http://forum.codecall.net/forumdisplay.php?f=64 http://forum.codecall.net/showthread.php?t=21216 http://forum.codecall.net/forumdisplay.php?f=16 http://forum.codecall.net/showthread.php?t=21276 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21290 http://forum.codecall.net/forumdisplay.php?f=73 http://forum.codecall.net/showthread.php?t=556 http://forum.codecall.net/forumdisplay.php?f=31 http://forum.codecall.net/showthread.php?t=21285 http://forum.codecall.net/forumdisplay.php?f=36 http://forum.codecall.net/showthread.php?t=21299 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21297 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21289 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21246 http://forum.codecall.net/forumdisplay.php?f=89 http://forum.codecall.net/showthread.php?t=21224 http://forum.codecall.net/forumdisplay.php?f=16 http://forum.codecall.net/showthread.php?t=21217 http://forum.codecall.net/forumdisplay.php?f=10 http://forum.codecall.net/showthread.php?t=21189 http://forum.codecall.net/forumdisplay.php?f=99 http://forum.codecall.net/showthread.php?t=21191 http://forum.codecall.net/forumdisplay.php?f=31 http://forum.codecall.net/showthread.php?t=21170 http://forum.codecall.net/forumdisplay.php?f=124 http://forum.codecall.net/showthread.php?t=21165 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21156 http://forum.codecall.net/forumdisplay.php?f=26 http://forum.codecall.net/showthread.php?t=21138 http://forum.codecall.net/forumdisplay.php?f=99 http://forum.codecall.net/showthread.php?t=21091 http://forum.codecall.net/forumdisplay.php?f=99 http://forum.codecall.net/showthread.php?t=21080 http://forum.codecall.net/forumdisplay.php?f=10 http://forum.codecall.net/showthread.php?t=21047 http://forum.codecall.net/forumdisplay.php?f=124 http://forum.codecall.net/showthread.php?t=21300 http://forum.codecall.net/forumdisplay.php?f=12 http://forum.codecall.net/showthread.php?t=21299 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21298 http://forum.codecall.net/forumdisplay.php?f=36 http://forum.codecall.net/showthread.php?t=21297 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21294 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21290 http://forum.codecall.net/forumdisplay.php?f=73 http://forum.codecall.net/showthread.php?t=21289 http://forum.codecall.net/forumdisplay.php?f=11 http://forum.codecall.net/showthread.php?t=21288 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21285 http://forum.codecall.net/forumdisplay.php?f=36 http://forum.codecall.net/showthread.php?t=21281 http://forum.codecall.net/forumdisplay.php?f=12 http://forum.codecall.net/showthread.php?t=21280 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21279 http://forum.codecall.net/forumdisplay.php?f=102 http://forum.codecall.net/showthread.php?t=21278 http://forum.codecall.net/forumdisplay.php?f=103 http://forum.codecall.net/showthread.php?t=21276 http://forum.codecall.net/forumdisplay.php?f=7 http://forum.codecall.net/showthread.php?t=21275 http://forum.codecall.net/forumdisplay.php?f=13 http://forum.codecall.net/showthread.php?t=21295 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=21256 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=21241 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=21201 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=21200 http://forum.codecall.net/forumdisplay.php?f=100 http://forum.codecall.net/showthread.php?t=21159 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=21076 http://forum.codecall.net/forumdisplay.php?f=100 http://forum.codecall.net/showthread.php?t=21073 http://forum.codecall.net/forumdisplay.php?f=100 http://forum.codecall.net/showthread.php?t=21077 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=21062 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=20998 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=20992 http://forum.codecall.net/forumdisplay.php?f=100 http://forum.codecall.net/showthread.php?t=20968 http://forum.codecall.net/forumdisplay.php?f=100 http://forum.codecall.net/showthread.php?t=20932 http://forum.codecall.net/forumdisplay.php?f=41 http://forum.codecall.net/showthread.php?t=20861 http://forum.codecall.net/forumdisplay.php?f=100 http://forum.codecall.net/software-development/ http://forum.codecall.net/web-development-forum/ http://forum.codecall.net/website-promotion/ http://forum.codecall.net/community/ http://forum.codecall.net/general/ http://forum.codecall.net/tech-support/ http://software.codecall.net/audio_multimedia/ http://software.codecall.net/business/ http://software.codecall.net/communications/ http://software.codecall.net/desktop/ http://software.codecall.net/education/ http://software.codecall.net/games_entertainment/ http://software.codecall.net/graphics/ http://software.codecall.net/home_hobby/ http://software.codecall.net/network__internet/ http://software.codecall.net/security/ http://software.codecall.net/servers/ http://software.codecall.net/development/ http://software.codecall.net/system_utilities/ http://software.codecall.net/web_development/ http://software.codecall.net/unsorted/ http://www.webmarketingexperts.com.au http://www.javaforstudents.co.uk/ http://www.algolist.net http://www.asciibin.com http://www.amphosted.com/billing/aff.php?aff=010 http://www.unseenbattle.com http://freelance.codecall.net/item/viewitem/75 http://freelance.codecall.net/item/viewitem/75 http://freelance.codecall.net/item/viewitem/74 http://freelance.codecall.net/item/viewitem/74 http://freelance.codecall.net/item/viewitem/73 http://freelance.codecall.net/item/viewitem/73 http://freelance.codecall.net/post http://freelance.codecall.net http://www.codecall.net/ http://www.codecall.net/index.php?option=com_contact&view=category&catid=0 http://www.codecall.net/index.php?view=article&id=92&option=com_content&Itemid=135 http://software.codecall.net/ http://software.codecall.net/edit_program.html http://www.webprofits.com.au/searchengineoptimisation.html http://www.floristone.com/ http://codeconnect.net http://www.freelancerstore.com http://www.kfupm.edu.sa http://www.ionicware.com http://freelance.codecall.net http://www.jordandelozier.com/ http://www.asciibin.com/ http://www.questionbin.com/ http://www.questionbin.com/debates.html
Now the only thing left to do spider the links
and report the results, which is exactly what
the second part of the script does.
In verbose mode it list 6 or 7 lines of data for
each link, so I'm only capturing the Resolving line
(the actual link) and the HTTP request sent line (the result).
Here is the link_result.txt file that the second part creates.
A simple way to verify your site links.Code:Resolving codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 200 OK Resolving forum.codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 200 OK Resolving blog.codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving wiki.codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving freelance.codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving twitter.com... 168.143.161.20 HTTP request sent, awaiting response... 200 OK Resolving www.facebook.com... 69.63.181.12 HTTP request sent, awaiting response... 302 Found HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving software.codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving www.webmarketingexperts.com.au... 125.214.72.59 HTTP request sent, awaiting response... 200 OK Resolving www.javaforstudents.co.uk... 69.147.246.162 HTTP request sent, awaiting response... 200 OK Resolving www.algolist.net... 174.132.156.99 HTTP request sent, awaiting response... 200 OK Resolving www.asciibin.com... 199.71.214.230 HTTP request sent, awaiting response... 200 OK Resolving www.amphosted.com... 67.220.197.212 HTTP request sent, awaiting response... 301 Moved Permanently Resolving account.amphosted.com... 67.220.197.212 HTTP request sent, awaiting response... 302 Moved Temporarily Resolving www.unseenbattle.com... 205.178.189.131 HTTP request sent, awaiting response... 301 Moved Permanently HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving www.codecall.net... 199.71.213.27 HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving www.webprofits.com.au... 209.235.208.34 HTTP request sent, awaiting response... 200 OK Resolving www.floristone.com... 204.15.131.12 HTTP request sent, awaiting response... 200 OK Resolving codeconnect.net... 174.36.6.243 HTTP request sent, awaiting response... 200 OK Resolving www.freelancerstore.com... 193.200.80.195 HTTP request sent, awaiting response... 200 OK Resolving www.kfupm.edu.sa... 212.26.1.169 HTTP request sent, awaiting response... 200 OK Resolving www.ionicware.com... 199.71.214.230 HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving www.jordandelozier.com... 199.71.214.230 HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK Resolving www.questionbin.com... 199.71.214.230 HTTP request sent, awaiting response... 200 OK HTTP request sent, awaiting response... 200 OK
The script could be easily modified to descend through directories
and collect all links from the entire site or even multiple domains,
but (IMHO), you get a lot of functionality for only 2 lines of code.
If you haven't done any Shell Scripting,
give it a try, it's FUN.
Last edited by debtboy; 10-04-2009 at 05:41 PM.
Wow, very cool and useful. You could actually use that to generate XML sitemaps.
+rep. Definitely +rep.
sudo rm -rf /
There are currently 1 users browsing this thread. (0 members and 1 guests)
Posting Permissions
LinkBack URL
About LinkBacks
Reply With Quote
Bookmarks