I am writing a script that follows links from a page and need to keep track of the links that have already been visited so i dont visit them again. I tried using a global array but am having trouble using my recursive method. For example:
http://example.com/foo will have links to
http://example1.com/foo1
http://example2.com/bar1
and these links may themselves contain http://example.com/foo as a link which i dont want to follow as it has already been traversed.
Any ideas would be greatly appreciated.