Closed Thread
Results 1 to 4 of 4

Thread: Robots.txt URL removal

  1. #1
    Join Date
    Aug 2006
    Posts
    11,209
    Blog Entries
    6
    Rep Power
    101

    Robots.txt URL removal

    I had my website using dynamic urls and it got indexed from Google, now I changed it to SEF links, and I want to remove the dynamic URLS, because Google did not remove them although it indexed my website another 2 or 3 times after that... and I read it should be made via robots.txt, but how? will the following work?:

    Code:
    User-agent: *
    Disallow: /index.php?cat=1
    Disallow: /index.php?cat=7
    Disallow: /index.php?cat=2
    Disallow: /index.php?cat=4
    Disallow: /index.php?cat=3
    Disallow: /cool.php?url=flash/SammyCamilleri.swf
    Disallow: /review.php?sid=2&recat=4
    Disallow: /review.php?sid=1&recat=3
    or I should use

    Disallow: /index.php?

    The google.com/robots.txt uses that code.but im not sure what to do!

  2. CODECALL Circuit advertisement
    Join Date
    Always
    Posts
    Many

     
  3. #2
    v0id is offline Retired
    Join Date
    Apr 2007
    Posts
    2,937
    Blog Entries
    3
    Rep Power
    42
    You can simply use Disallow: ./index.php.

  4. #3
    Join Date
    Aug 2006
    Posts
    11,209
    Blog Entries
    6
    Rep Power
    101
    Well but that means that the bot will not crawl my website! But I want it to!

  5. #4
    v0id is offline Retired
    Join Date
    Apr 2007
    Posts
    2,937
    Blog Entries
    3
    Rep Power
    42
    If you want the Googlebot to crawl through some pages, you simply don't but them in your robots.txt file. If you want the Googlebot to crawl through index.php then don't put it in your robots.txt - or do an:
    Code:
    Allow: index.php
    You cannot allow - or disallow pages like index.php?id=X, only the page itself, index.php.

Closed Thread

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Removal of items from cart removes more than I want!
    By Cyberen in forum PHP Development
    Replies: 2
    Last Post: 03-09-2011, 05:03 PM
  2. phpbb forum timestamp removal
    By Brok3n in forum PHP Development
    Replies: 1
    Last Post: 07-02-2010, 08:23 AM
  3. Freshman Guide to Bra Removal
    By mnhaqeel in forum The Lounge
    Replies: 0
    Last Post: 01-16-2007, 11:49 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts