DelphiFAQ Home Search:

Using robots.txt to block spiders crawling your web site

 

comments27 comments. Current rating: 3 stars (4 votes). Leave comments and/ or rate it.

'Robots.txt' is a plain text file that through its name has special meaning to most decent robots on the web. By defining a few rules in this text file instruct robots to not crawl and index certain files or directories within your site.

If you do not want Google to crawl your site's /pictures folder, you can protect this folder from Google's crawler.

The following gives a few examples how to write a robots.txt file. It has to be placed in the www root directory of your server. On Linux boxes, this is typically /var/www/html.

The following example shows several versions of robots.txt files, separated by a line.


; block Google's image crawler completely User-agent: Googlebot-Image Disallow: /
; block all spiders and bots from those 2 directories User-agent: * Disallow: /cgi-bin/ Disallow: /pictures/
; allow Googlebot to access everything except /cgi-bin ; and all other bots can access nothing ; finally allow ia_archive (alexa.com) to access everything! User-agent: * Disallow: / User-agent: Googlebot Disallow: /cgi-bin/ User-agent: ia_archiver Allow: /

Content-type: text/html

Comments:

You are on page 2 of 2, other pages: 1 [2]
2017-03-16, 09:38:37
Brayden from Frankfurt Am Main, Germany  
rating
Nice weblog right here! Additionally your site so much up very fast!
What web host are you the usage of? Can I get your affiliate hyperlink on your
host? I wish my website loaded up as fast as yours lol
2017-08-21, 04:51:50
Arleen from Taipei, Taiwan  
rating
I have been surfing online greater than three hours nowadays,
yet I never found any fascinating article like yours. It's pretty worth sufficient for me.
In my view, if all website owners and bloggers made excellent content material as you probably did, the
internet can be a lot more useful than ever before.
You are on page 2 of 2, other pages: 1 [2]

 

 

NEW: Optional: Register   Login
Email address (not necessary):

Rate as
Hide my email when showing my comment.
Please notify me once a day about new comments on this topic.
Please provide a valid email address if you select this option, or post under a registered account.
 

Show city and country
Show country only
Hide my location
You can mark text as 'quoted' by putting [quote] .. [/quote] around it.
Please type in the code:

Please do not post inappropriate pictures. Inappropriate pictures include pictures of minors and nudity.
The owner of this web site reserves the right to delete such material.

photo Add a picture: