Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers ...

About /robots.txt In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robo...

robots.txt generator designed by an SEO for public use. Includes tutorial.

User-agent: * Disallow: /search Disallow: /groups Disallow: /images Disallow: /catalogs Allow: /catalogs/about Disallow: /catalogues Disallow: /news

This file must be accessible via HTTP on the local URL "/robots.txt". The contents of this file are specified below. This approach was chose...

Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.