The /robots.txt checker can check your site's /robots.txt file and meta tags. The IP Lookup can help find out more about what robots are visiting you.

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders a...

# robots.txt for http://www.example.com/ User-agent: * Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Cybermapper knows where to...

robots.txt generator designed by an SEO for public use. Includes tutorial.

Information on using the robots.txt file to keep web crawlers, spiders and robots from indexing certain sections of a site.

User-agent: * Crawl-delay: 10

Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.