Advanced Robots.txt Generator

Create, customize, and validate your robots.txt file with ease

Generator

Validator

Guide

User Agent Rules

Sitemap Configuration

Sitemap URL

Advanced Options

Crawl-Delay (seconds)

Clean-Param Specify URL parameters that should be ignored by crawlers

Host Directive (Preferred Domain) Only one host directive is allowed per robots.txt

Generated Robots.txt

Robots.txt Validator

Paste your robots.txt content to validate

Robots.txt Guide

What is robots.txt?

The robots.txt file is a text file that tells web robots (typically search engine crawlers) which pages on your site they can request. It's part of the robots exclusion protocol (REP).

Basic Directives

• User-agent: Specifies which crawler the rules apply to (* for all)
• Disallow: Tells crawlers not to access certain paths
• Allow: Overrides Disallow for specific paths (not all crawlers support this)
• Sitemap: Specifies the location of your sitemap(s)

Advanced Directives

• Crawl-delay: Specifies the number of seconds to wait between requests
• Clean-param: Indicates URL parameters that don't change page content
• Host: Specifies the preferred domain (only one allowed)

Best Practices

• Place your robots.txt file in the root directory (e.g., https://example.com/robots.txt)
• Use specific user-agent rules when needed
• Don't use robots.txt to hide sensitive information (use authentication instead)
• Test your robots.txt with Google Search Console's robots.txt tester

Copied to clipboard!