Robots.txt Generator

Create a robots.txt file to control how search engines crawl your website.

User Agent

Select which bot this rule applies to.

Default Access

Allow All

Disallow All

Optimization Delay (Crawl-delay)

Sitemap URL

Restricted Directories

User-agent: *
Disallow: /cgi-bin/