robots.txt generator

Create your robots.txt file to control what Google and other search engine and AI bots can crawl on your WordPress site. Once generated, upload it to your web server root folder.

Block crawling of common folders

Crawl rate control (Crawl-delay)

⚠ Google ignores this directive, but other bots like Bing, Yandex and Baidu do respect it. Useful for limiting server resource usage.
Includes by default: CompSpyBot, Curious George, Ezooms, Screaming Frog, rogerbot, yacybot and 30+ bots that typically provide no SEO value but consume server resources.

⚠ Warning for online stores: If you block AI bots, your products may not appear in ChatGPT, Gemini or other AI recommendations when users search for purchase suggestions. Note that some bots do not respect robots.txt.

Select the AI bots you want to block:

Generated code

# The generated code will appear here

robots.txt validator

Paste your robots.txt content here to validate its syntax and detect potential issues.

You might also like

Config file

security.txt Generator

Create your security.txt file following the RFC 9116 standard.

 Config file

humans.txt Generator

Create your humans.txt file to credit the team behind your site.

 Config file

Theme Readme Generator

Generate your theme readme.txt valid for WordPress.org.