What is robots.txt?
The robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or sections they are allowed to access. It follows the Robots Exclusion Protocol, a standard that has been used since the early days of the web. When a crawler visits your site, it checks for robots.txt first and follows the directives it finds. Properly configuring this file helps you control your crawl budget, protect private areas from being indexed, and guide search engines to your most important content.
How to Use This Tool
Start by selecting a quick preset (Allow All, Block All, Block AI Bots, or Standard) to populate common configurations instantly. Then customize your rules by adding or removing user-agent groups, disallow paths, and allow paths. You can select from a list of common bot names and frequently blocked paths using the dropdown menus. Add a sitemap URL to help crawlers discover your content. The generated robots.txt output updates in real time as you make changes. Copy the result and paste it into a robots.txt file at your domain root.
Common Use Cases
- Blocking admin panels, API endpoints, and private directories from search engines
- Preventing AI training bots (GPTBot, CCBot, Google-Extended) from scraping your content
- Setting crawl-delay directives to reduce server load from aggressive crawlers
- Declaring your sitemap location so search engines can discover all your pages
Why Use a Client-Side Tool?
This generator runs entirely in your browser with no server-side processing. Your site structure and configuration details are never transmitted externally. The output updates instantly as you adjust settings, and the tool works offline. This makes it safe to use even when planning the robots.txt configuration for a site that has not launched yet.
Frequently Asked Questions
Where should I place the robots.txt file?
The robots.txt file must be placed at the root of your domain, accessible at https://yourdomain.com/robots.txt. Search engine crawlers look for it at this exact location. Placing it in a subdirectory will not work.
Does robots.txt prevent pages from appearing in search results?
No. Robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead.
How do I block AI training bots like GPTBot?
Add a separate User-agent block for GPTBot (or other AI bots like CCBot, ChatGPT-User, and Google-Extended) with Disallow: / to block them from your entire site. This tool includes a "Block AI Bots" preset that sets this up automatically.