Robots TXT Toolsti Generator
Fast, accurate and free online robots txt toolsti generator tool running directly in your browser.
-
1Enter data
Enter content, paste text or load a file from disk. -
2Click the button
The tool will immediately process your data in the browser. -
3Get the result
Copy the finished text or save the file to your device.
return "Result ready in 0.1s";
}
# robots.txt generated by Toolsti Robots.txt Generator # Place this file at the root of your domain: /robots.txt # Review rules carefully before publishing. User-agent: * Allow: / Allow: /public/ Allow: /blog/ Disallow: /admin/ Disallow: /login/ Disallow: /private/ Disallow: /search Disallow: /tmp/ Crawl-delay: 10 Host: example.com Sitemap: https://example.com/sitemap.xml Sitemap: https://example.com/sitemap-blog.xml
Rate this tool:
Related tools
Other tools you may find usefulRobots.txt Generator
Robots.txt Generator is a free online tool that allows you to create a valid robots.txt file in seconds - without having to write any code by hand. Just select the crawler, enter the paths and click "Generate". You can copy or download the finished file immediately.
What is the robots.txt file for?
The robots.txt file is a standard mechanism for telling search engine robots (crawlers) which areas of a site they can crawl and which areas they should skip. Thanks to it, you can block access to the admin panel, login pages or temporary directories, and at the same time clearly indicate the paths allowed for indexing. You can also provide sitemap addresses and crawl-delay, which helps protect the server from overload.
How to use the generator?
Select the target crawler (e.g. Googlebot, Bingbot or all crawlers), enter Disallow paths (blocked) and optionally Allow paths (explicitly allowed). If you have a sitemap, please provide its URL. Optionally, set crawl-delay and provide the Host domain. After clicking "Generate robots.txt" you will receive a ready-made file that you can copy to the clipboard or download asrobots.txtand upload to the main directory of your domain.
FAQ
Where to place the robots.txt file?
The robots.txt file must be located in the domain root directory, located athttps://example.com/robots.txt. Google and Bing crawlers automatically check this location before indexing your site.
Does robots.txt block access to the site?
No - the robots.txt file is just a guideline for robots that respect it. It does not provide protection against human access or irresponsible bots. If you want to protect resources, use HTTP authorization or server rules (.htaccess).
What is the difference between Allow and Disallow?
TheDisallow directiveprohibits the crawler from visiting the specified path. TheAllowdirective explicitly allows access to a path, even if the parent Disallow rule blocks it. The most common use of Allow is to make a subset of a locked directory available, e.g. allowing/admin/public/while blocking/admin/.
What is crawl-delay?
Crawl-delay specifies the minimum time (in seconds) between consecutive crawler requests. Helps reduce server load during intensive crawling. Google does not officially support this directive (manage indexing speed in Google Search Console), but Bingbot and other crawlers respect it.