Robots.txt Generator

Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

The Robots.txt Generator is a convenient web-based tool that allows users to create a robots.txt file with a single click. This free tool is accessible online and as a Google Chrome extension. Users can generate two types of robots.txt files: the standard version and the mobile version.

No registration or login is required to use the generator, and there are no usage limits. The Robots.txt Generator helps you create files that provide search engine crawlers with instructions on which content to avoid indexing or crawling.

The tool is user-friendly and intuitive. Simply enter the URL of your website and click "generate" to download the robots.txt file, which you can then upload to your server.

Key features of the Robots.txt Generator include:

1. Free and Easy to Use: Create your robots.txt file in just a few minutes.
2. No Registration Needed: Use the tool without any signup process.
3. Unlimited Usage: Generate as many files as you need, with no daily or yearly limits.
4. Flexible Options: Choose between standard and mobile versions of the robots.txt file.

How to Create a Robots.txt File

1. Select Search Engines: Choose the search engines whose robots you want to prevent from accessing certain pages of your site.
2. Specify Directories and Pages: Identify the directories and pages you don't want to be indexed.
3. Generate and Upload: Let the generator compile your file, then upload it to your website via FTP or save it to your computer.

Robots.txt Generator is the ideal solution for creating and updating your robots.txt file to ensure your website content is indexed according to your preferences.