Protect Your SEO With This Robots.txt Generator

by Stephen Alberts

robots txt generator

A few days ago I found an awesome tool that all of us can use. Whether you’re designing mobile sites, web sites or even just own a site you can use this robots.txt generator. Websites use a robots.txt file to let the crawlers know about their site. The crawler visits your site and looks for the robots.txt file. If that file says not to crawl, the robot will not crawl your site. It’s that simple.

Why wouldn’t I want a robot to crawl my site?
Good question. When building a site you might put filler text as a place holder. It may be duplicate content that you stuffed in there while designing. Or you may have their current site up on your own host while building and testing. You do not want spiders to crawl this temporary information because it can hurt your search engine rankings.

By using a robots.txt generator and creating a disallow statement, the crawlers will stay far far away. The tool I discovered makes this super easy and you’ll have your robots.txt file uploaded in no time.

Using the Robots.txt Generator

To start off visit this site, Robots.txt Generator.
For testing purposes I would disallow all the spiders. To do this select Refused under the Default – All Robots are: tab.
Click Create Robots.txt and it will generate your file.
Copy and paste those few lines into a text file and name it robots.txt. Upload that to your root and you’re all set (make sure it’s in the root!).

*Now if you’re designing a mobile site in a specific directory, then just exclude that directory under Restricted Directories. Keep the Allow selection active in Step 2 and then list the directory path you do not want crawled.

Do I need a robots.txt file to allow search engines to crawl?

It’s a tough call. Some people say to use a simple robots.txt file to let crawlers know about your site, and others say it doesn’t matter. For me I’m listening to what Google says:

They say you don’t need even an empty robots.txt file for your site. Most spammers will ignore your robots.txt file. Google recommends password protecting any content that you don’t want anyone to see or using a robots.txt file to disallow.

How I Used This Tool the Other Day

I actually used this exact tool just a few days ago. I’m developing a site for one of my clients on my server. There site is still live but I have all the current content in my sample site. I created a disallow rule for that one directory. I then uploaded the robots.txt file to my root. Super simple and it protects your clients SEO. Even explain to them what you’re doing and I’ll guarantee they will be impressed that you care about their search engine rankings.