How to Use a Robots.txt to Control Your Website?

A robots.txt file is a simple text file that is used to tell the search engines (robots) of what should not be visited and indexed. However, you can not tell search engines of what to do. A robots.txt should be placed in your root directory of your web server. Each website should only have one robots.txt file. The main purpose of having a robots.txt is to save your bandwidth from your web server, to give you a very basic of protection, to clean up your logs, and to prevent spams and penalties related to duplicated contents.

If you want the search engines to visit and index your site, then you need to use the following instructions to tell them to do so or some people just leave it blank:

User-Agent: * //Here “*” means all robots Disallow:

If you do not want the search engines to visit and index your site at all, then you have to use the following instructions to tell them not to do so:

User-Agent: * //Here “*” means all robots Disallow: /

Can you see the difference here, the slash after Disallow: makes the opposite meaning.

If you only want the search engines not to visit and index certain pages, then you have to list those pages for them not to do so. See an example below:

User-agent: * Disallow: /administrator/ Disallow: /cache/ Disallow: /editor/ Disallow: /cgi-bin/ Disallow: /images/ Disallow: /media/ Disallow: /about/

Please also note that for the entire website, you may want to use a robots.txt to control it. However, for a single web page, you may want to use a robots meta tag to limit it being visited and indexed; and for a single link, you may consider using “no follow” attribute to prevent search engines to index that link.

Everybody can access your robots.txt file by typing http://yoursitename.com/robots.txt; so you should not list your secret directories in your robots.txt file because some ill-behaved people use some robots to target your robots.txt file for harvesting this kinds of information.

Robots.txt Checker is an useful tool that you can use to check whether your robots.txt is validated or not.

Related articles
Using The Robots Meta Tag
Preventing Comment Spams

SEO Tips For Your Blogs

The topic of SEO (Search Engine Optimization) has been hot and it has been discussed all over the Internet. Still it is worth mentioning here. Why do you want to optimize your blog/website for search engines? How can you optimize your blog/website for SEO? You may find the answers to these questions in the following articles:

• Free SEO Tips for WordPress – If you search in Google or Yahoo about SEO tips, there are countless information out there. Optimizing your blog/website using these tips here.

• Search Engine Optimization – Find out how to optimize your blog by adopting the methods offered by Brad Callen, who is an expert in SEO field. You can download his Search Engine Optimization Make Easy from this site.

• Top SEO Tips for WordPress Blog – Top SEO Tips for your WordPress Blog. Very useful tips for beginners and even advanced bloggers. Especially for those who are using WordPress software.

• SEO Secrets to Writing the Perfect Blog Post – The best SEO tips and tricks revealed by this article. You can use the complete list of all the SEO tips and tricks to generate more traffic to your blog/website.

• 20 Practical SEO Tips to Super-Charge Your WordPress Blog! – Use these 20 prastical SEO tips to supercharge more traffic to your blog/website. Great tips for WordPress bloggers.

• Search Engine Optimization for Blogs – SEO – Uncovered the secrets of SEO by applying offsite and onsite techniques. Leap over there to see what Darren Rowse is talking about.

• ^new SEO Tips for Your Blog – This article talks about the basics of SEO. Then It goes to talk about more advanced stuff such as checking your broken links, building a sitemap for your blog/website, and using a robots.txt wisely.

• ^new SEO Tips – These Will Make A Massive Difference – Drive more quality traffic to your blog/website with these SEO tips listing in this article. Mostly people find your blog/website by search engines using keywords. So learn how to optimize your blog/website for search engines.

By now you should know why optimizing your blog/website is so important. Hope the above links help you optimize your blog/website efficiently. If you have any opinions about this topic, why not talk about it in comment section?