Why the robots.txt file is important ??
Printable View
Why the robots.txt file is important ??
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
robots.text file that tells web robots which pages on your site to crawl. It also tells web robots which pages not to crawl.
The slash after “Disallow” tells the robot to not visit any pages on the site.
Do one thing, don't keep robots.txt for your website and you will get why it is important! ;)Quote:
Originally Posted by Skymedia
A robots.txt file is a file at the root of your site that indicates some parts of your site you don’t want accessed by search engine crawlers.
it's really not. only if you use it and get it wrongQuote:
Originally Posted by Skymedia
so just don't use it ;p
Robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots which pages to crawl & which not.
Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to.
It is very important for indexing a website. We can control robots (search engine crawler) through it. Suppose, I want to block some page to the indexing of a website. Then I can do it through it.
Robots txt is a text file that tells web robots which pages on your site to crawl in search engine.
The Importance of a Robots.txt File for Your SEO. Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. ... Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.