Magento Expert Forum - Improve your Magento experience

Results 1 to 16 of 16

What is robots.txt used for?

  1. #1

  2. #2
    Junior Member
    Join Date
    Mar 2016
    Posts
    95
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robots file tells search engines about websites page, which one is allowed or not.

  3. #3
    Junior Member
    Join Date
    Mar 2015
    Posts
    295
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Default

    The robots.txt file is a simple method of essentially easing the process for the spiders to return the most relevant search results.This also increases spiderability for the search engines.

  4. #4
    Junior Member
    Join Date
    Aug 2015
    Posts
    93
    Thanks
    6
    Thanked 6 Times in 6 Posts

    Default

    In simple words, Robots.txt is a text file which you put on your site root folder to tell search engine bot which pages you would like them not to crawl.

  5. #5
    Junior Member
    Join Date
    Feb 2016
    Posts
    190
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    Robots.txt helps prevent the webpages you don't want search engine to crawl or indexed. If you don't want your page or information private than you can use it.

  6. #6
    Junior Member
    Join Date
    Jan 2015
    Location
    Delhi
    Posts
    106
    Thanks
    1
    Thanked 6 Times in 5 Posts

    Default

    Robot.txt is a user agent provided by you in your site for google crawler for follow or nofollow your website Page

  7. #7
    Junior Member
    Join Date
    Mar 2016
    Posts
    209
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    Robots.txt file to give instructions about their website to web robots is allowed or not

  8. #8
    Junior Member
    Join Date
    Sep 2015
    Location
    India
    Posts
    52
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Robots is set of instruction for spiders.

  9. #9
    Junior Member
    Join Date
    Apr 2016
    Posts
    44
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robots.txt is a such file which helps URL being not crawled.

  10. #10
    Junior Member
    Join Date
    Mar 2016
    Location
    Mumbai, India
    Posts
    61
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by danielnash View Post
    I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

    Please guys clear it.
    Robots.txt file is placed in the root of the domain. Robots when crawl website they look for Robots.txt file to know which pages and folder of the website is allowed for crawling and which are blocked. It is recommended to mention xml sitemap at the end of robots.txt file.

  11. #11
    Junior Member
    Join Date
    Feb 2015
    Posts
    55
    Thanks
    1
    Thanked 2 Times in 2 Posts

    Default

    robots.txt is use for control the website indexing and cache method. if you want to search engine bot do not index my particular web pages then you can block that web pages from search engine bot.

  12. #12
    Junior Member
    Join Date
    Mar 2016
    Posts
    168
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Thanks, for sharing this post.

  13. #13
    Junior Member
    Join Date
    Apr 2016
    Location
    Delhi
    Posts
    112
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Quote Originally Posted by danielnash View Post
    I learnt here about sitemap.xml file. Now I'm little bit confused for what is robots.txt used for?

    Please guys clear it.
    Robots.txt is a text file where you can suggested to google which pages should be index or not. Robots.txt file is most important think for website.

  14. #14
    New member
    Join Date
    May 2016
    Location
    Jalandhar, Punjab, India
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    It is important file. It must in every website. It contains instructions for robots.

  15. #15
    Junior Member
    Join Date
    Feb 2016
    Posts
    42
    Thanks
    1
    Thanked 2 Times in 2 Posts

    Default

    Robots.txt give instruction to crawler which page to crawl or not.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •