Magento Expert Forum - Improve your Magento experience

Results 1 to 6 of 6

Is it necessary to use Robots.txt?

  1. #1

  2. #2
    Junior Member
    Join Date
    Mar 2016
    Posts
    181
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robots.txt is text file inserted into the website to contain instructions for search engine robots. The file lists web pages that are allowed and disallowed from search engine crawling. They help to control the search engine robots in your website.

  3. #3
    Junior Member
    Join Date
    Mar 2015
    Posts
    298
    Thanks
    0
    Thanked 3 Times in 3 Posts

    Default

    Yes, Robots helps to avoid the secured webpages crawl by search engines and search engine are not show in results.

  4. #4
    Junior Member
    Join Date
    Jun 2016
    Posts
    32
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    robots.txt is very important for any website to control from search engine bot

  5. #5
    Junior Member
    Join Date
    Jul 2016
    Posts
    100
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Yes, Robots text is helps to avoid the protected webpages crawl by search engines and search engine are not show in results.

  6. #6
    Junior Member
    Join Date
    Jul 2016
    Posts
    100
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Yes Robots.txt is text file put into the website to contain instructions for search engine robots. The file lists web pages that are allowed and disallowed from search engine crawling. They help to control the search engine robots in your website.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •