Magento Expert Forum - Improve your Magento experience

Results 1 to 11 of 11

Why the robots.txt file is important ??

  1. #1

  2. #2

  3. #3
    Junior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    425
    Thanks
    6
    Thanked 5 Times in 5 Posts

    Default

    robots.text file that tells web robots which pages on your site to crawl. It also tells web robots which pages not to crawl.
    The slash after “Disallow” tells the robot to not visit any pages on the site.

  4. #4
    Junior Member denverlawyers's Avatar
    Join Date
    May 2018
    Location
    Colorado
    Posts
    204
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by Skymedia View Post
    Why the robots.txt file is important ??
    Do one thing, don't keep robots.txt for your website and you will get why it is important!

  5. #5
    Junior Member kanetailor's Avatar
    Join Date
    Apr 2018
    Posts
    69
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    A robots.txt file is a file at the root of your site that indicates some parts of your site you don’t want accessed by search engine crawlers.

  6. #6
    New member
    Join Date
    Jun 2018
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by Skymedia View Post
    Why the robots.txt file is important ??
    it's really not. only if you use it and get it wrong

    so just don't use it ;p

  7. #7
    Junior Member
    Join Date
    Sep 2017
    Posts
    32
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots which pages to crawl & which not.

  8. #8
    Junior Member
    Join Date
    Jul 2018
    Posts
    563
    Thanks
    6
    Thanked 3 Times in 3 Posts

    Default

    Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to.

  9. #9
    Junior Member
    Join Date
    Jul 2018
    Posts
    41
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    It is very important for indexing a website. We can control robots (search engine crawler) through it. Suppose, I want to block some page to the indexing of a website. Then I can do it through it.

  10. #10
    Junior Member
    Join Date
    Jan 2015
    Posts
    97
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robots txt is a text file that tells web robots which pages on your site to crawl in search engine.

  11. #11
    Junior Member
    Join Date
    Sep 2018
    Location
    United Kingdom
    Posts
    635
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Default

    The Importance of a Robots.txt File for Your SEO. Your Robots.txt file is what tells the search engines which pages to access and index on your website on which pages not to. ... Keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •