Magento Expert Forum - Improve your Magento experience

Results 1 to 10 of 10

What is robots.txt?

  1. #1

  2. #2

  3. #3
    Junior Member
    Join Date
    Jul 2018
    Posts
    289
    Thanks
    4
    Thanked 0 Times in 0 Posts

    Default

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  4. #4
    Junior Member
    Join Date
    Feb 2016
    Posts
    190
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    The robots.txt is a simple text file in your web site that notifies search engine bots how to crawl and index website or web pages.

  5. #5
    Junior Member
    Join Date
    Jul 2018
    Posts
    366
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl.

  6. #6
    Junior Member
    Join Date
    Sep 2018
    Location
    CA
    Posts
    140
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Robot.txt file instruct to search engines for which pages, posts should be indexed OR not.

  7. #7
    Junior Member
    Join Date
    Sep 2018
    Location
    United Kingdom
    Posts
    445
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.
    Automatic Driving Lessons Birmingham|Automatic Driving Lessons Redditch|Automatic driving lessons Wolverhampton|Segway for Sale|hoverboards|hoverboard seat|seat for hoverboard

  8. #8
    Junior Member
    Join Date
    Aug 2018
    Location
    Ludhiana
    Posts
    72
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    robots.txt file lives at the root of your site. A robots.txt file tells search engine crawlers which pages or files the crawler can or can't request from your site.

  9. #9
    Junior Member
    Join Date
    Jan 2018
    Location
    Bangalore
    Posts
    87
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. The slash after “Disallow” tells the robot to not visit any pages on the site.

  10. #10
    Junior Member
    Join Date
    Jul 2019
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl.Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •