Magento Expert Forum - Improve your Magento experience

Results 1 to 11 of 11

What is robots.txt?

  1. #1

  2. #2
    Junior Member kajal's Avatar
    Join Date
    Sep 2014
    Location
    Bangalore
    Posts
    1,446
    Thanks
    0
    Thanked 20 Times in 20 Posts

    Default

    Hi Friends,

    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  3. #3
    Junior Member
    Join Date
    Jul 2019
    Posts
    418
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    A robots. txt file tells search engines where they can and can't go on your site. Primarily, it lists all the content you want to lock away from search engines like Google. You can also tell some search engines (not Google) how they can crawl allowed content.

  4. #4
    Junior Member
    Join Date
    Jul 2019
    Location
    Kemp House 152-160 City Road, London
    Posts
    246
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robot.txt are the files that we add in our code of website to tells the search engine to do not crawl or a section. In other words, we can say that we use robot.txt files to stop the search engine for crawling

    SEO Services Company
    Last edited by rickylarson; 07-04-2020 at 09:13 AM.

  5. #5

  6. #6
    Junior Member
    Join Date
    Jul 2018
    Posts
    563
    Thanks
    6
    Thanked 3 Times in 3 Posts

    Default

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots.

  7. #7

  8. #8

  9. #9
    Junior Member
    Join Date
    Sep 2019
    Location
    US
    Posts
    235
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl

  10. #10
    Junior Member
    Join Date
    Jul 2019
    Location
    Ludhiana
    Posts
    98
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robots.txt is a text file webmasters create to instruct web robots ( search engine robots ) which pages on your website to crawl or not to crawl.

  11. #11
    Junior Member
    Join Date
    Feb 2015
    Posts
    316
    Thanks
    0
    Thanked 3 Times in 3 Posts

    Default

    The robots.txt file is primarily used to specify which parts of your website should be crawled by spiders or web crawlers. It can specify different rules for different spiders.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •