Magento Expert Forum - Improve your Magento experience

Results 1 to 5 of 5

What is robots.txt?

  1. #1

  2. #2
    Junior Member kajal's Avatar
    Join Date
    Sep 2014
    Location
    Bangalore
    Posts
    1,446
    Thanks
    0
    Thanked 20 Times in 20 Posts

    Default

    Robots.txt is one way of telling the Search Engine Bots about the web pages on your website which you do not want them to visit.

    Robots.txt is useful for preventing the indexation of the parts of any online content that website owners do not want to display.

  3. #3
    Junior Member
    Join Date
    Jun 2018
    Location
    surat
    Posts
    791
    Thanks
    0
    Thanked 6 Times in 6 Posts

    Default

    Robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl & index pages on their website.

  4. #4
    Junior Member
    Join Date
    Jul 2018
    Posts
    563
    Thanks
    6
    Thanked 3 Times in 3 Posts

    Default

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •