Magento Expert Forum - Improve your Magento experience

Results 1 to 5 of 5

What is robots.txt?

  1. #1

  2. #2
    Junior Member kajal's Avatar
    Join Date
    Sep 2014
    Location
    Bangalore
    Posts
    1,446
    Thanks
    0
    Thanked 20 Times in 20 Posts

    Default

    Hi Friends,
    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.


    Magento web design bangalore | website developer in bangalore | Php Web Development Company Bangalore
    Last edited by kajal; 14-11-2019 at 04:05 AM.

  3. #3
    Junior Member
    Join Date
    Jul 2019
    Posts
    418
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    The robots exclusion protocol (REP), or robots.txt is a text file webmasters create to instruct robots (typically search engine robots) how to crawl and index pages on their website .

  4. #4
    Junior Member
    Join Date
    Jul 2018
    Posts
    625
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”).

  5. #5
    Junior Member
    Join Date
    Jul 2019
    Location
    Kemp House 152-160 City Road, London
    Posts
    246
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Robot.txt is a file that pastes in the code to stop the google crawler to crawl the web site's article or website's page

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •