Magento Expert Forum - Improve your Magento experience

Results 1 to 5 of 5

What should be in robots txt file?

  1. #1

  2. #2
    Junior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    425
    Thanks
    6
    Thanked 5 Times in 5 Posts

    Default

    mention user-agents,
    allow all pages to index or disallow all
    or disallow few and allow rest!

    You can create robots.txt file as per your requirements.

  3. #3
    Junior Member kajal's Avatar
    Join Date
    Sep 2014
    Location
    Bangalore
    Posts
    1,446
    Thanks
    0
    Thanked 20 Times in 20 Posts

    Default

    Hi Friends,
    The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

  4. #4
    Junior Member
    Join Date
    Jul 2019
    Posts
    418
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    The robots. txt file tells bots which pages on your site you want to be crawled and indexed, and which you don't. You should include a list of Allow and Deny rules paired with the urls.

  5. #5
    Junior Member
    Join Date
    Sep 2018
    Location
    CA
    Posts
    192
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    We write search engine instruction in Robots.txt for search engine indexing.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •