Magento Expert Forum - Improve your Magento experience

Results 1 to 8 of 8

What is the Different Between Sitemap and Robots.txt?

  1. #1

  2. #2
    Junior Member
    Join Date
    Mar 2015
    Posts
    295
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Default

    A site map is like the table of contents for your site... It provides search engines with a direct path to each page of your site, which favors quick indexing for all the pages.

    On the other hand, robots.txt is typically used to tell search engines to exclude from the index a specific page or folder of your site.

  3. #3
    Junior Member
    Join Date
    Mar 2016
    Posts
    95
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    the sitemap is a way for the bots to index unlinked pages or pages linked with javascript - and to make a list of pages quickly

    if you put a page on your sitemap but then ban it with the robots.txt the bots should NOT index it still... bit of a odd thing to do though - you'd usually not add it to the site map if you dont want it indexed too.

  4. #4
    Junior Member Ryna Lagger's Avatar
    Join Date
    Mar 2016
    Posts
    242
    Thanks
    19
    Thanked 13 Times in 11 Posts

    Default

    robots.txt is used to block the search engine and other crawlers reading the data from a website. sitemap.xml is to provide the website page content structure to search engines and other crawlers visiting website.

  5. #5
    Junior Member
    Join Date
    Feb 2016
    Posts
    93
    Thanks
    9
    Thanked 0 Times in 0 Posts

    Default

    Sitemap is provides website url to indexing for all website pages to search engine. Robots.txt is used to block specific pages for search engine.

  6. #6
    Junior Member
    Join Date
    Mar 2016
    Posts
    209
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    Quote Originally Posted by RichardWashington View Post
    Sitemap is provides website url to indexing for all website pages to search engine. Robots.txt is used to block specific pages for search engine.
    Thanks for giving bro

  7. #7
    Junior Member
    Join Date
    Jul 2018
    Posts
    366
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    The purpose of a Sitemap is to indicate the search engines of all the webpages that it should crawl on your website. Robots.txt: A robots.txt file is created to signify the search engines which pages to crawl and which pages to omit.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •