Magento Expert Forum - Improve your Magento experience

Results 1 to 10 of 10

Is it necessary to use Robots.txt?

  1. #1

  2. #2
    Junior Member
    Join Date
    Dec 2015
    Posts
    74
    Thanks
    2
    Thanked 7 Times in 6 Posts

    Default

    It is not important use robots. txt file but it is advisable to use it, if there is any page on your websites which you don't want search engine to crawl. Robots.txt also improves better indexing for the website.

  3. #3

  4. #4

  5. #5
    Junior Member
    Join Date
    Apr 2016
    Posts
    110
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    to control the google bot.. you should use the robotx.txt file

  6. #6
    Junior Member orientation's Avatar
    Join Date
    Feb 2016
    Location
    Bangalore
    Posts
    93
    Thanks
    2
    Thanked 0 Times in 0 Posts

    Default

    it is not compulsory but it is very use full for web site .

  7. #7
    Junior Member
    Join Date
    Apr 2016
    Location
    Delhi
    Posts
    117
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Quote Originally Posted by EmilyPete View Post
    Hi,

    My question is, Is it necessary to use Robots.txt?
    not necessary , but robots.txt to help to unnecessary page not index.

  8. #8
    Junior Member
    Join Date
    May 2016
    Location
    Punjab India
    Posts
    67
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Iis is essential.

  9. #9
    New member comercepundit's Avatar
    Join Date
    May 2016
    Location
    Atlanta, Georgia
    Posts
    9
    Thanks
    3
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by astrologerhome View Post
    Iis is essential.
    Not exactly ... but yes robots.txt will makes SEO more meaningful as you will control crawling and indexing of your website through google.

  10. #10
    New member
    Join Date
    Apr 2016
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    No, robots.txt file is not compulsory. If you want to hide or block any content or page on website then you must have to use robot.txt.file.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •