Magento Expert Forum - Improve your Magento experience

Results 1 to 9 of 9

Is it necessary to use Robots.txt file?

  1. #1

  2. #2
    Junior Member
    Join Date
    Sep 2016
    Posts
    181
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.


    Corporate Gifts

  3. #3
    Member
    Join Date
    Jun 2017
    Location
    jalandhar
    Posts
    36
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

  4. #4
    Junior Member
    Join Date
    Jan 2018
    Location
    Agra, U.P.
    Posts
    110
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Not necessary but if you will use then it will no harm to your website.

  5. #5
    Junior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    108
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Quote Originally Posted by latestnewsheadlines View Post
    Hello friends- Is it necessary to use Robots.txt file? It can be affect on website ranking in search engine.
    I think you should read the use of the robots.txt file to clear your doubts.
    Must use the robots.txt file to give instructions to the web crawlers.

  6. #6
    Junior Member
    Join Date
    Feb 2018
    Posts
    33
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    yes it is necessary as it tells crawler which part need to index or not

  7. #7

  8. #8
    Junior Member
    Join Date
    Oct 2017
    Posts
    40
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Robots.txt file let the search engines know which page to crawl & which page to Ignore.It gives instructions on crawling the website.

  9. #9
    Junior Member
    Join Date
    May 2016
    Location
    Punjab India
    Posts
    135
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    It makes your website SEO friendly and google spider can easy to crawl your website with the help of robot file

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •