Magento Expert Forum - Improve your Magento experience

Results 1 to 15 of 15

Is it necessary to use Robots.txt file?

  1. #1

  2. #2
    Junior Member
    Join Date
    Sep 2016
    Posts
    228
    Thanks
    0
    Thanked 3 Times in 3 Posts

    Default

    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.


    Best Dentist in Shalimar Bagh | Book Tender Notice Ad in Newspaper
    Last edited by vishnu; 16-11-2020 at 09:29 AM.

  3. #3
    Member
    Join Date
    Jun 2017
    Location
    jalandhar
    Posts
    34
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    A robots.txt file gives instructions to web robots about the pages the website owner doesn’t wish to be ‘crawled’. For instance, if you didn’t want your images to be listed by Google and other search engines, you’d block them using your robots.txt file.

  4. #4
    Junior Member
    Join Date
    Jan 2018
    Location
    Agra, U.P.
    Posts
    109
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Not necessary but if you will use then it will no harm to your website.

  5. #5
    Junior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    425
    Thanks
    6
    Thanked 5 Times in 5 Posts

    Default

    Quote Originally Posted by latestnewsheadlines View Post
    Hello friends- Is it necessary to use Robots.txt file? It can be affect on website ranking in search engine.
    I think you should read the use of the robots.txt file to clear your doubts.
    Must use the robots.txt file to give instructions to the web crawlers.

  6. #6
    Junior Member
    Join Date
    Feb 2018
    Posts
    33
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    yes it is necessary as it tells crawler which part need to index or not

  7. #7
    Junior Member
    Join Date
    Feb 2018
    Location
    Singapore
    Posts
    122
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    robots.txt is used for this

  8. #8
    Junior Member
    Join Date
    Oct 2017
    Posts
    40
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Robots.txt file let the search engines know which page to crawl & which page to Ignore.It gives instructions on crawling the website.

  9. #9
    Junior Member
    Join Date
    May 2016
    Location
    Punjab India
    Posts
    135
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    It makes your website SEO friendly and google spider can easy to crawl your website with the help of robot file

  10. #10
    Junior Member
    Join Date
    Jul 2018
    Posts
    366
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    You are in SEO & doubting on the use of robots.txt! Not done!

  11. #11
    Junior Member
    Join Date
    Feb 2018
    Posts
    113
    Thanks
    0
    Thanked 2 Times in 2 Posts

    Default

    Robots.txt files are useful if you want search engines not to index:

    Duplicate pages on your website
    Internal search results pages
    Certain areas of your website or a whole website
    Certain files on your website such as images and PDFs
    Your sitemap


    Book Ad in Newspaper | Newspaper Ad Agency in Lucknow
    Last edited by trzy; 16-11-2020 at 09:50 AM.

  12. #12
    Junior Member
    Join Date
    Jul 2019
    Posts
    418
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    I agree with john's answer that @op should read the use of robots.txt to clear the doubts!!

  13. #13
    Junior Member
    Join Date
    Sep 2019
    Location
    US
    Posts
    349
    Thanks
    4
    Thanked 2 Times in 2 Posts

    Default

    One of the best uses of the robots. txt file is to maximize search engines' crawl budgets by telling them to not crawl the parts of your site that aren't displayed to the public.

  14. #14
    Junior Member
    Join Date
    Jul 2018
    Posts
    563
    Thanks
    6
    Thanked 3 Times in 3 Posts

    Default

    One of the best uses of robots. txt file is to maximize search engines' crawl budgets by telling them to not crawl the parts of your site that aren't displayed to the public.
    So you should use it.

  15. #15
    Junior Member
    Join Date
    Jul 2019
    Location
    Kemp House 152-160 City Road, London
    Posts
    246
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, it is very necessary for SEO. It is the security of the website.

    SEO Services Company
    Last edited by rickylarson; 13-02-2020 at 09:34 AM.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •