Magento Expert Forum - Improve your Magento experience

Results 1 to 9 of 9

What is the limit of a robot.txt file?

  1. #1

  2. #2
    Junior Member
    Join Date
    Jul 2018
    Posts
    52
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler.

  3. #3

  4. #4
    Junior Member
    Join Date
    Sep 2018
    Location
    United Kingdom
    Posts
    31
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.text file in terms of file size 500 KB as Vi pin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

  5. #5
    Junior Member
    Join Date
    Jun 2015
    Posts
    868
    Thanks
    0
    Thanked 9 Times in 9 Posts

    Default

    There is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".

  6. #6
    Junior Member
    Join Date
    Sep 2018
    Location
    Delhi
    Posts
    75
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by sudhakar View Post
    What is the limit of a robot.txt file?
    The limit of a robot.txt file IS 500KB

  7. #7
    Junior Member Simi123's Avatar
    Join Date
    May 2018
    Posts
    121
    Thanks
    0
    Thanked 3 Times in 3 Posts

    Default

    I am think that there is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".

  8. #8
    Junior Member
    Join Date
    Jul 2018
    Posts
    184
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by sudhakar View Post
    What is the limit of a robot.txt file?
    Limitations of robots.txt in which sense?

  9. #9
    Junior Member
    Join Date
    Jan 2018
    Posts
    60
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •