Magento Expert Forum - Improve your Magento experience

Results 1 to 11 of 11

What is the limit of a robot.txt file?

  1. #1

  2. #2
    Junior Member
    Join Date
    Jul 2018
    Posts
    40
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler.

  3. #3
    Junior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    110
    Thanks
    1
    Thanked 1 Time in 1 Post

    Default

    Hope you got your answer!
    So now you can mark this thread as solved!

  4. #4
    Junior Member
    Join Date
    Sep 2018
    Posts
    24
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.text file in terms of file size 500 KB as Vi pin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

  5. #5
    Junior Member
    Join Date
    Jun 2015
    Posts
    855
    Thanks
    0
    Thanked 8 Times in 8 Posts

    Default

    There is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".

  6. #6
    Junior Member
    Join Date
    Sep 2018
    Location
    Delhi
    Posts
    71
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by sudhakar View Post
    What is the limit of a robot.txt file?
    The limit of a robot.txt file IS 500KB

  7. #7
    Junior Member Simi123's Avatar
    Join Date
    May 2018
    Posts
    68
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    I am think that there is no limit but while searching on Google I found one article- in this John Mueller said "Google has a limit of only being able to process up to 500KB of your robots.txt file".

  8. #8
    Junior Member
    Join Date
    Sep 2018
    Location
    Mumbai
    Posts
    12
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Hi

    500 kb is limit of Robots.txt

    Rice Rubber Rolls

  9. #9
    Junior Member
    Join Date
    Jul 2018
    Posts
    152
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Quote Originally Posted by sudhakar View Post
    What is the limit of a robot.txt file?
    Limitations of robots.txt in which sense?

  10. #10
    Junior Member
    Join Date
    Aug 2018
    Posts
    50
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

  11. #11
    Junior Member
    Join Date
    Jan 2018
    Posts
    60
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    Yes, there is a limit to robots.txt file in terms of filesize 500 KB as Vipin points out. A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •