How to Configure Robots.txt and Meta Robots for Magento 2

Robots.txt is a text file that instructs the web robots, i.e., search engines which pages to crawl and which not! Meta robots are a way for webmasters to offer search engines with details about their stores.

Attachment 14153


Read about:

1. Importance of Robots.txt file in Magento 2

2. Steps to create Magento 2 Robots.txt

3. Meta robots tags

4. The recommended default settings for meta robots tags

5. Magento 2 robots.txt file examples

6. Common Web crawlersVisit

How to Configure Robots.txt and Meta Robots for Magento 2

Following strides to Configure robots.txt in Magento 2

1. On the Admin board, click Stores . In the Settings segment, select Configuration .

2. Select Design under General in the board on the left.

3. Open the Search Engine Robots segment, and proceed with following: ...

3. Whenever complete, click Save Config .