What should be in robots txt file?

What should be in robots txt file?

mention user-agents,
allow all pages to index or disallow all
or disallow few and allow rest!

You can create robots.txt file as per your requirements.

Hi Friends,
The robots. txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

The robots. txt file tells bots which pages on your site you want to be crawled and indexed, and which you don't. You should include a list of Allow and Deny rules paired with the urls.

We write search engine instruction in Robots.txt for search engine indexing.