Robots.txt – SEO WIKI
The robots.txt file is a text file that contains special instructions for search robots.
This file is located in the root directory of the site. In robots.txt, you can set the rules for search engine robots and prevent them from indexing certain pages, register the correct “mirror” of the domain, specify the interval between downloading documents from the site, and so on.
Any search robot when it reaches the site first of all determines the existence of the robots.txt file and follows the instructions that are in it.