robots.txt generator
robots.txt is the file that tells automated visitors which pages they may read. Assistants send a crawler before they will repeat anything from a site. This writes the lines that let them in.
This is 1 of 11 things we check.
See all of them