What is robots.txt for?

What’s the sandbox?
May 31, 2019
What’s a Spamblog?
May 31, 2019

What is robots.txt for?

Robots.txt is a text file you place in the main directory of your website, which tells search engines not to crawl a specific page or set of pages.You should add this file if there are pages on your website that you don’t want to be indexed, although the page title and URL will often still appear in search rankings.

A search result indicating robots.txt is being used will say, “A description for this result is not available because of this site’s robots.txt.”

However, search engines will still crawl around the content you want excluding, meaning your pages can still appear in search results.You only need use robots.txt if you have some content you want excluding from search engines, for whatever reason.

Please note that robots.txt doesn’t hold any legal connotations and search spiders can still crawl the page if they choose to do so, although the leading search providers like Google, Bing, and Yahoo! will generally take notice of the file.

Was this answer helpful ? Yes / No