What is a Robots.txt File?
A simple text file placed on the web server that instructs how search engine spiders like Googlebot should see and interact with your webpages.Importance of Robots.txt File
Robots.txt is the fundamental part of how search engines work. The first thing a search engine spider looks at when it is visiting a website is the robots.txt file.Improper usage of the robots.txt file can hurt search ranking. Hence it is important to determine if robots.txt is blocking important files.
How to enable a custom Robots.txt File in Blogger?
Step 1: Go to your Blogspot Blog.Step 2: Navigate to Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots.txt
Step 3: Choose Edit ›› Yes
Step 4: Paste the below robots.txt file code in the box below.
User-agent: Mediapartners-GoogleChange example.blogspot.com accordingly. For example - For this blog, it is amprandom.blogspot.com.
Disallow:
User-agent: *
Disallow: /search
Allow: /
Sitemap: http://example.blogspot.com/sitemap.xml
Step 5: Click on “Save Changes” button.
What is the use of Robots meta tags?
These tags can serve as an easy shortcut for noindex, nofollow, or basically instructing computer bots on how to behave.How to enable a custom Robots Header tags?
Step 1: Go to your Blogspot Blog.Step 2: Navigate to Settings ›› Search Preferences ›› Crawlers and indexing ›› Custom robots header tags
Step 3: Choose Edit ›› Yes
Step 4: Place tick marks on the check boxes as seen in the screenshot below.
Step 5: Click on “Save Changes” button.