Meta name= robots content= noindex nofollow means

Leave a reply to meta name=robots content=noindex, follow

Table of Contents Show

Default Meta robots tag
Meta robots noindex
Meta robots nofollow
Meta robots noarchive
Meta robots noodp
Meta robots noydir
What if page has more than one Meta robots tag
How to block specific search engine bot
Indexation-controlling parameters:
Types of robots meta directives
Meta robots tag
X-robots-tag
SEO best practices with robots meta directives
Improve your technical site health with Moz Pro
Keep learning
Put your skills to work

Local SEO web design services for plumbers, electricians etc… looking for better Google rankings…
© Local SEO Gold Coast

Meta robots tag controls spiders from crawling and indexing your webpage or post. In this post, we will see all types of tags in detail. We can use more than one meta tags in a webpage but its not a good practice because it can cause conflicts. Hence it should occur only once in a webpage.

Default Meta robots tag

If you don’t specify any meta tag in a webpage then by default spider will crawl and index your webpage. you can say that it considers below meta tag by default.

Meta robots noindex

It prevents page from being indexed. It means if you specify below meta tag for a page then that page would not come up in search results. For example: you wouldn’t want your author pages to come up in search results so to prevent it you can use below Meta robots tag in author pages.

Meta robots nofollow

This meta tag prevents spiders to follow any link on the page. If you are a newbie then it may be confusing to you. Let me explain you in detail. Whenever spider( Google bot) finds a link (internal or external) on a page, it lands on that webpage for crawling and indexing, to enforce spider for not following any link on the page, we can use this tag. To understand it better you can refer my post Dofollow vs nofollow.

Meta robots noarchive

You must have seen in search results that whenever you hover on any result it will show a snapshot of that page at the right side of the Google search (see the below sreenshot!!). That we call a cached copy of the webpage. To prevent a page from being cached we can specify below tag.

Meta robots noodp

Prevents Search engine (Google, Yahoo and MSN) to display its own description taken from its directory, instead of you meta description. Using below tag would ensure that Search engine would display your meta description below search results and it would be useful to improve your CTR.

Meta robots noydir

It is same as above NOODP tag, the only difference is that it would be applicable for Yahoo only.

Best Practice to use noodp and noydir: You can use the below tag for all of your webpages-

OR (above tag and below two tags- both are same)

What if page has more than one Meta robots tag

We should use it only once per page and its the best practice, but sometimes because of combination of SEO friendly themes and SEO plugins, we may have multiple instances of such tags. So for such cases lets discuss how spiders (or crawlers) would interpret it.

CASE 1: No conflict values in robots meta tag.

Above two tags would be taken same as: <meta name="robots" content="noindex, nofollow" />

CASE 2: Conflicts value present in robots meta tag.

First instance: <meta name="robots" content="noindex" />

Second instance: <meta name="robots" content="index" />

In this case both values are conflicting as one tell to noindex and other is enforcing spider to index the page. For such cases crawlers considers first instance of conflicts.

So search engine bot would take it as <meta name="robots" content="noindex" /> because noindex encountres before than index one.

How to block specific search engine bot

By specifying meta name as “robots”,you are actually blocking all search engine robots. However if you want to block specific search engine bot then you can specify bot name instead of “robots”

BOT Name:

Google: GOOGLEBOT

Yahoo: SLURP

MSN: MSNBOT

ASK: TEOMA

So, if you want to make your page nofollow for Google bot only then you can use below tag.

Note: In meta name I have given “Googlebot” instead of “robots” to make it applicable for Google search engine bot only.

Reference

Meta robots tag: Google Webmaster blog

Robots meta directives (sometimes called "meta tags") are pieces of code that provide crawlers instructions for how to crawl or index web page content. Whereas robots.txt file directives give bots suggestions for how to crawl a website's pages, robots meta directives provide more firm instructions on how to crawl and index a page's content.

There are two types of robots meta directives: those that are part of the HTML page (like the meta robots tag) and those that the web server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as "noindex" and "nofollow" in the example above) can be used with both meta robots and the x-robots-tag; what differs is how those parameters are communicated to crawlers.

Meta directives give crawlers instructions about how to crawl and index information they find on a specific webpage. If these directives are discovered by bots, their parameters serve as strong suggestions for crawler indexation behavior. But as with robots.txt files, crawlers don’t have to follow your meta directives, so it’s a safe bet that some malicious web robots will ignore your directives.

Below are the parameters that search engine crawlers understand and follow when they’re used in robots meta directives. The parameters are not case-sensitive, but do note that it is possible some search engines may only follow a subset of these parameters or may treat some directives slightly differently.

Indexation-controlling parameters:

Noindex: Tells a search engine not to index a page.
Index: Tells a search engine to index a page. Note that you don’t need to add this meta tag; it’s the default.
Follow: Even if the page isn’t indexed, the crawler should follow all the links on a page and pass equity to the linked pages.
Nofollow: Tells a crawler not to follow any links on a page or pass along any link equity.
Noimageindex: Tells a crawler not to index any images on a page.
None: Equivalent to using both the noindex and nofollow tags simultaneously.
Noarchive: Search engines should not show a cached link to this page on a SERP.
Nocache: Same as noarchive, but only used by Internet Explorer and Firefox.
Nosnippet: Tells a search engine not to show a snippet of this page (i.e. meta description) of this page on a SERP.
Noodyp/noydir [OBSOLETE]: Prevents search engines from using a page’s DMOZ description as the SERP snippet for this page. However, DMOZ was retired in early 2017, making this tag obsolete.
Unavailable_after: Search engines should no longer index this page after a particular date.

Types of robots meta directives

There are two main types of robots meta directives: the meta robots tag and the x-robots-tag. Any parameter that can be used in a meta robots tag can also be specified in an x-robots-tag.

We’ll talk about both the meta robots and x-robots tag directives below.

Meta robots tag

The meta robots tag, commonly known as "meta robots" or colloquially as a "robots tag," is part of a web page’s HTML code and appears as code elements within a web page’s <head> section:

Code sample:

While the general <meta name="robots" content="[PARAMETER]"> tag is standard, you can also provide directives to specific crawlers by replacing the "robots" with the name of a specific user-agent. For example, to target a directive specifically to Googlebot, you’d use the following code:

Want to use more than one directive on a page? As long as they’re targeted to the same "robot" (user-agent), multiple directives can be included in one meta directive – just separate them by commas. Here’s an example:

This tag would tell robots not to index any of the images on a page, follow any of the links, or show a snippet of the page when it appears on a SERP.

If you’re using different meta robots tag directives for different search user-agents, you’ll need to use separate tags for each bot.

X-robots-tag

While the meta robots tag allows you to control indexing behavior at the page level, the x-robots-tag can be included as part of the HTTP header to control indexing of a page as a whole, as well as very specific elements of a page.

While you can use the x-robots-tag to execute all of the same indexation directives as meta robots, the x-robots-tag directive offers significantly more flexibility and functionality that the meta robots tag does not. Specifically, the x-robots permits the use of regular expressions, executing crawl directives on non-HTML files, and applying parameters at a global level.

To use the x-robots-tag, you’ll need to have access to either your website’s header .php, .htaccess, or server access file. From there, add your specific server configuration’s x-robots-tag markup, including any parameters. This article provides some great examples of what x-robots-tag markup looks like if you’re using any of these three configurations.

Here are a few use cases for why you might employ the x-robots-tag:

Controlling the indexation of content not written in HTML (like flash or video)
Blocking indexation of a particular element of a page (like an image or video), but not of the entire page itself
Controlling indexation if you don’t have access to a page’s HTML (specifically, to the <head> section) or if your site uses a global header that cannot be changed
Adding rules to whether or not a page should be indexed (ex. If a user has commented over 20 times, index their profile page)

SEO best practices with robots meta directives

All meta directives (robots or otherwise) are discovered when a URL is crawled. This means that if a robots.txt file disallows the URL from crawling, any meta directive on a page (either in the HTML or the HTTP header) will not be seen and will, effectively, be ignored.
In most cases, using a meta robots tag with parameters "noindex, follow" should be employed as a way to to restrict crawling or indexation instead of using robots.txt file disallows.
It is important to note that malicious crawlers are likely to completely ignore meta directives and as such, this protocol does not make a good security mechanism. If you have private information that you don’t want to make publicly searchable, choose a more secure approach, such as password protection, to keep visitors from viewing confidential pages.
You do not need to use both meta robots and the x-robots-tag on the same page – doing so would be redundant.

Improve your technical site health with Moz Pro

Keep tabs on critical crawler issues (and tons of other details) with the help of Moz Pro's Site Crawl. Take a 30-day free trial on us and see what you can achieve:

Start my free trial

Keep learning

Put your skills to work

Moz Pro lets you run crawls, research and report on keyword ranking, and track your site's SEO performance, including its accessibility, over time. Try it >>