Meta Robots Tag: Every Rule Google Supports, What It Does to AI Overviews, and How to Audit It

The meta robots tag is an HTML element in a page’s <head> that tells search engine crawlers whether to index the page, whether to follow its links, and how much of it to show in results. It looks like <meta name="robots" content="noindex">. It only works if the crawler is allowed to fetch and read the page.

This guide covers the syntax and every rule Google supports, including the ones it has dropped. It explains what the snippet rules now do to AI Overviews, AI Mode and Copilot, and the JavaScript case that catches single-page apps. It ends with a short Python script that audits the tag across a folder of HTML, run here against a set of test pages and this site’s own build. Every rule below was read from the vendor’s documentation in October 2026, and each one links to the page it came from.

Key takeaways

• The meta robots tag controls indexing and snippets, not crawling. If robots.txt blocks a URL, Google never reads the tag on it.
• nosnippet now reaches AI features: Google says it also prevents the content being used as a direct input for AI Overviews and AI Mode.
• noarchive, nocache and nositelinkssearchbox do nothing in Google Search today, although some ranking guides still list them as working rules.
• A noindex in the initial HTML stops Google queueing the page for rendering, so JavaScript cannot remove it later.
• data-nosnippet hides part of a page from snippets. Google honours it only on span, div and section elements.

What the meta robots tag is

A meta robots tag has two attributes. name says which crawler the rule is for, and content holds one or more rules separated by commas:

<head>
  <meta name="robots" content="noindex, nofollow">
</head>

name="robots" addresses every crawler. To address one crawler, replace robots with its user agent token. Google’s specification says it supports two user agent tokens in the robots meta tag, googlebot for all text results and googlebot-news for news results, and that other values are ignored. Both attributes are case-insensitive, so NOINDEX and noindex are read the same way.

One scope detail catches ad and landing-page teams. Google says noindex in name="robots" applies to search engine crawlers, and that non-search crawlers such as AdsBot-Google may need a tag addressed to them by name. A meta robots tag aimed at all robots is not aimed at every Google crawler.

The tag is a convention rather than a standard. MDN describes it as “not part of any specification” but a de-facto standard for talking to crawlers, and only cooperative robots follow it. That is why the useful reference is each search engine’s own documentation, not a generic list of directives.

The tag has a twin. Any rule you can put in the meta tag can also be sent as an HTTP response header. That is the only option for PDFs, images and other files without a <head>. Google’s guide to blocking indexing says the two methods have the same effect.

Server, CDN and framework configuration for the header is covered in the guide to the X-Robots-Tag header. This page stays with the tag in your HTML.

The rules Google supports, grouped by job

Google’s list is long, but each meta robots rule does one of three jobs: keep the page out of the index, stop link discovery, or limit what is shown from the page. Grouped that way, the full set from Google’s robots meta tag specification looks like this:

Job	Rule	What it does in Google
Index	noindex	Do not show the page in search results
Index	none	Same as noindex, nofollow
Index	unavailable_after: [date]	Stop showing the page after a date
Index	indexifembedded	Allow indexing when embedded in an iframe, only alongside noindex
Links	nofollow	Do not follow the links on the page
Display	nosnippet	No text snippet or video preview
Display	max-snippet: [n]	Cap the text snippet at n characters
Display	max-image-preview: [setting]	none, standard or large
Display	max-video-preview: [n]	Cap a video preview at n seconds
Display	notranslate	Do not offer a translated result
Display	noimageindex	Do not index the images on the page
None	all	The default; listing it has no effect

A few details matter in practice. max-snippet:0 is equivalent to nosnippet, and -1 lets Google choose the length. unavailable_after is more than a timed noindex: Google says Googlebot will decrease the crawl rate of the URL considerably after the date. If you publish for Google Discover, read that guide before you change max-image-preview.

You never need index or follow. Indexing and following links are the defaults, so <meta name="robots" content="index, follow"> changes nothing. It is harmless, but a template that prints it on every page is a template where someone will eventually change one word.

Rules that no longer do anything in Google

Google keeps a short list of rules it now ignores. The specification says noarchive, nocache and nositelinkssearchbox aren’t used by Google Search, because the cached link and the sitelinks search box no longer exist. Leaving them in place does no harm in Google, and noarchive and nocache still mean something to Bing, as the next section shows.

This is where third-party guides go stale. Of the ranking guides I read for these keywords, one still lists nositelinkssearchbox as a working rule and another says nosnippet also works as noarchive. Neither is true in Google today.

What the tag does to AI Overviews, AI Mode and Copilot

The snippet rules used to be a cosmetic choice about how a result looked. They now decide how much of a page AI answers can use, and that is the part most guides to this tag have not caught up with.

Google

Google’s specification now names AI features directly. nosnippet applies to web search, Images, Discover, AI Overviews and AI Mode, and will also prevent the content from being used as a direct input for AI Overviews and AI Mode. max-snippet works the same way at a smaller scale: it limits how much of the content may be used as a direct input for those features.

The flip side is eligibility. Google’s guide to AI features says that to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet. So a site-wide nosnippet, for example one added years ago to stop a description showing, also removes the page from the pool of sources those answers link to. How the two surfaces differ is covered in AI Mode and AI Overviews compared.

Google is explicit that these are the controls. The same guide says that to limit the information shown from your pages in Search, use nosnippet, data-nosnippet, max-snippet, or noindex, and points to Google-Extended for training and grounding in Google’s other systems. Google-Extended is a robots.txt token, not a meta robots value, so it does not belong in this tag.

There is one exception worth knowing. Snippet limits do not apply to structured data: Google says robots meta tag limitations don’t affect the use of that structured data, with the exception of article.description and the description values of other creative works. A nosnippet page can still show a rich result built from its markup.

Bing and Copilot

Bing reads two rules that Google ignores. Content tagged NOCACHE may appear in Bing’s chat answers with only the URL, title and snippet shown. Content tagged NOARCHIVE is left out of those answers and not linked from them. A page with both is treated as NOCACHE, according to Bing’s September 2023 announcement.

Two cautions. That post describes Bing Chat, before Microsoft moved to the Copilot name, so check Bing’s current documentation before you depend on it. And because noarchive is ignored by Google but removes a page from Bing’s chat answers, a leftover noarchive from the cached-page era can cost you Copilot visibility without changing anything in Google. The wider picture for Microsoft’s assistant is in the guide to getting cited by Microsoft Copilot.

data-nosnippet: the section-level control

data-nosnippet is an HTML attribute, not a meta robots rule, but it is the finer-grained version of nosnippet. You put it on part of the page you want kept out of snippets, and the rest of the page stays eligible:

<p>This sentence can appear in a snippet.</p>
<div data-nosnippet>
  <p>This block will not.</p>
</div>

Google applies it on span, div, and section elements only. Any value is ignored because it is a boolean attribute, and the HTML must be valid: Google’s own example shows an unclosed div swallowing everything after it. Google also warns that extraction can happen before and after rendering, so it says to not add or remove the data-nosnippet attribute of existing nodes through JavaScript.

Bing added support in October 2025. Marked content is kept out of Bing’s snippets and AI answers, but it is still indexed and available for ranking. Bing also allows the attribute on any HTML element, which is looser than Google, per Bing’s data-nosnippet announcement. Wrap the content in a div, span or section and both engines will honour it.

Good candidates are the parts of a page you would never want quoted as its summary, such as subscriber-only text, legal boilerplate and cookie notices. Structured data inside the wrapped block is not affected: Google says it remains usable for search results when declared within a data-nosnippet element.

Combining rules, and what happens when they conflict

You can combine rules in one comma-separated content attribute or split them across several tags. These two forms mean the same thing:

<meta name="robots" content="noindex, nofollow">

<meta name="robots" content="noindex">
<meta name="robots" content="nofollow">

When meta robots tags for different crawlers disagree, Google does not pick a winner. Its specification says the search engine will use the sum of the negative rules: a page with nofollow for robots and noindex for googlebot is read by Googlebot as noindex, nofollow. In practice, any restrictive rule anywhere wins.

Other engines are less predictable. MDN notes that for mutually exclusive values such as index and noindex, behaviour is undefined. The fix is to emit one tag per crawler from one place in your templates, and to decide whether a page should be indexed at all before reaching for the tag. That decision is covered in canonical vs noindex.

When an audit finds two meta robots tags on one page, look for two systems printing them. A theme and an SEO plugin can each emit a tag, and a framework’s head manager can add a third. Remove all but one, then confirm in the raw source that a single tag per crawler name is left.

robots.txt hides the tag

The most common failure with this tag has nothing to do with its syntax. If robots.txt disallows a URL, the crawler never fetches it, so it never sees the meta robots noindex. Google’s specification says that in that case the rules will not be found and will therefore be ignored, and the URL can stay in the index from links alone. That is the situation behind the Indexed, though blocked by robots.txt status.

Putting noindex in robots.txt is not a workaround. Google says specifying the noindex rule in the robots.txt file is not supported.

To remove a page, allow crawling and serve noindex. To save crawl capacity on a section, block it in robots.txt and accept that it may linger in the index. Which bots read which file is set out in robots.txt for developers.

Expect a delay either way. Google has to recrawl the page to see a new tag, and its guide warns that it may take months for Googlebot to revisit a page of low importance. URL Inspection shows the HTML Googlebot received and lets you request a recrawl.

The JavaScript trap

Single-page apps often decide in the browser whether a page should be indexed. That works in one direction only.

Google’s JavaScript guide says Googlebot queues every page with a 200 status code for rendering, unless a robots meta tag or header tells Google not to index the page. If the initial HTML ships noindex, the page is never rendered. Script that later removes or rewrites the tag never runs for Googlebot. A template that defaults to noindex until the app “hydrates” will keep every route out of the index. A meta robots noindex in the first response is final for Googlebot.

The other direction works. The same guide confirms that you can use JavaScript to add a robots meta tag to a page or change its content, and recommends injecting noindex on client-side error views to avoid soft 404s.

That approach depends on rendering, so a server-rendered tag is still the safer choice. Note that Google’s own sample assumes there is no other robots meta tag present in the HTML; if one is, update it rather than adding a second. The broader rendering picture is in the JavaScript SEO guide.

The practical rule: decide indexability on the server, ship the final tag in the first HTML response, and only ever use JavaScript to make a page more restrictive. On this site, that decision is one line in the shared layout, driven by a page-level flag:

{noindex && <meta name="robots" content="noindex, follow" />}

Audit the tag across a site

Browser extensions show the meta robots tag on one page at a time. To check a whole site, you want the raw HTML a crawler receives, the effective rules Googlebot ends up with, and the mistakes from this guide flagged automatically.

The script

The script below reads saved HTML files, for example a crawler’s export or a static build folder. It uses only Python’s standard library and does not render JavaScript, which is deliberate: it reports what is in the initial HTML, the version that decides whether Google renders at all.

#!/usr/bin/env python3
"""Audit robots meta tags and data-nosnippet in saved HTML files."""
import sys
from html.parser import HTMLParser
from pathlib import Path

GOOGLE_NAMES = {"robots", "googlebot", "googlebot-news"}
OTHER_CRAWLERS = {"bingbot", "adsbot-google"}
IGNORED_BY_GOOGLE = {"noarchive", "nocache", "nositelinkssearchbox"}
SNIPPET_BLOCKERS = {"nosnippet", "max-snippet:0"}
NOSNIPPET_ELEMENTS = {"span", "div", "section"}


class Collector(HTMLParser):
    def __init__(self):
        super().__init__()
        self.in_head = False
        self.tags = []          # (name, content, in_head)
        self.nosnippet_bad = []  # elements Google will not honour

    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or "") for k, v in attrs}
        if tag == "head":
            self.in_head = True
        elif tag == "body":
            self.in_head = False
        if tag == "meta" and "name" in a:
            name = a["name"].strip().lower()
            if name in GOOGLE_NAMES or name in OTHER_CRAWLERS:
                self.tags.append((name, a.get("content", ""), self.in_head))
        if "data-nosnippet" in a and tag not in NOSNIPPET_ELEMENTS:
            self.nosnippet_bad.append(tag)

    def handle_endtag(self, tag):
        if tag == "head":
            self.in_head = False


def rules(content):
    return {r.strip().lower().replace(" ", "") for r in content.split(",") if r.strip()}


def audit(path):
    c = Collector()
    c.feed(path.read_text(encoding="utf-8", errors="replace"))
    issues = []
    googlebot = set()
    for name, content, in_head in c.tags:
        r = rules(content)
        if name in ("robots", "googlebot"):
            googlebot |= r          # Google sums the negative rules
        if not in_head:
            issues.append(f"<meta name=\"{name}\"> is outside <head>")
        for dead in sorted(r & IGNORED_BY_GOOGLE):
            if name in GOOGLE_NAMES:
                issues.append(f"'{dead}' is ignored by Google")
    if {"index", "noindex"} <= googlebot:
        issues.append("index and noindex both present: Googlebot gets noindex")
    if googlebot & SNIPPET_BLOCKERS:
        issues.append("snippet blocked: not usable as AI Overviews / AI Mode input")
    for tag in sorted(set(c.nosnippet_bad)):
        issues.append(f"data-nosnippet on <{tag}>: Google only honours span, div, section")
    verdict = "noindex" if {"noindex", "none"} & googlebot else "indexable"
    return verdict, sorted(googlebot), issues


def main(args):
    files = []
    for arg in args:
        p = Path(arg)
        files += sorted(p.rglob("*.html")) if p.is_dir() else [p]
    counts = {"indexable": 0, "noindex": 0}
    for f in files:
        verdict, effective, issues = audit(f)
        counts[verdict] += 1
        if verdict == "noindex" or issues:
            print(f"{f}: {verdict} [{', '.join(effective) or 'no tag'}]")
            for i in issues:
                print(f"  - {i}")
    print(f"\n{len(files)} files: {counts['indexable']} indexable, {counts['noindex']} noindex")


if __name__ == "__main__":
    main(sys.argv[1:] or ["."])

It merges the robots and googlebot tags the way Google’s specification describes, so the verdict is the rule set Googlebot ends up with rather than whichever tag appears first.

What it found

I ran it first on four test pages, each built to contain one mistake from this guide: a conflicting robots and googlebot pair, legacy rules, max-snippet:0 with data-nosnippet on a paragraph, and a tag placed in the <body>. This is the output, unedited:

$ python3 meta_robots_audit.py fixtures
fixtures/body-tag.html: noindex [noindex]
  - <meta name="robots"> is outside <head>
fixtures/conflict.html: noindex [follow, index, noindex]
  - index and noindex both present: Googlebot gets noindex
fixtures/legacy.html: indexable [noarchive, nositelinkssearchbox]
  - 'noarchive' is ignored by Google
  - 'nositelinkssearchbox' is ignored by Google
fixtures/snippet.html: indexable [max-snippet:0]
  - snippet blocked: not usable as AI Overviews / AI Mode input
  - data-nosnippet on <p>: Google only honours span, div, section

4 files: 2 indexable, 2 noindex

Then I ran it on this site’s production build, 178 HTML files including this page:

$ python3 meta_robots_audit.py dist
dist/404.html: noindex [follow, noindex]
dist/contact-thanks/index.html: noindex [follow, noindex]
dist/free-audit-thanks/index.html: noindex [follow, noindex]
dist/optix/thanks/index.html: noindex [noindex]

178 files: 174 indexable, 4 noindex

Four pages are noindex, all of them on purpose: the 404 page and three form confirmation pages. I checked the sitemap separately and none of the four is listed in it, which is the other half of the job. A noindex page in your sitemap sends Google two opposite signals about the same URL. A full crawl adds what a folder of files cannot, such as redirects and status codes, so treat this as one step in a technical SEO audit checklist, not the whole audit.

Mistakes that cost indexing or AI visibility

These are the meta robots tag mistakes covered above, ordered by how much they can cost:

1. A staging noindex that ships to production. One template flag, every page gone. Check the live HTML after every deploy, not the CMS setting.
2. noindex on a URL blocked by robots.txt. The tag is never read. Unblock first, then let Google recrawl.
3. A site-wide nosnippet or max-snippet:0. It hides descriptions and removes pages from the AI Overviews and AI Mode source pool. Use data-nosnippet on the sections that need hiding instead.
4. Leftover noarchive. Ignored by Google, but it removes the page from Bing’s chat answers under the 2023 rules.
5. data-nosnippet on the wrong element. On a p or li it is ignored by Google. Wrap the content in a div.
6. A noindex that JavaScript is meant to remove. It never will for Googlebot. Fix it on the server.
7. Different rules in the meta tag and the X-Robots-Tag header. Two systems now own one decision, and the next person to change either will not see the other. Pick one method per URL.
8. Trusting the CMS checkbox. The setting is an intention; the HTML is what crawlers read. Check the response, not the admin screen.

Where this fits in an audit

The meta robots tag is one line of HTML, but it is set by templates, plugins, frameworks and deploy flags that different people own. When indexing drops after a release, look at those layers first. Finding the cause means reading the raw HTML, the rendered DOM and the server headers side by side. That is the kind of check built into my GEO and technical SEO services.

Frequently asked questions

What does the meta robots tag do?

It tells search engine crawlers whether to index an HTML page, whether to follow its links, and how much of the page to show in results. In Google, the snippet rules also control how much of the page can be used as input for AI Overviews and AI Mode. The tag only works if crawlers can fetch the page.

What is the difference between robots.txt and the meta robots tag?

robots.txt controls crawling: whether a bot may request a URL at all. The meta robots tag controls indexing and snippets for a page the bot has already fetched. A URL disallowed in robots.txt is never fetched, so its meta robots tag is never read and the URL can stay indexed from links.

Does nosnippet remove a page from AI Overviews?

Google says nosnippet stops the page’s content being used as a direct input for AI Overviews and AI Mode. A page also needs to be indexed and eligible for a snippet to appear as a supporting link. If you only want to hide part of a page, use data-nosnippet on a div, span or section instead.

Should I use the meta robots tag or the X-Robots-Tag header?

Google says the two have the same effect. Use the meta tag for HTML pages your templates control, and the X-Robots-Tag header for PDFs, images and other files that have no <head>. Sending different rules through both on the same URL makes the result harder to predict, so pick one per URL.

Do I need to add index, follow to my pages?

No. Indexing and following links are the defaults, and Google says the all rule has no effect when listed. A page with no meta robots tag is treated as indexable and followable. Adding index, follow is harmless but gives a template one more value someone can change by mistake.