When should you use noindex on a page?

Use noindex on pages that provide no search value or could dilute your SEO performance: duplicate content pages, internal search results, paginated archive pages beyond page one, staging or test environments, thank-you and confirmation pages, login and account pages, and thin content pages that exist for navigation but lack substantive content worth ranking.

Does noindex waste crawl budget?

A noindex tag does not prevent crawling - it only prevents indexing. Search engines must still crawl the page to discover the noindex directive, which means those pages do consume crawl budget on every visit. If crawl budget is a concern for large sites, combining noindex with a robots.txt disallow or removing internal links to those pages is more efficient than noindex alone.

What Is Noindex?

Noindex is a directive that instructs search engines not to include a specific page in their search index, effectively preventing it from appearing in search results. It is implemented either as an HTML meta tag in the page's head section or as an HTTP response header, and it is one of the most important tools in technical SEO for controlling which pages search engines show to users.

How Does Noindex Work?

Noindex can be implemented in two ways:

Meta robots tag. The most common method is adding a meta tag to the HTML head of the page:

<meta name="robots" content="noindex">

This tells all search engine crawlers that encounter the page not to add it to their index. You can also target specific search engines by replacing "robots" with the crawler name, such as "googlebot."

X-Robots-Tag HTTP header. For non-HTML resources like PDFs or for pages where you cannot modify the HTML, you can send the directive as an HTTP response header:

X-Robots-Tag: noindex

Both methods achieve the same result. The key distinction to understand is that noindex does not prevent crawling. A search engine bot will still visit and download the page - it simply will not add the page to its searchable index after processing it. This differs fundamentally from robots.txt, which prevents crawling entirely.

When Should You Use Noindex?

Duplicate Content

If your site generates multiple URLs with identical or near-identical content - common with faceted navigation, URL parameters, or print-friendly versions - noindexing the duplicates prevents search engines from splitting ranking signals across multiple pages. According to Google's Search Central documentation, noindex is one of the recommended approaches for handling duplicate content alongside canonical tags.

Internal Search Results Pages

When users search your site, the resulting pages are dynamically generated, often thin on unique content, and can create an infinite number of URL combinations. Noindexing internal search results prevents search engines from wasting resources on these pages and keeps them from appearing in search results where they provide poor user experience.

Paginated Archive Pages

Blog archives, category pages beyond page one, and other paginated content often contain only links to other pages without substantial unique content. Noindexing pages 2 and beyond while keeping page 1 indexed ensures search engines focus on your primary category pages.

Staging and Development Environments

Staging sites should always use noindex to prevent test content from appearing in search results. A common SEO mistake is launching a production site with noindex tags accidentally carried over from staging, or conversely, forgetting to add noindex to a publicly accessible staging environment.

Thank-You and Confirmation Pages

Post-conversion pages like order confirmations, form submission thank-you pages, and email signup confirmations have no search value and should not be indexed.

Thin Content Pages

Pages that exist for site navigation but contain little unique content - such as tag pages with only one or two posts, empty category pages, or author archive pages - can dilute your site's overall quality in the eyes of search engines.

How Does Noindex Differ from Robots.txt?

This is a critical distinction that many site owners get wrong. Robots.txt tells search engines not to crawl a page. Noindex tells search engines not to index a page they have already crawled.

If you block a page with robots.txt, search engines cannot see the noindex tag because they never download the page. Paradoxically, a robots.txt-blocked page can still appear in search results if other pages link to it - Google may index the URL based on anchor text and link context without ever seeing the page content.

For pages you want completely excluded from search results, use noindex rather than robots.txt. Reserve robots.txt for pages you do not want crawled at all - such as admin panels, API endpoints, or resource-heavy pages that waste crawl budget.

What Are Common Noindex Mistakes?

Noindexing pages that should rank. Accidentally applying noindex to important pages - often through a global setting left over from development - can cause devastating traffic losses. According to a study by Ahrefs analyzing 2 million pages, approximately 4.5 percent of pages with noindex tags appear to be unintentional, suggesting a widespread problem with noindex misapplication.

Using noindex with canonical tags. Pointing a canonical tag to a different URL while also using noindex on the same page sends conflicting signals. If you want to consolidate duplicate content, use a canonical tag alone. If you want to remove a page from the index entirely, use noindex alone.

Forgetting noindex on staging sites. When staging environments are publicly accessible without noindex, search engines may index test content, duplicate content, or placeholder pages that damage the brand's search presence.

Relying on noindex for sensitive content. Noindex prevents indexing but does not prevent access. Anyone with the URL can still visit the page. For genuinely sensitive content, use authentication or access controls rather than relying on noindex.

How Does Noindex Affect Crawl Budget?

Noindexed pages still consume crawl budget because search engines must crawl them to discover the noindex directive. For small to medium sites with under 10,000 pages, this rarely matters. For large programmatic SEO sites with millions of pages, the crawl budget consumed by noindexed pages can slow down the discovery and indexation of pages you actually want ranked.

For large-scale sites, the most efficient approach is to remove internal links to pages you do not want indexed and use robots.txt to block crawling of entire URL patterns you know should never appear in search results. Reserve noindex for individual pages that need to remain crawlable for other reasons - such as pages that pass link equity or need to be accessible to users through direct links.

Noindex is a precision tool for search index management. Used correctly, it keeps your search presence clean and focused on the pages that drive value. Used incorrectly, it can silently remove important pages from search results or waste crawl budget on pages that should never be visited by search engines in the first place.