Technical SEO
Faceted Navigation SEO: Controlling Crawl and Index on Filtered Listings
Faceted navigation SEO is the work of deciding which filtered views of a listing (level, price, duration, language, sort) get a crawlable, indexable URL, and making sure every other combination costs Google as little crawling as possible. Done well, a handful of filter pages rank for real searches and the rest never reach the crawl queue.
This guide uses a course catalogue as the running example, because catalogues and course marketplaces filter on exactly the facets that cause trouble. It covers how to decide which combinations deserve a URL, what robots.txt, noindex, canonicals and nofollow each actually control, a robots.txt policy I tested with a parser, and how to confirm the result in your server logs.
Key takeaways
- Google’s own guidance names two costs of parameter-based facets: overcrawling and slower discovery crawls, because crawlers cannot tell a filter URL is useless until they have fetched it.
- Give a URL only to combinations with search demand and enough distinct inventory to be a real page. Everything else should be a parameter you can block with one pattern.
- robots.txt is the only control that stops crawling.
noindexandrel="canonical"both need the page to be fetched first, and Google calls canonical andnofollowgenerally less effective in the long term. - The Search Console URL Parameters tool is gone: Google deprecated it in 2022. Parameter handling now lives in robots.txt and your templates.
- Check the result in access logs, not only in Search Console: group verified Googlebot requests by facet bucket and watch the share going to URLs that should be blocked.
Why faceted navigation breaks crawling
A filter is a query against your inventory, and each query written as a link becomes a URL crawlers have not seen.
Google’s faceted navigation documentation describes the result plainly. Because the URLs “seem to be novel and crawlers can’t determine whether the URLs are going to be useful without crawling first”, crawlers “will typically access a very large number of faceted navigation URLs” before deciding they are useless. The time spent there is time not spent on new pages, which Google calls slower discovery crawls.
The numbers grow faster than most teams expect. Take a modest catalogue: 12 subjects, 3 levels, 4 price bands, 5 duration bands, 6 teaching languages and 4 sort orders. I calculated the URL space for it in Python:
| Filter behaviour | Distinct listing URLs |
|---|---|
| One value per facet, parameters always in the same order | 50,400 |
| Multi-select on level, price, duration and language | 15,728,640 |
| Curated landing pages worth indexing (all courses, subject, subject + level, subject + free) | 61 |
The first row is 12 × 4 × 5 × 6 × 7 × 5, with each facet unset or set to one value; the second allows any subset of values. If parameters can appear in any order, five of them can be written 120 ways, and every row multiplies again. The 61 pages anyone would search for stay the same.
Google’s URL structure guidance makes the same point with hotels: when filters combine in an additive manner, “the number of URLs (views of data) in the sites explodes”, yet Googlebot “only needs to see a small number of lists from which it can reach the page for each hotel.” For a course catalogue, read “course” for “hotel”. The course detail pages are what must be crawled; the listings exist to reach them.
This is why faceted navigation sits at the top of most crawl budget investigations on large sites, and why it so often shows up as a pile of URLs in Discovered, currently not indexed.
Which filter combinations deserve a URL
Decide this before touching robots.txt. I use four tests, and a combination gets an indexable URL only if it passes all four.
- Search demand. People search for this combination as a phrase. “Beginner data science course” and “free Python course” are real queries. “Data science course, 4 to 8 weeks, sorted by rating” is not.
- Distinct inventory. The filtered page lists a meaningfully different set of courses from its parent, and enough of them to be a useful page. A combination that returns two courses is a thin page, and one that returns none should not exist at all.
- Stable over time. The page will still have courses next term. Cohort dates and temporary promotions fail this test.
- You would link to it on purpose. If you would put it in the navigation, a hub page or the sitemap, it is a page. If it only exists because a user ticked boxes, it is a view.
A facet policy for a course catalogue
Here is how those tests usually play out for a course catalogue or marketplace. Your keyword data overrules the table where they disagree.
| Facet | Typical verdict | Why |
|---|---|---|
| Subject or category | Indexable path | The main demand. These are your category pages. |
| Level (beginner, intermediate, advanced) | Indexable path, combined with subject only | ”Beginner [subject] course” has demand; level on its own rarely does. |
| Price: “free” | Indexable path, combined with subject | ”Free [subject] course” is a common query pattern. |
| Price range or slider | Blocked parameter | Nobody searches “courses £120 to £340”. Infinite values if it is a slider. |
| Duration | Blocked parameter | Low demand, and the bands overlap. |
| Teaching language | Parameter, crawlable, noindex | Useful to users, rarely searched as a listing. If you target other languages properly, build localised sections with hreflang instead. |
| Sort order, view, page size | Blocked parameter | Same courses, different order. Never a page. |
| Multi-select (two levels at once) | Blocked parameter | Duplicates the union of pages you already have. |
Pagination (?page=2) | Crawlable, self-canonical | Needed to reach every course. See below. |
Pagination is the exception people get wrong. Page 2 of a subject listing is not a facet: it holds different courses, and blocking it can cut crawlers off from the courses that only appear deep in the list. How to keep paginated and infinite-scroll listings crawlable is covered in pagination SEO and in infinite scroll vs pagination for SEO.
Give indexable combinations clean, fixed-order paths
Once you have the list, split the two populations by URL shape. Indexable combinations live on paths the site generates on purpose:
/courses/ all courses
/courses/data-science/ subject
/courses/data-science/beginner/ subject + level
/courses/data-science/free/ subject + free
Everything else stays in the query string. That split is what makes the blocking rules in the next section short and safe: you block parameters by name and never risk matching a page you want indexed.
Google’s documentation adds three rules for filter URLs you do want crawled. Use the standard & separator, because characters like commas, semicolons and brackets “are hard for crawlers to detect as parameter separators”. If filters are encoded in the path, make sure “the logical order of the filters always stays the same and that no duplicate filters can exist”. And return a real 404 when a combination has no results, rather than redirecting to a generic not-found page (Google, faceted navigation).
That last rule matters on course sites, where an empty “advanced cooking, free” page returning 200 with “No courses found” is a soft 404 waiting to happen.
The controls, and what each one actually does
The options are often listed as if interchangeable. They are not: each acts at a different stage, and two of them cannot work together on the same URL.
| Control | Stops crawling? | Stops indexing? | Consolidates signals? | Main catch |
|---|---|---|---|---|
robots.txt Disallow | Yes | Not reliably: the URL can be indexed without content | No | Google cannot see a noindex or canonical on a blocked page |
noindex (meta or header) | No | Yes, once crawled | No | The page must stay crawlable for the rule to be read |
rel="canonical" | Only gradually, over time | Usually, as a hint | Yes | A hint, not a directive; filtered subsets are not true duplicates |
rel="nofollow" on filter links | Partly | No | No | Every link to the URL needs it |
Filters in URL fragments (#) | Yes, no new URLs | Yes | Not applicable | Filtered views can never rank |
| 404 for empty combinations | Reduces recrawling | Yes | Not applicable | Only covers the empty cases |
robots.txt: the only control that prevents the crawl
For facets you never want in Search, Google’s recommendation is to disallow them. Its documentation is direct: “Oftentimes there’s no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit; instead, allow crawling of just the individual items’ pages along with a dedicated listing page that shows all products without filters applied” (Google, faceted navigation).
Understand what a disallow does not do. Google’s robots.txt specification says Google “can’t index the content of pages which are disallowed for crawling, but it may still index the URL and show it in search results without a snippet” (robots.txt specification). That is how blocked facet URLs end up in the Indexed, though blocked by robots.txt report, usually because something links to them.
noindex: removes from the index, costs a crawl
noindex is the right tool for filter pages that are useful to users and harmless to crawl in moderation, such as the teaching-language filter in the table. Google drops the page from results “when Googlebot crawls that page and extracts the tag or header” (Block Search indexing with noindex).
The same page names the conflict: if a page is still appearing in results, “another reason could also be that the robots.txt file is blocking the URL from Google web crawlers, so they can’t see the tag.” You cannot noindex and disallow the same URL and expect both to work. Pick one per URL pattern.
rel=“canonical”: a hint that consolidates, slowly
Canonicals suit parameters that produce true duplicates, such as tracking parameters or a view=grid toggle you have not blocked. Google describes the canonical link as “a strong signal”, not a directive (consolidating duplicate URLs), and says of faceted URLs specifically that it “may, over time, decrease the crawl volume of non-canonical versions.”
A filtered listing is a subset of its parent, not a copy. When the content differs, Google can ignore your canonical and choose its own, which is how facet URLs turn up under Duplicate without user-selected canonical or as an alternate page with proper canonical tag. How canonicals are chosen is covered in canonical tags.
Two canonical rules from Google apply directly to facets. “Don’t use the robots.txt file for canonicalization purposes”, and Google does not recommend noindex “to prevent selection of a canonical page within a single site” (consolidating duplicate URLs). Each tool has one job.
nofollow, fragments and JavaScript filters
rel="nofollow" on filter links can help, but Google notes that “every anchor pointing to a specific URL must have the rel="nofollow" attribute in order for it to be effective” (Google, faceted navigation). One unmarked link from a sitemap, an email or a partner site and the URL is discoverable again.
Fragment-based filters (/courses/#level=beginner) “will have no impact on crawling (positive or negative)”, but Google Search also generally doesn’t support URL fragments, so they can never rank. Fine for sort and view, wrong for any facet you want indexed. For client-side filters, see the JavaScript SEO rules on real links and the History API.
The URL Parameters tool no longer exists
Older guides still tell you to configure parameters in Search Console. Google deprecated the URL Parameters tool in March 2022, noting that only about 1% of the parameter configurations set in it were useful for crawling. Its replacement advice is robots.txt rules. There is no parameter setting left to configure.

A robots.txt policy for a course catalogue, tested
Here is the policy from the facet table, written as robots.txt. It blocks presentation parameters, the facets with no demand of their own, any multi-value selection and internal search, and leaves clean paths, pagination and the teaching-language filter crawlable.
User-agent: *
# Presentation parameters: they never change which courses are listed
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?view=
Disallow: /*&view=
Disallow: /*?per_page=
Disallow: /*&per_page=
# Facets with no search demand of their own
Disallow: /*?price_min=
Disallow: /*&price_min=
Disallow: /*?price_max=
Disallow: /*&price_max=
Disallow: /*?duration=
Disallow: /*&duration=
# Any multi-value selection, e.g. level=beginner,intermediate
Disallow: /*?*,
# Internal site search results
Disallow: /search
Sitemap: https://example.com/sitemap.xml
I ran these rules through Protego 0.7.0, an open-source Python robots.txt parser that supports the * and $ wildcards, against a set of catalogue URLs. This is the actual output:
allow /courses/
allow /courses/data-science/
allow /courses/data-science/beginner/
allow /courses/data-science/free/
allow /courses/data-science/?lang=es
allow /courses/data-science/?page=2
BLOCK /courses/data-science/?sort=rating
BLOCK /courses/data-science/?lang=es&sort=rating
BLOCK /courses/data-science/?level=beginner,intermediate
BLOCK /courses/data-science/?duration=under-4-weeks
BLOCK /courses/data-science/?price_min=0&price_max=200
allow /courses/?provider=acme&subsort=newest
BLOCK /search?q=python
Protego is not Google’s parser, so treat this as a test of your own logic. Google’s robots.txt specification is the authority on matching: the most specific (longest) rule wins, and when rules conflict, Google uses the least restrictive one.
The substring trap
Why two lines per parameter (?sort= and &sort=) rather than one? Google’s own example uses the shorter form, disallow: /*?*color=. That pattern matches color= anywhere after the ?, including inside a longer parameter name. I tested both forms:
loose BLOCK /courses/?provider=acme&subsort=newest (Disallow: /*?*sort=)
tight ALLOW /courses/?provider=acme&subsort=newest (Disallow: /*?sort= and /*&sort=)
With the loose form, subsort= is blocked by a rule meant for sort=, and nothing warns you. Anchoring each name to ? or & avoids it.
Two more details from the specification. Rules only apply within the group for the matching user agent, and Google does not combine a Googlebot group with the * group, so if your file has a User-agent: Googlebot section, the facet rules must be repeated there. And robots.txt is cached for up to 24 hours, so changes do not take effect instantly. For the rest of the file’s behaviour, see robots.txt for developers.
Retrofitting a site whose facets are already indexed
The order matters when facet URLs are already in the index. If you disallow them first, Google stops fetching them, never sees a noindex, and the URLs can linger as indexed without content.
- Add
noindex(or fix the canonical) on the facet patterns you want out, and leave them crawlable. - Stop linking to them: remove them from sitemaps, and switch sort and view controls to buttons or fragments.
- Watch the Page Indexing report until the patterns show as excluded by
noindex. - Then add the robots.txt disallow to stop the crawling.
Step 3 can be slow: Google notes it “may take months for Googlebot to revisit a page” of low importance (Block Search indexing with noindex). If crawl load is hurting the server now, block first and accept some URL-only index entries for a while.
Checking the result in log files
Search Console tells you what Google indexed; only the access log tells you what Googlebot requested, which is what these controls are meant to change. The general method is in log file analysis for SEO. For facets, group every Googlebot request by the bucket your policy puts it in.
This script reads a combined-format access log and does exactly that. The bucket rules mirror the policy above; change the parameter names to yours.
import re, sys
from collections import Counter
from urllib.parse import urlsplit, parse_qsl
# Combined log format: IP - - [time] "METHOD PATH PROTO" STATUS BYTES "REF" "UA"
LINE = re.compile(r'^(\S+) \S+ \S+ \[[^\]]+\] "(?:GET|HEAD) (\S+) [^"]*" (\d{3}) \S+ "[^"]*" "([^"]*)"')
BLOCKED = {"sort", "view", "per_page", "price_min", "price_max", "duration"}
NOINDEX = {"lang"}
def bucket(path):
parts = urlsplit(path)
if not parts.path.startswith("/courses/"):
return "other"
params = parse_qsl(parts.query, keep_blank_values=True)
keys = {k for k, _ in params}
if not keys:
return "clean facet path"
if keys & BLOCKED or any("," in v for _, v in params):
return "should be blocked"
if keys <= {"page"}:
return "pagination"
if keys <= NOINDEX | {"page"}:
return "noindex parameter"
return "unknown parameter"
hits, status = Counter(), Counter()
for line in open(sys.argv[1], encoding="utf-8", errors="replace"):
m = LINE.match(line)
if not m or "Googlebot" not in m.group(4):
continue
b = bucket(m.group(2))
hits[b] += 1
status[(b, m.group(3))] += 1
total = sum(hits.values()) or 1
for b, n in hits.most_common():
codes = ", ".join(f"{c}:{k}" for (bb, c), k in sorted(status.items()) if bb == b)
print(f"{b:20} {n:6} {n/total:6.1%} {codes}")
Run it as python3 facet_log_audit.py access.log. It prints one row per bucket with the hit count, the share of Googlebot requests and the status codes returned. I checked the parsing against a small hand-written test log, not a client site.
Filtering on the user-agent string is only a first pass, since anyone can send “Googlebot”. Before you act on the numbers, verify the IPs: Google’s method is a reverse DNS lookup that resolves to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup that returns the original IP, or a match against its published crawler IP ranges.
What to read in the output
- “should be blocked” above zero after the robots.txt change. Allow a day or two for the cached file to refresh. After that, verified Googlebot requests to disallowed patterns mean the rule does not match the way you think, or there is a
Googlebotgroup without it. - “unknown parameter” growing. A new parameter has appeared, often a tracking tag on internal links or a filter shipped without an SEO decision.
- 404s under “clean facet path”. Fine for empty combinations removed on purpose, a bug for pages you link to.
- The share for clean paths and pagination. This is the crawling you want. It should grow as the blocked share falls.
Compare two weeks before the change with two weeks starting a few days after it, and read the shares rather than raw totals, since crawl rate also responds to server health and demand.
Then cross-check in Search Console: the Page Indexing report should show your noindex facets as excluded, and URL Inspection on a sample from each bucket confirms the canonical Google selected. The course detail pages the listings lead to deserve their own markup, covered in course schema.
Faceted navigation is one phase of a larger crawl and indexation review. If you want the policy, the robots.txt and the log check done on your own catalogue, that is part of the technical SEO and GEO work I take on.
Frequently asked questions
What is faceted navigation in SEO?
Faceted navigation is the set of filters on a listing page (such as level, price, duration or language on a course catalogue) that change which items are shown. In SEO terms, it matters because each filter combination can become a separate URL. Faceted navigation SEO is deciding which of those URLs should be crawled and indexed and controlling the rest.
Should faceted navigation pages be indexed?
Only the combinations that have search demand, list a meaningfully different set of items, stay stable over time and that you would link to on purpose. For a course catalogue that usually means subject pages, subject plus level, and subject plus free. Sort orders, price ranges, durations and multi-select combinations should not be indexed.
Is it better to use robots.txt or noindex for filter URLs?
They do different jobs. robots.txt stops crawling but can leave the URL indexed without content; noindex removes the page from the index but only works if Google can crawl the page and read it. For facets that were never indexed, robots.txt is usually right. For facets already in the index, add noindex first, wait for them to drop out, then block.
Should filtered pages canonicalise to the unfiltered category?
For true duplicates, such as sort orders or view toggles, yes. For filtered subsets it is a weaker fix, because a canonical is a hint and Google may ignore it when the content differs. Google’s faceted navigation guidance describes canonicals as generally less effective in the long term than robots.txt or fragment-based filters.
How do I handle URL parameters now that the Search Console tool is gone?
Google deprecated the URL Parameters tool in 2022 and points site owners to robots.txt rules instead. Keep parameters for filters you do not want indexed, block them by name in robots.txt, and anchor each rule to ? or & so it does not match longer parameter names.
How can I tell if faceted navigation is wasting crawl budget?
Group verified Googlebot requests in your access logs by URL pattern and compare the share going to filter and sort URLs with the share going to category and detail pages. A large share on parameter URLs, alongside new pages sitting in Discovered, currently not indexed, is the usual sign.