Crawl Errors in Google Search Console: Every Status Explained and Fixed

Crawl errors are the failures Googlebot hits when it tries to fetch a URL on your site: a server error, a redirect it cannot follow, a page it is refused or told does not exist. In today’s Search Console they no longer have a report of their own. They appear as reasons in the Page indexing report and as host problems in the Crawl Stats report.

This page is the index for every one of those statuses. For each one it gives what Google’s documentation says the status means, whether it needs fixing at all, and the fix in a paragraph, with a link to the full guide where this site has one. It also covers the site-level errors that never appear against a single URL, and how to check a fix before you ask Google to validate it.

Key takeaways

• Google removed the old Crawl Errors report in 2019 and moved the issues into what is now the Page indexing report. Most articles that rank for “crawl errors” still describe the old site errors and URL errors split.
• Only a handful of statuses are errors. Google’s report says not-indexed pages are not indexed “not necessarily because of an error”, and many exclusions (noindex, redirects, duplicates) are the site working as designed.
• Triage by two questions: is the affected URL one you want indexed, and does the report list the Source as “Website”? Google says that, in general, you can fix only issues where the source is listed as “Website”.
• Site-wide failures (robots.txt fetching, DNS, server connectivity) live in the Crawl Stats report’s host status, not in the Page indexing report. A robots.txt that returns a 5xx can stop crawling of the whole site.
• Fix every instance, test the live URL, then click Validate fix once. Google says validation typically takes up to about two weeks.

What “crawl errors” means in Search Console now

Search Console used to have a report called Crawl Errors. In 2012 Google split it into site errors and URL errors: site errors covered DNS, server connectivity and robots.txt fetching, and URL errors covered problems with one page after those checks had passed.

That report is gone. In January 2019 Google announced it would remove the old Crawl Errors report for desktop, smartphone and site-wide errors, along with the crawl errors API, because the list “was not actionable when it came to setting priorities”. The replacement was the Index Coverage report, which Search Console now calls the Page indexing report.

The old split still maps onto the two places the data lives today:

Old label	Where it lives now	What you see
URL errors	Indexing > Pages (the Page indexing report)	A reason per URL, such as Server error (5xx) or Not found (404)
Site errors	Settings > Crawl stats (the Crawl Stats report)	Host status for robots.txt fetching, DNS resolution and server connectivity

Note Google’s own scoping first. If your site has fewer than 500 pages you probably don’t need the Page indexing report, and the Crawl Stats report is aimed at advanced users with sites of a thousand pages or more.

Every status at a glance

The table covers every reason Google’s Page indexing report documentation lists, plus the host status categories from the Crawl Stats report. “Fix?” is the default answer for a URL you want in Google. If you do not want the URL indexed, most of these need no action.

Status	Kind	Fix?	Full guide
Server error (5xx)	Crawl error	Yes	Server errors
Redirect error	Crawl error	Yes	SEO redirects
Blocked due to unauthorized request (401)	Crawl error	If the page should be public	401 and 403
Blocked due to access forbidden (403)	Crawl error	If the page should be public	401 and 403
URL blocked due to other 4xx issue	Crawl error	Yes	Other 4xx
Not found (404)	Crawl error	Only if you link to it or it moved	Not found
Soft 404	Content verdict	Yes	Soft 404 errors
URL blocked by robots.txt	Intentional block	Only if blocked by mistake	Robots.txt for developers
URL marked ‘noindex’	Intentional exclusion	Only if noindex is a mistake	Canonical vs noindex
Page with redirect	Informational	No	Page with redirect
Alternate page with proper canonical tag	Informational	No	Alternate page with proper canonical tag
Duplicate without user-selected canonical	Canonical choice	If Google picked the wrong URL	Duplicate without user-selected canonical
Duplicate, Google chose different canonical than user	Canonical choice	If your canonical is right	Canonical tags
Discovered - currently not indexed	Crawl scheduling	Yes, for important URLs	Discovered, currently not indexed
Crawled - currently not indexed	Quality verdict	Yes, for important URLs	Crawled, currently not indexed
Indexed, though blocked by robots.txt	Warning	Yes	Indexed, though blocked
Page indexed without content	Warning	Yes	Indexed without content
Host status: robots.txt fetching	Site error	Yes, urgently	Site-level errors
Host status: DNS resolution	Site error	Yes	Site-level errors
Host status: server connectivity	Site error	Yes	Site-level errors

How to triage the report before fixing anything

The report sorts its “Why pages aren’t indexed” table by what Google thinks are the most important issues to address. That order is a reasonable start, but three checks decide whether a row is worth your time.

1. Is the URL one you want indexed? Google’s goal for the report is that the canonical version of every important page is indexed, and it says duplicate or alternate pages shouldn’t be indexed. A thousand parameter URLs marked duplicate is usually the report doing its job.

2. What is the Source? Each reason carries a Source of “Website” or “Google”. Fix the Website rows first. Google also recommends prioritising issues whose validation state is “failed” or “not started”.

3. Filter by sitemap. The dropdown above the chart can limit the report to URLs in a submitted sitemap. If your sitemap lists only pages you want indexed, which it should, this filter removes most of the noise in one click.

The example table for each reason is capped at 1,000 rows and is not guaranteed to show every affected URL, so read it for patterns. For a single page, use URL Inspection; for a list of URLs, the Search Console API exposes the same inspection data.

Errors where Google could not fetch the page

These are the crawl errors in the strict sense: Googlebot asked for the URL and did not get a usable page back.

Server error (5xx)

Google’s definition is short: your server returned a 500-level error when the page was requested, meaning Googlebot could not access the URL, the request timed out or the site was busy. The consequences are set out in Google’s page on how HTTP status codes affect its crawlers: 5xx and 429 responses make the crawlers slow down, already indexed URLs are kept for a while but “eventually dropped”, and the drop in crawl rate is proportionate to the number of URLs returning an error.

Fix: check host status in the Crawl Stats report to see whether the problem is site-wide. Then work through the causes Google lists: an overloaded or misconfigured server, slow dynamic pages that time out, and firewall or DoS protection blocking Googlebot because it makes more requests than a human user. A passing live test does not prove the crawl passed; your server logs show what Googlebot actually received.

Redirect error

Google reports a redirect error for a chain that was too long, a redirect loop, a redirect URL that exceeded the maximum URL length, or a bad or empty URL in the chain. Google’s crawlers follow up to 10 redirect hops by default.

Fix: trace the chain with the curl commands further down, point every old URL straight at its final destination with one 301, and update internal links and sitemaps to the final URL. Redirect types are covered in SEO redirects.

Blocked due to unauthorized request (401) and access forbidden (403)

A 401 means the page asked Googlebot for authorisation. A 403 is odder: Google points out that it means credentials were provided but refused, yet Googlebot never provides credentials, so your server is returning this error incorrectly. Both are common on staging sites, members’ areas and pages behind a bot-protection rule.

Fix: if the page should be private, this is the right outcome; keep it out of your sitemap and internal links. If it should be public, remove the login or the rule returning 403, and if Googlebot must pass a wall, verify it really is Googlebot. Do not use 401 or 403 to slow crawling: 4xx codes other than 429 have no effect on crawl rate.

URL blocked due to other 4xx issue

This covers any 4xx error not covered by another issue type, such as 400, 410 or 411. Google treats all 4xx errors except 429 the same way: the content does not exist, and an indexed URL is removed.

Fix: a 410 on a page you removed is fine. Anything else is usually an application bug, such as a route returning 400 for a query string. Reproduce the request and fix the handler.

Not found (404)

A 404 here is a URL Google found, often as a link from another page, or a page that existed before and was deleted. Google says 404s are “not necessarily a problem, if the page has been removed without any replacement”, and that Googlebot will crawl such a URL less and less often. The report only shows URLs that returned 404 in the past month.

Fix: Google recommends fixing only 404 errors that you link to yourself or list in a sitemap. If the page moved, 301 it to the new location. If it was deleted with no replacement, leave the 404 and remove your own links to it. A crawler such as those compared in Screaming Frog alternatives will list the internal links that still point at the dead URL.

Soft 404

A soft 404 is a page that returns a user-friendly “not found” message but not a 404 HTTP response code. Google also reports it when a 200 response suggests an error, such as an empty page or an error message. It is a verdict on the content, not the status code, which is why it can catch out-of-stock product pages, empty search results and thin category pages.

Fix: return a real 404 or 410 for pages that are genuinely gone. For pages that should exist, make them substantive enough that they are clearly not an error page, and check the rendered screenshot in URL Inspection in case the content only loads client-side. The causes and fixes by page type are in soft 404 errors.

Exclusions that are usually intentional

These rows inflate the “not indexed” count and cause the most unnecessary work. Each follows a signal you sent, so fix it only when the signal was a mistake.

URL blocked by robots.txt

The page was blocked by your site’s robots.txt file. Google adds a warning that matters: this does not guarantee the page stays out of the index, because Google can still index it from links elsewhere.

Fix: if the block is intended, leave it and do not validate it. If it is a mistake, remove the rule. If you wanted the page out of search, use noindex and remove the block, since Google cannot see a noindex on a page it may not crawl. Rules and testing are in robots.txt for developers.

URL marked ‘noindex’

Google found a noindex directive in the page or its HTTP headers and did not index the page. Google’s own comment is “If you do not want this page indexed, congratulations!”

Fix: only act if the page should be indexed. Check both the HTML and the X-Robots-Tag header, since a CMS or CDN can add the header outside the template. Confirm with a live test, then request indexing. When a canonical is the better signal is covered in canonical vs noindex.

Page with redirect

This is a non-canonical URL that redirects to another page, so it is not indexed. The target may or may not be.

Fix: none for the redirect itself. Check the target URL separately in URL Inspection. If a URL you expected to be indexed sits here, something is redirecting it that should not be, often a trailing slash, protocol or locale rule.

Alternate page with proper canonical tag

The page is an alternate version (AMP, mobile or desktop) that correctly points to the canonical page, which is indexed, so there is nothing you need to do. It only needs attention if a page you wanted indexed in its own right has ended up here because a template gave it the wrong canonical. That case is covered in alternate page with proper canonical tag.

Duplicate without user-selected canonical

Google decided the page is a duplicate of another page, and it doesn’t indicate a preferred canonical, so Google chose one for you. Google calls this “working as intended”.

Fix: inspect the URL to see which page Google chose. If the choice is right, declare it with a canonical tag; if it is wrong, declare the one you want or make the pages substantially different. The full diagnosis is in duplicate without user-selected canonical.

Duplicate, Google chose different canonical than user

You declared a canonical, and Google thinks another URL makes a better canonical. Google’s key point is that a duplicate must be similar to its canonical: “If the user-declared canonical is not similar to the current page, then Google won’t ever choose that URL as canonical.”

Fix: compare the page, your declared canonical and Google’s choice side by side. Then make your signals agree: the canonical tag, internal links, sitemap entries, redirects and hreflang should all point at the same URL. Google’s live test does not check duplicate or canonical conditions, so use the indexed data. The implementation rules are in canonical tags.

Scheduling and quality verdicts

Neither is a crawl error, but both are often treated as one.

Discovered - currently not indexed

Google found the URL but has not crawled it yet. Typically Google wanted to crawl it but expected that doing so would overload the site, so it rescheduled, which is why the last crawl date is empty.

Fix: on a small site, the usual cause is discovery: pages listed only in the sitemap with few or no internal links. On a large site, it can be a crawl capacity problem. The diagnosis for both is in discovered, currently not indexed, and the large-site case is in crawl budget.

Crawled - currently not indexed

Google crawled the page but did not index it, and says there is no need to resubmit. The fetch worked. This is a decision about the page, usually on value, duplication or importance.

Fix: improve the page or consolidate it with a stronger one, add contextual internal links, and remove or noindex pages that should never have been candidates. Resubmitting an unchanged page does not help. The decision tree is in crawled, currently not indexed. Pages with no internal links at all are covered in orphan pages.

Warnings on pages that are indexed

Google lists two statuses in the “Improve page experience” table. They do not stop indexing, but Google says they reduce its ability to understand and index your pages.

Indexed, though blocked by robots.txt

The page is indexed even though robots.txt blocks it, because someone else links to your page. Google indexes it from the linking page without crawling it, so the snippet is usually very limited.

Fix: to keep the page out of Google, remove the block and add noindex so Google can see it. To have it indexed properly, just remove the block. The three fixes, and when each one is right, are in Indexed, though blocked by robots.txt.

Page indexed without content

Google indexed the page but could not read the content, possibly because it is cloaked to Google or in a format Google cannot index. Google notes this is not a robots.txt block.

Fix: inspect the URL and check the rendered HTML and screenshot. Common causes are a bot-protection layer serving Googlebot a challenge page, or content injected by JavaScript that fails during rendering. The rendering side is covered in JavaScript SEO.

Site-level crawl errors: the Crawl Stats report

The old “site errors” moved to the host status panel of the Crawl Stats report, under Settings. Host status is assessed in three categories: robots.txt fetching, DNS resolution and server connectivity, and the panel shows whether Google had a significant availability problem in the past 90 days, and whether that was within the last week.

robots.txt fetching is the one that can halt a whole site. Google treats a 200 (even an empty or invalid file) and a 403, 404 or 410 as a successful robots.txt response. A 429 or 5xx is unsuccessful. The Crawl Stats documentation sets out what happens next:

• For the first 12 hours, Google stops crawling the site but keeps requesting robots.txt.
• From 12 hours to 30 days, Google uses the last robots.txt it fetched successfully.
• After 30 days, Google crawls as if there were no robots.txt if the homepage is available, and stops crawling if it is not.

So a firewall rule that starts returning 429 or 503 for /robots.txt stops crawling of every URL on the host, and none of it shows up as a Page indexing reason.

DNS resolution failures mean Google could not resolve your hostname. The chart marks an issue when the failure rate crosses a threshold line; Google’s example is DNS resolution failing for more than 5% of requests on a given day. Check with your registrar and DNS provider.

Server connectivity covers requests where the server was unresponsive or did not send a full response. The fixes are the same as for 5xx errors above.

The report’s Crawl responses table counts requests, not URLs, and lists each hop in a redirect chain as a separate request, which makes it the place to spot a spike. If you must throttle Googlebot during an incident, return 503 or 429, but not for more than two or three days, or Google may crawl the site less often in the long term.

Reproduce what Googlebot got before you fix it

Search Console shows what Google saw at crawl time. Confirm the current response outside the browser first, because caches, cookies and logged-in sessions hide most of these errors. These commands were checked against a local test server.

Check the status code and headers of one URL, including any X-Robots-Tag:

curl -sS -o /dev/null -D - -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/page/

Trace a redirect chain hop by hop, stopping at 10 hops like Googlebot:

curl -sS -o /dev/null -L --max-redirs 10 \
  -w '%{http_code} %{num_redirects} redirects, final: %{url_effective}\n' \
  https://example.com/old-page/

A loop ends with curl: (47) Maximum (10) redirects followed. Print each hop, to find which rule adds the extra step:

curl -sS -I -L --max-redirs 10 https://example.com/old-page/ | grep -iE '^(HTTP|location)'

Check robots.txt the way Google needs it: a 200, 403, 404 or 410 is fine, while a 429 or 5xx is a site-level error:

curl -sS -o /dev/null -w '%{http_code}\n' https://example.com/robots.txt

Two cautions. A firewall that verifies Googlebot by IP address can still block the real crawler when curl succeeds, so check the Crawl Stats report and your logs. And curl does not render JavaScript, so for soft 404s and pages indexed without content, use URL Inspection’s live test.

Validate the fix, once

Validation asks Google to recheck every URL with a given issue. The documented rules:

1. Fix every instance first. Validation stops when Google finds a single remaining instance of the issue.
2. Click Validate fix on the issue’s detail page, and do not click it again until validation has passed or failed.
3. Wait. Validation typically takes up to about two weeks and can take longer. Google emails you when it succeeds or fails.
4. If it fails, open See details, fix the URL that failed and start a new validation, which rechecks URLs in the Pending or Failed state.

Filtering to a sitemap of your most important pages before validating can make validation finish faster. You do not have to validate at all: counts update as Google recrawls, and an issue with no new instances drops out of the table 90 days after the last instance was marked gone. Never validate intentional rows, such as a robots.txt block you meant or a redirect after a migration.

Where this fits in an audit

These two reports are the fastest view of what Google fails to fetch, but they are samples and they lag. In a full audit I read them alongside a crawl of the site and the server logs, so a status in Search Console can be matched to the template, rule or link that causes it. That sequence is set out in the technical SEO audit checklist, and the tools for each step are compared in best technical SEO tools.

If the report shows more than a few hundred URLs in error states on pages you want indexed, the cause is usually one template or one rule, not hundreds of separate problems. Finding that rule is the kind of work covered by GEO and technical SEO services.

Frequently asked questions

What are crawl errors in Google Search Console?

Crawl errors are the failures Googlebot hits when it requests a URL: server errors, redirect errors, 401, 403, 404 and other 4xx responses, and soft 404s. Google removed the separate Crawl Errors report in 2019. URL-level errors now appear as reasons in the Page indexing report, and site-wide errors as host status in the Crawl Stats report.

Do crawl errors hurt SEO?

They hurt the URLs they affect and, in some cases, the whole site. A page returning 4xx is not indexed, and one that keeps returning 5xx is eventually dropped, while server errors also make Google slow its crawl of the site. A 404 for a page you deleted on purpose is not a problem, and Google says 404s are not necessarily a problem when a page was removed without a replacement.

How do I fix crawl errors?

Start by filtering the Page indexing report to your sitemap and fixing rows whose Source is “Website” on URLs you want indexed. Reproduce each error with URL Inspection or curl, fix the cause (server capacity, redirect chain, access rule or broken link), then click Validate fix once. Check the Crawl Stats host status as well, because robots.txt, DNS and connectivity failures do not appear against individual URLs.

Should I fix every 404 in Search Console?

No. Google recommends fixing only 404 errors that you link to yourself or list in a sitemap. If a page moved, redirect it with a 301 to the new location. If it was deleted with no replacement, a 404 is the correct response, and Google will crawl the URL less often over time.

How long does it take for crawl errors to clear after a fix?

Google says validation typically takes up to about two weeks, and sometimes longer. Without validation, counts update as Google recrawls the affected URLs. An issue with no remaining instances drops out of the report 90 days after the last instance was marked gone.