AI Overview CTR: How to Quantify the Impact on Your Clicks From Search Console Data, With a Method That Survives Scrutiny

Everyone can see that clicks are down where AI Overviews appear. Almost nobody can put a number on it that would survive a sceptical analyst, because the obvious comparison, CTR before against CTR after, is contaminated by everything else that changed in the same window. Positions moved. The query mix shifted. Seasonality happened. Google changed how position is counted inside the AI Overview itself. A number produced without handling those is a story, not a measurement.

This is the method I use to quantify AI Overview CTR impact from Search Console data. It is a difference-in-differences design: queries that gained an AI Overview are compared with queries that did not, matched on the position they held before, and the estimate is the gap between how the two groups’ CTR moved. It comes with a confidence interval, a minimum sample size, and a list of the ways it can still be wrong. If you want the measurement stack around it, the reporting layers are in how to track AI Overviews; this article is the analysis that sits on top.

Key takeaways

• A before-and-after CTR comparison on the same queries is not evidence, because position, query mix and season move in the same window; the estimate needs a control group.
• Search Console assigns every link inside an AI Overview the Overview’s single position, so match queries on their pre-period position, never on the post-period one.
• Google reports AI Overview and AI Mode traffic inside the Web search type with no separate breakdown, so AI Overview presence has to be flagged from an external SERP feature dataset.
• The estimate is a difference-in-differences: the CTR change on treated queries minus the CTR change on matched control queries, with a bootstrap interval over queries.
• Detecting a 30% relative CTR drop from a 2% base needs roughly 7,300 impressions per group; below that the interval is wider than any plausible effect.

Why “our CTR dropped” is not a finding

The Search Console Performance report makes the wrong analysis easy. Filter to a date range, compare it with the previous one, and CTR is down. Three things are wrong with treating that as the AI Overview effect.

The first is position. CTR is a steep function of rank, and if the pages behind those queries slipped from third to fifth in the same window, most of the CTR change is the slip, not the Overview. The second is composition. Queries enter and leave the report as impressions change, and the average CTR of a different set of queries is not the CTR of the old set. Pew Research’s widely cited finding, that users clicked a traditional result on 8% of visits with an AI summary against 15% without, is comparing query populations, not the same queries before and after, which is exactly the limitation a site-level analysis has to design out. The third is the surface itself. Google’s documentation says AI Overviews are shown only when its systems judge them additive and that they often do not trigger, so presence is intermittent, and a query is not simply “an AI Overview query” for a whole quarter.

A defensible AI Overview CTR number therefore has to hold position constant, hold the query set constant, and compare against queries that went through the same weeks without an Overview. That is what a matched difference-in-differences does.

What Search Console actually records for AI Overviews

The method depends on three documented properties of the data.

Clicks and impressions from AI features are inside the ordinary numbers. Per Google’s AI features guidance, sites appearing in AI Overviews and AI Mode are included in overall search traffic and reported in the Performance report within the Web search type. There is no filter that isolates them, which is why an external flag is needed.

Position inside an Overview is the Overview’s position. Search Console’s methodology page states that an AI Overview occupies a single position in search results and all links in the AI Overview are assigned that same position, that clicking an external link in it counts as a click, and that standard impression rules apply. So when a page is cited in an Overview at the top of the page, its recorded position improves, sometimes dramatically, while its CTR may fall. Matching on post-period position would therefore compare Overview-cited pages with genuinely top-ranking pages and understate the effect. Match on the pre-period position, when the Overview was not there.

Follow-ups are new queries. The same page documents that a follow-up question inside AI Mode is essentially a new query, so AI Mode sessions generate impressions on queries you may never have ranked for in classic search. That is the mechanism behind the impression inflation examined in the AI Mode impressions study, and it is a reason to run this analysis on query-level rows rather than on site totals.

The method: difference-in-differences on position-matched queries

The design has two groups and two periods.

• Treated queries are those where an AI Overview was present in the post period and absent in the pre period.
• Control queries are those where no AI Overview was present in either period, matched to treated queries on pre-period average position bucket, impression decile and query type.
• Pre period is a window before the Overview appeared; post period is the window after. Use the same length, and avoid windows that straddle a core update.

The estimate is:

effect = (CTR_treated_post − CTR_treated_pre) − (CTR_control_post − CTR_control_pre)

The control group absorbs everything that happened to comparable queries in the same weeks: seasonality, algorithm changes, the general drift in CTR. What is left is the change specific to the treated group, which, if the matching is good, is the Overview.

Two refinements make it hold up. Compute CTR from summed clicks and impressions within each group, not as an average of per-query CTRs, so that high-impression queries carry their weight. And express the effect both as absolute percentage points and as a relative change on the treated group’s pre-period CTR, because a two-point drop means something different at 3% than at 12%.

Step 1: pull query-level rows from the API, not the interface

The interface aggregates, samples and truncates. Use the Search Console API to pull daily rows by query and page for the whole window, filtered to search type web and, if you serve several countries, one country at a time. Keep every row, including the ones with zero clicks; a query with 400 impressions and no clicks is data.

Exclude three things before matching. Branded queries, because their CTR is governed by navigation rather than by the results page. Queries with fewer than a minimum impression count in each period, because a CTR from 12 impressions is noise. And pages you changed materially in the window, because a rewrite is a confound the control cannot remove.

Step 2: flag AI Overview presence

Presence has to come from outside Search Console. Any SERP feature dataset that records AI Overview presence per query at a point in time will do: an SEO platform’s keyword export, a rank tracker with feature detection, or your own SERP captures. Capture presence at least twice, once inside the pre period and once inside the post period, and define treated as absent-then-present.

On this site the flag looks like this for a dozen of the queries with the most impressions in the last 90 days, checked against a US SERP feature dataset this week:

Query	AI Overview present
angular seo	yes
hreflang canonical	yes
surfer seo alternative	yes
single page app seo	yes
frase alternatives	yes
disavow google search console	yes
javascript seo	no
astro seo	no
chatgpt search	no
ai citation decay	no
how to get cited by perplexity	no
geo audit checklist	no

The split is the useful part. Half of these queries have an Overview and half do not, at comparable positions, which is the raw material for a matched comparison. It is also a reminder that presence is not predictable from topic: the JavaScript SEO query has no Overview while the Angular one does.

Because Overviews are intermittent, a single capture misclassifies some queries. Two captures reduce it; a weekly capture over the whole window lets you define treated as “present on most captures in the post period”, which is more robust.

Step 3: match on pre-period position and volume

Bucket every query by its pre-period average position (1 to 3, 4 to 6, 7 to 10, 11 to 20), by pre-period impression decile, and by shape (question-form or not, since question-shaped queries behave differently, as the AI Mode study found). For each treated query, draw control queries from the same bucket. With enough queries you can do exact matching within buckets; with fewer, pool the buckets and weight controls so the treated and control groups have the same position distribution.

Check the match before you look at the result. If the treated group’s pre-period CTR differs sharply from the control group’s, the match is poor and the estimate will inherit the gap. The pre-period difference should be close to zero; if it is not, tighten the buckets.

Step 4: compute the estimate and its interval

CTR is a proportion, but the sampling unit is the query, not the impression, because impressions within a query are not independent. The right interval is a bootstrap over queries: resample the treated and control query sets with replacement, recompute the pooled CTRs and the difference-in-differences, and repeat a few thousand times. The 2.5th and 97.5th percentiles of the resampled effects are the 95% interval.

import numpy as np

def pooled_ctr(rows):
    clicks = sum(r["clicks"] for r in rows)
    impr = sum(r["impressions"] for r in rows)
    return clicks / impr if impr else 0.0

def did(treated_pre, treated_post, control_pre, control_post):
    return (pooled_ctr(treated_post) - pooled_ctr(treated_pre)) \
         - (pooled_ctr(control_post) - pooled_ctr(control_pre))

def bootstrap_did(treated, control, n=4000, seed=1):
    # treated/control: dicts keyed by query -> {"pre": row, "post": row}
    rng = np.random.default_rng(seed)
    tq, cq = list(treated), list(control)
    effects = []
    for _ in range(n):
        ts = rng.choice(tq, size=len(tq), replace=True)
        cs = rng.choice(cq, size=len(cq), replace=True)
        effects.append(did([treated[q]["pre"] for q in ts], [treated[q]["post"] for q in ts],
                           [control[q]["pre"] for q in cs], [control[q]["post"] for q in cs]))
    return did([treated[q]["pre"] for q in tq], [treated[q]["post"] for q in tq],
               [control[q]["pre"] for q in cq], [control[q]["post"] for q in cq]), \
           np.percentile(effects, [2.5, 97.5])

Convert the point estimate into clicks by multiplying it by the treated group’s post-period impressions. That is the number the business wants: clicks per month attributable to the Overview on the queries that gained one.

How much data you need

Run the power calculation before the analysis, because on many sites the answer is “not enough yet”. For a two-proportion comparison at a 5% significance level and 80% power, the impressions needed per group are roughly:

n ≈ 2 × (1.96 + 0.84)² × p̄(1 − p̄) / (p₁ − p₂)²

where p₁ and p₂ are the CTRs you want to tell apart and p̄ is their mean. To detect a fall from 2.0% to 1.4%, a 30% relative drop, that gives about 7,300 impressions per group per period. To detect a 15% relative drop you need roughly four times as many. Impressions on low-CTR queries are cheap, so on a mid-sized site this is a month of data; on a small site it can be a year.

This site is the honest example. Over the last 90 days, the query with the most impressions on the treated side, hreflang canonical, has 627 impressions and a CTR that rounds to zero, and the totals across the dozen flagged queries fall far short of the threshold. The method runs, the code returns a number, and the interval around that number is wider than any effect the Overview could plausibly have. The correct output in that situation is “not measurable at this volume yet”, stated as such, and a plan to accumulate the window rather than a headline.

Threats to validity, and what to do about each

• Position contamination. Overview citations move recorded position. Mitigation: match on pre-period position only, and report the post-period position shift separately as a descriptive.
• Intermittent presence. A query flagged as treated may have shown an Overview on a fraction of days. Mitigation: multiple captures, and a sensitivity run that drops queries with ambiguous presence.
• Query mix drift. New queries enter with AI Mode follow-ups. Mitigation: restrict both periods to queries present in both, which the matching already requires.
• Spillover. A control query can share a page with a treated one, and the page’s changes affect both. Mitigation: match at query level but exclude controls that share a landing page with a treated query.
• Seasonality unequal across groups. If treated queries are informational and controls are commercial, their seasonal curves differ. Mitigation: include query shape in the match, and compare pre-period trends, not just levels.
• Being cited versus being displaced. Being cited inside the Overview and being pushed below it are different treatments with different effects. Mitigation: if your feature dataset records citation, split the treated group into cited and not cited, and report both.

None of these are reasons not to measure. They are the list a reviewer will bring, and having answered each in the write-up is what makes the number survive the meeting.

What the number can and cannot claim

The output is a sentence with three parts: the estimate, its interval, and its scope. “On queries that gained an AI Overview between May and August, CTR fell by 1.8 points more than on position-matched queries that did not, 95% interval 0.9 to 2.6, equivalent to about 640 clicks a month at current impressions” is a claim that holds. Generalising it to the whole site, to queries that had an Overview all along, or to next quarter, is not.

It also does not measure the assistants. ChatGPT, Claude and Gemini referrals are a separate attribution problem, handled in attributing conversions from AI assistants, and the two numbers should never be added. Search Console measures Google. Keep the AI Overview CTR estimate as the Google line in your SEO KPIs, refresh it quarterly as the windows lengthen, and let the interval narrow before anyone builds a forecast on it.

Frequently asked questions

Can Search Console show AI Overview clicks separately?

No. Google reports clicks and impressions from AI Overviews and AI Mode inside the Web search type of the Performance report with no filter to isolate them. AI Overview presence has to be flagged from an external SERP feature dataset and joined to the query-level rows.

Why match on pre-period position instead of current position?

Because Search Console assigns every link inside an AI Overview the Overview’s own single position. A page cited in an Overview records a better position while losing clicks, so matching on current position pairs it with genuinely top-ranking pages and hides the effect. The pre-period position reflects where the query sat before the Overview existed.

How many impressions do I need to measure AI Overview CTR impact?

To detect a 30% relative CTR drop from a 2% base at 80% power you need roughly 7,300 impressions per group per period. Smaller effects need proportionally more. Below that threshold the confidence interval is wider than the effect, and the right report is that the impact is not yet measurable.

Is a before-and-after CTR comparison ever enough?

Only as a descriptive. Without a control group it cannot separate the AI Overview from position changes, query mix drift and seasonality that happened in the same window. A difference-in-differences against position-matched control queries is the minimum design that can attribute the change.

What is the difference between AI Overview CTR impact and AI referral traffic?

AI Overview CTR impact is a change in clicks from Google search, measured in Search Console. AI referral traffic is sessions arriving from assistants such as ChatGPT or Perplexity, measured in analytics. They are different surfaces with different data sources and should be reported separately.