Detect Cloaking: When Googlebot Sees Another Page

What cloaking means technically

Technically, cloaking is a branch in the delivery path. The server or an upstream script reads one attribute of the request, usually the user agent, sometimes the peer address, the referrer or the device, and returns a different response depending on the result. Google gives as an example the insertion of text or keywords into a page that only happens when the user agent requesting the page is a search engine rather than a human visitor (Google Search Central). A second pattern is the redirect whose destination depends on the referrer, the user agent or the device (Google Search Central), which is why it can be uncovered with the same means as an untidy redirect chain. Both patterns share one property: they stay invisible during a normal page view, because the operator's browser does not meet the condition under which the deviating response is produced in the first place.

Not every response that differs by request is cloaking. A language branch based on Accept-Language, a mobile version, a paywall or a running A/B test also change the response and are still permissible, as long as they treat search engine and visitor alike. The difference lies in the addressee: permissible adaptation follows what the visitor needs, cloaking follows who is asking. For measurement, that implies a strict rule. You do not compare two arbitrary fetches, but two fetches that differ exclusively in the identifier of the requester. Everything else, meaning language, device width, entry path and time of day, is held constant. Otherwise you measure your own test setup along with it and end up explaining a difference you produced yourself.

• Same identifier, same response: two fetches with the same user agent, the same language and the same region return the same body. If it still differs, the cause is not the visitor but the document.
• Different identifier, same substance: a mobile version may look different, but it has to carry the same statements, links and prices.
• Declared dependency: if the response really does depend on a request field, that belongs in the Vary header. RFC 9110 describes its value as the wildcard member or a list of request field names that might have had a role in selecting the representation for this response (RFC 9110).
• No special path for search engines: a rule that applies exclusively to a crawler identifier is the core of the problem, regardless of whether someone set it deliberately or slipped it in.

Why a break-in looks exactly like this

The most common route to a cloaking website does not run through a marketing decision but through a foreign file in a directory. Google describes the goal of such injections plainly: it is about adding content to your site that search engines can see but which may be harder for you and your users to spot (Google Search Central). The same documentation records that after a break-in an attacker typically uses cloaking to make detection harder (Google Search Central). Search Console describes the same process from the other direction: in the security issues report it says that attackers might hide the spammy content using cloaking techniques to avoid detection by site owners (Google Search Console Help).

This construction creates a measurement problem that many companies notice only weeks later. The home page stays inconspicuous because it is being watched. What is affected instead are deep URLs, often newly created directories that nobody links from the menu. Opening them in your own browser returns an empty page or an error; only a fetch with a crawler identifier brings the injected links to light. Anyone checking the estate at the surface only sees none of this while the index keeps filling up. The same blind spot is familiar to anyone who has investigated a website defacement: the visible part is rarely the part where the damage occurs.

Four fetches that show the difference

The measurement itself is unspectacular. You fetch the same URL several times, change exactly one attribute each time and compare the responses. Four fetches are usually enough to cover the typical case. More important than the number is the order of the questions. First: is there any difference at all? Only then: what causes it? Anyone mixing both questions tends to arrive quickly at a clean explanation for a difference that does not exist.

Two points in this table are regularly overlooked. First, the comparison is only meaningful if it covers the body and not the headers: a differing Set-Cookie or another Date value means nothing. Second, no fetch replaces a look at the server log. That is where you find which response a genuinely verified crawler received, and only there can you spot a branch tied to the origin address rather than the identifier. How to read that address correctly behind proxies and load balancers is a topic of its own, described in the article on the real visitor IP.

Why the identifier alone proves nothing

A user agent is a freely chosen block of text. Google explicitly points out that the HTTP user-agent request header used by Googlebot is often spoofed by other crawlers, and recommends checking a problematic request through a reverse DNS lookup on the source IP or by matching it against the published IP ranges (Google Search Central). Two things follow from that. For measurement it means: your own fetch carrying a Googlebot identifier is a test of the branch, not a fetch by Google. For operations it means: before you book a request in the log as a crawler, it needs verifying, otherwise you attribute foreign traffic to a search engine. How much weight that share now carries, and how to keep it out of your reports, is shown in the article on filtering bot traffic.

The official route consists of two steps with one check each. First a reverse DNS lookup on the accessing IP address from your logs; the returned name has to end in googlebot.com, google.com or googleusercontent.com (Google). Then a forward lookup of that name, whose result must be the same address again (Google). If you do not want to compute that on every request, take the published list: the file with the IP ranges of the common Google crawlers held exactly 317 prefixes on 4 September 2026, of them 170 in IPv4 and 147 in IPv6 (Google). This list changes, so it belongs fetched rather than copied down.

1. Run a reverse DNS lookup on the accessing IP address from the server log.
2. Verify that the returned name ends in googlebot.com, google.com or googleusercontent.com (Google).
3. Run a forward DNS lookup on exactly that name.
4. Verify that the result is the originally accessing address again (Google). Only then is the request attributed; before that it is a claim.

Search Console does not see the same thing

The obvious reflex is URL inspection in Search Console. It is useful, but it answers a different question than expected. According to the help pages, the indexed version report explicitly does not show the live state but the most recently indexed version of a page (Google Search Console Help). The live test next to it does fetch the URL in real time (Google Search Console Help), but with an identifier of its own. That very difference decides whether you get to see a cloaking branch or not.

This identifier is called Google-InspectionTool. Google lists two strings for it: a desktop agent Mozilla/5.0 (compatible; Google-InspectionTool/1.0) and a mobile agent that begins with Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) and also ends in (compatible; Google-InspectionTool/1.0) (Google). For a branch that compares strings the difference matters: a pattern on Mozilla/5.0 (compatible; matches the desktop agent only. Google records that crawling preferences addressed to this user agent affect Search testing tools and have no effect on Google Search or other products (Google). For the cloaking question that is the decisive point: a branch that checks for the string Googlebot does not fire during the live test. The tool reports a clean page while the actual crawl receives something else. The help itself narrows the claim further, because the live test does not check that your page conforms to quality and security guidelines, and it does not cover manual actions, content removals or temporarily blocked URLs (Google Search Console Help).

• What the indexed version shows: the state Google last took in, including canonical URL and indexing decision (Google Search Console Help).
• What the live test shows: whether the URL is reachable and renderable at this moment (Google Search Console Help).
• What neither shows: whether the response differs for the Googlebot identifier compared with the identifier of the inspection tool.
• Why that is: the live test confirms that Google-InspectionTool can access the page, and no more (Google Search Console Help). A clean finding there is a partial statement, not an all-clear.

As continuous observation this route is of limited use as well. URL inspection through the API is capped per site at 2,000 queries per day and 600 queries per minute (Google). For a handful of URLs that is comfortable; for an estate of several thousand pages it is a sample with a ceiling. Continuous observation therefore belongs on your own server, where you set the cadence, the scope and the comparison method yourself. How quietly such a deviation works in the index is shown in the article about an accidentally delivered noindex directive: it only became visible once the positions dropped.

Comparing without false alarms

A dependable comparison has three parts: normalisation, checksum and threshold. Normalisation removes everything that changes between two fetches anyway. The checksum turns the remainder into a value you can store, compare and write into an alert. The threshold decides from which difference onwards an alert is justified. If one of the three is missing, you get either a permanent finding or a blind spot. And a checker that has not fired once is not an all-clear, it is a place where nothing was measured. The article on detecting content changes describes the same pattern for the simpler case without an identifier switch.

import hashlib
import re

NOISE = [
    re.compile(r'name="csrf[-_]token"\s+content="[^"]*"', re.I),
    re.compile(r'nonce="[^"]*"', re.I),
    re.compile(r'\b\d{2}/\d{2}/\d{4},\s*\d{2}:\d{2}\b'),
    re.compile(r'<!--.*?-->', re.S),
]

def normalise(body: str) -> str:
    """Removes everything that changes between two fetches anyway."""
    for pattern in NOISE:
        body = pattern.sub("", body)
    return re.sub(r"\s+", " ", body).strip()

def checksum(body: str) -> str:
    return hashlib.sha256(normalise(body).encode("utf-8")).hexdigest()

def compare(fetches: dict) -> dict:
    """fetches: {identifier: body or None}. None means: not measured."""
    sums = {}
    for identifier, body in fetches.items():
        if body is None:
            sums[identifier] = {"measured": False, "reason": "fetch failed"}
        else:
            sums[identifier] = {"measured": True, "sum": checksum(body)}
    values = [e["sum"] for e in sums.values() if e["measured"]]
    return {
        "sums": sums,
        "deviation": len(set(values)) > 1 if len(values) > 1 else None,
    }

Two details in this skeleton weigh more than the rest. A failed fetch does not get a result but the field measured: false with a reason. A timeout booked as "no deviation" is the most expensive way to feel safe, because it reads like a green result in every report. And the return value for the deviation stays empty as long as fewer than two fetches succeeded: a comparison with a single value is not a comparison but a number with a timestamp.

The comparison also calls for a look at the Vary header. If your response really does depend on the user agent, that field has to be there, otherwise the next cache hands a copy to the wrong recipient. If nothing is there and the responses still differ, you either have an undeclared branch or a false alarm from your own normalisation. Both are worth a look, but only one of them is an incident. If you do not want to run that check by hand, the building blocks are described in the article on automated technical SEO monitoring.

What belongs in the alert

An alert that only says content has changed gets clicked away after the third message. It becomes useful only once it names the difference and delivers the location inside the document with it. The question an alert has to answer is not "did something change" but "which fetch deviates from what, at which spot, and since when". That turns a notification into a basis for a decision, and the recipient can judge within the same minute whether to get up at night or look in the morning.

• Which URL: in full, with protocol and any parameters, not just the path.
• Which two fetches: each with the identifier that was set, in plain text, so the measurement stays reproducible.
• What differs: number of changed lines, new outgoing links with their target host, changed meta data, changed indexing rules.
• Since when: the last point in time at which both fetches matched. Without that value the incident cannot be placed in time.
• What was not measured: failed fetches, timeouts, rejected requests. A report without that section reads like completeness without being one.

The channel this message arrives on determines its value. A finding inside an umbrella report that someone opens at the end of the month does not help during an ongoing break-in. Such deviations therefore belong on the channel that is actually read in your organisation, be that a mailbox, a team chat or an on-call route. In XICTRAQ this is handled by the notifications, which deliver the same finding across several routes and group repeated messages about the same incident.

A checker that has not fired once is not an all-clear. It is a place where nothing was measured. The difference between the two decides whether a report is a basis or a reassurance.

— XICTRAQ

When the finding is real

Once a deviation is confirmed, the order of work changes. First the state is preserved, then it is cleaned up, not the other way round. Deleting the injected file straight away loses the trace that explains how it got in, and the second entry point stays unfound. In practice that second entry point is exactly why the same website shows up again four weeks later.

1. Preserve both fetches unchanged, with headers, timestamp and checksum.
2. Copy the server logs for the affected period before rotation overwrites them.
3. Review and renew access: application accounts, database credentials, deployment keys, extensions with write permissions.
4. Look for the trigger of the branch, not just its output: usually an injected file, altered application code or a rule in the server configuration.
5. Restore from a state known to be clean and then compare the delivered state again with all four identifiers.
6. Only at the end, check in Search Console whether a security notice or a manual action exists, and request a review if needed.

For the last point, the list of manual actions is worth a look. There is a dedicated entry for cloaking and sneaky redirects; the help describes the trigger as your site possibly showing different pages to users than are shown to Google, or redirecting users to a different page than Google saw (Google Search Console Help). Such an action hits you even when the cause was a break-in. That is why removing the cause belongs before the review request, not after it. Which headers, scripts and access paths need checking along the way is covered in a single pass by the security scan.

Measuring continuously instead of checking once

A one-off check answers the question for the moment of the fetch. Cloaking, however, is a state with a date: it begins when a rule is set and ends when it disappears. In between lie the weeks in which the index fills up. The crawl itself also runs at a cadence you do not control. Google gives as an order of magnitude that for most sites Googlebot should not access a site more than once every few seconds on average (Google Search Central), and that it crawls the first 2 MB of a supported file type and the first 64 MB of a PDF file (Google Search Central). Whatever sits below that in the document does not enter indexing. An injected link list right at the bottom can therefore be visible or not, depending on file size, which makes judging a single finding harder still.

The second time axis is robots.txt. RFC 9309 recommends not using a cached version for more than 24 hours unless the file is unreachable (RFC 9309), and prescribes a parsing limit of at least 500 kibibytes (RFC 9309). So anyone changing a rule sees the effect the next day at the earliest, and anyone looking at the file once a quarter notices a slipped-in line only when the damage is already in the index. For context: in the Web Almanac 2024 analysis, 76.6 percent of the robots.txt files found in the desktop crawl address the catch-all user agent *, while 5.9 percent carry a dedicated rule for Googlebot (Web Almanac 2024). A targeted exception for a single crawler identifier is therefore rare enough to stand out, provided somebody is looking. Which mistakes are most common in that file is covered in the article on robots.txt mistakes.

That is exactly what a monitoring system is for. XICTRAQ fetches monitored URLs at a fixed cadence, records the delivered state and reports deviations. SEO monitoring tracks title, description, canonical URL and indexing rules, while the uptime check supplies the cadence and the status code alongside. Especially ahead of periods with elevated traffic it pays to set both up beforehand rather than in the middle, as the article on traffic peaks in the fourth quarter shows.

The organisational gain is larger than the technical one. Anyone who can say at any time which state was delivered under which identifier and when it last changed answers follow-up questions from the record instead of from memory. That holds towards customers just as much as towards a search engine where a review is requested. What such an account costs and which minimum term applies is set out on the pricing overview; anyone wanting to see the workflow first will find it in the live demo without signing up.