AI crawler monitor

The monitor checks whether AI models can read your site at all. This is the foundation layer: if GPTBot or ClaudeBot never reaches the page, no content optimisation helps, because there is nothing to cite.

What exactly we measure

Layer	Question	Points
robots.txt	Does the file allow AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended)?	4
HTTP access	Does the site actually answer bots with a 200, or refuse them with 403/406?	3
Structured data	Does the page carry the schema.org markup relevant to its type?	2
Sitemap	Does sitemap.xml exist and can it be fetched?	1

The score is 0–10, computed only from the layers we managed to measure (see below).

Three states, not two

Every layer can come out three ways — and the third one matters most:

• ✅ measured, fine — e.g. robots.txt explicitly allows GPTBot,
• ❌ measured, there is a problem — e.g. Disallow: / for ClaudeBot,
• ⬜ not measured (grey dot or ?) — your firewall/WAF refused our prober, so we do not know and we do not guess.

„Not measured" never lowers the score: the layer leaves both the numerator and the denominator. If we measured 6 out of 10 points, the score shows „Measured 6/10 pts" underneath. We do not punish you for what we cannot see — but we do not pretend to see it either.

Why this matters

Previously a failed measurement was recorded as a verdict („bots are blocked", „no sitemap"). Such a report sends your team hunting for a problem that does not exist. The third state removes that whole class of error.

How to unblock the audit on your site

The audit knocks on your site from a datacenter server. Bot protections (Cloudflare, Akamai, Imperva) routinely refuse such traffic with a 403 or 406 — and then the layers turn grey.

Recommended: allow the audit's User-Agent

Our own fetches (robots.txt, sitemap, page content) identify themselves as:

GEOPlatformAudit/1.0 (+https://audit.geoplatform.pl)

A rule on this User-Agent is the recommended option, because it does not spoil the measurement. Separately from those fetches we send probes carrying the user agents of the real AI bots — those must stay subject to your actual policy.

Cloudflare: Security → WAF → Custom rules → new rule, condition User Agent contains GEOPlatformAudit, action Skip (tick every security module to skip).

nginx / Apache / ModSecurity: add an exception matching GEOPlatformAudit in the User-Agent header, before your bot-blocking rules.

Alternatively: allow the IP address

For firewalls that cannot match on User-Agent. The current address and UA are always available at a stable URL:

https://audit.geoplatform.pl/api/crawl/audit-info

The endpoint is public (no login) — your IT team can consume it with a script, the same way as openai.com/gptbot.json or Google's lists.

Opening the same host in a browser — audit.geoplatform.pl — shows a page explaining who is fetching your site, how to verify our requests through reverse DNS, and how to block the audit. It is the hostname you will find in your server logs and the target of the +URL in our User-Agent, so you can hand it to your IT team as-is.

You can also simply resolve the address from the name — same value, always current:

dig +short audit.geoplatform.pl

We deliberately do not hardcode the address here: a value printed in the docs would go stale the moment we migrate our infrastructure, leaving you with a firewall rule pointing at nothing. The name audit.geoplatform.pl and the audit-info endpoint are stable — the address behind them can move.

The name is verifiable by forward-confirmed reverse DNS, the way Googlebot is verified (the PTR resolves to the name, and the name resolves back to the same address):

dig +short audit.geoplatform.pl                   # the audit's address
dig +short -x $(dig +short audit.geoplatform.pl)  # → audit.geoplatform.pl

Allowing by IP has a side effect

An IP rule exempts all of our requests, including the probes impersonating AI bots. The „HTTP access for bots" layer will then show access regardless of your real policy. If you care about that measurement being trustworthy, use the User-Agent rule.

What NOT to do

Do not add exceptions for the AI bots' own user agents (GPTBot, ClaudeBot, PerplexityBot, Googlebot). The audit would then report that you let them in — even if you actually block them for the entire world. That layer is meant to measure your policy, not our exemption.

Special case: the sitemap returns 403/406

This is the most common leftover after unblocking the audit: the UA rule already lets the page through, but the sitemap stays grey.

A User-Agent rule usually does NOT help here

We measured this on a real shop. Same sitemap URL, same moment:

Client	Network	Result
our audit (minimal headers)	datacenter	406
full Chrome header set	datacenter	406
real headless Chromium (HTTP/2, browser TLS fingerprint)	datacenter	406
the same fetch	a different network	200, valid XML

Since even a real browser gets a 406, the block does not depend on what the client looks like — what decides is the reputation of the source network. Protections such as Akamai or Cloudflare Bot Management refuse datacenter traffic on selected paths regardless of headers.

So there are two effective levers: an IP rule (resolve audit.geoplatform.pl) or a rule allowing clients that pass reverse DNS verification for that name — the way Googlebot is allowed. Keep the UA rule for the page and robots.txt layers.

This affects AI bots too

GPTBot, ClaudeBot and PerplexityBot also egress from datacenters. If your protection refuses such traffic on the sitemap path and you have no exception for them (verified bots / allow-list), they most likely cannot see it either. That is worth unblocking even if you do not care about our audit — we record „not measured", but the real consequence may be harder.

What we do when we cannot fetch the sitemap

Before recording „not measured", we check the web archive (Wayback Machine). If it has your sitemap, we say so plainly: „exists in the archive (as of DATE), unreachable from our network".

This does not restore the point for the sitemap — the question is „can a crawler fetch it today", and a past snapshot does not answer that. But it turns a blank „we do not know" into something concrete: the sitemap exists, it is simply unreachable from datacenter networks. We skip the archive when the sitemap URLs return clean 404s — then the absence is measured, and a snapshot from before the removal would only mislead.

What else to check:

1. Which URLs we request. First the one declared in robots.txt via Sitemap: https://…, and if there is none, we guess: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /sitemap/sitemap.xml. If your sitemap lives under a different name, declare it in robots.txt — then we go exactly where it is.
2. A rule matching the path or extension. Check whether any rule covers *.xml / sitemap* (Cloudflare: Security → WAF → Custom rules and Managed Rules). The exception for the audit — by IP or by rDNS — has to sit above such a block in the order.
3. A 200 that is not XML. If the sitemap URL returns an HTML page (a typical „soft 404"), we record it as a measured absence of a sitemap, not as a failed measurement — we did get an answer, it just contains no sitemap.
4. A sitemap behind a login or restricted to selected IPs. If you serve it only to search engines, add an exception for the audit as well — otherwise this layer stays grey permanently (it will not lower your score, but you will not learn whether AI bots can see it).

Other pitfalls

1. Rate limiting — the audit sends several requests in a row. A limit returns 429, which we report as „not measured" (it is an artefact of our own pace, not your policy). Disable the limit for the audit.
2. Cloudflare Bot Fight Mode / Under Attack Mode — these modes bypass some ordinary rules, so the exception has to be added there as well.
3. A block above the WAF — if layers stay grey after adding the exception, the traffic is being refused at your CDN provider or on a network firewall in front of the server.
4. robots.txt is not the obstacle here — the audit only reads it to see what you allow AI bots to do. You do not need to change anything in it to unblock the audit.

After adding the exception, come back to the monitor page and click Check now — the grey layers should get a real result and count towards the score again.

Site type and structured data

The markup we expect depends on the site type (Profile → „Site type"):

Profile	Key markup
general (brand, content)	Organization, WebSite, BreadcrumbList, FAQPage, Article
ecommerce (shop)	Product, Offer, AggregateRating, Review, BreadcrumbList, Organization
local (local business)	LocalBusiness, Organization, FAQPage

By default the profile follows your package, but you can override it manually. „Optional" markup never lowers the score — a shop is not penalised for missing LocalBusiness.

For shops the audit additionally scans a sample product page, because Product/Offer live there rather than on the homepage.

Check history

The monitor keeps the full history (the list scrolls, ~10 entries visible). Remove a single entry with the ✕, or the whole history with Clear. A new check starts the history over.