EDGE OBSERVABILITY (CLOUDFLARE ANYCAST + MICROSOFT CLARITY + GA4)
---------------------------------------------------------------------------------
TOTAL REQUESTS PROCESSED : ~1.23M (100% Edge Coverage)
LIKELY HUMAN TRAFFIC : 857,050 (69.63%)
TOTAL AI & BOT ACTIVITY : ~373K (30.37%)
ROBOTS.TXT VIOLATIONS : 0.98% of requests (Clarity Violations card)
CONTENT TYPE SERVED : HTML: 84.18% | Text: 3.17% | JS: 3.13% | Image: 2.59%
REQUEST STATUS : ~46% successful | ~7% redirected | ~48% unsuccessful
AI OPERATOR FOOTPRINT (SELECTED OPERATORS):
• Microsoft (Bing / Copilot) : 43,380 requests (3.52%)
• Google (Gemini / AI Overviews): 42,050 requests (3.42%)
• Meta AI Ingestion : 35,690 requests (2.90%)
• Anthropic (ClaudeBot) : 19,730 requests (1.60%)
• OpenAI (GPTBot / ChatGPT) : 14,900 requests (1.21%)
AUDIENCE SOURCES (BLOGGER PAGEVIEWS vs GA4 USERS, DIFFERENT UNITS):
• Western Europe (NL, DE, FR) : 105,700 pageviews
• United States (Primary B2B) : 47,100 pageviews (3,916 GA4 active users)
• Singapore & Canada : 12,180 pageviews (Singapore: 4,618 GA4 active users)
• United Kingdom : 12,500 pageviews
90-day edge telemetry audit of SEOSiri combining Cloudflare request logs, Microsoft Clarity bot analysis, and Google Analytics 4 data — SEOSiri
The Imperative for Empirical Web Infrastructure Observability
While modern digital strategy frequently relies on third-party estimates, evaluating genuine platform vitality requires direct observability. This 90-day telemetry audit of 1.23M edge requests provides empirical insight into real-world AI crawler behavior, human traffic ratios, and edge infrastructure health across SEOSiri's network perimeter. In the era of Generative Engine Optimization (GEO) and autonomous agent retrieval, understanding platform performance demands direct, edge-level telemetry.
Every time a browser client, an enterprise proxy, or an autonomous LLM indexer requests an endpoint, the interaction leaves a verifiable footprint across the network edge. Over a 90-day observational window, SEOSiri processed about 1.23 million total HTTP requests via our Cloudflare edge perimeter, analyzed with Microsoft Clarity bot analytics and compared against Google Analytics 4 telemetry.
This empirical case study details the architectural metrics of this dataset: dissecting the 69.6% human vs. 30.4% automated traffic ratio, mapping the crawl behavior of frontier AI foundation models (including OpenAI, Anthropic, Microsoft, and Google), and explaining the 0.98% robots.txt violation rate alongside the DevSecOps controls we run across 1.23M requests. This analysis provides actionable technical context for our foundational framework on Enterprise Content Infrastructure as Code (CIaC).
Macro Traffic Composition: The 69.6% Human-to-Bot Equilibrium
A fundamental metric of platform vitality is the ratio between likely human users and automated traffic. Across poorly guarded web properties, scrapers and spam bots can account for a much larger share of total hits, degrading origin performance and exhausting hosting budgets. Conversely, static, abandoned sites experience minimal scraper interest, reflecting near-zero domain discovery.
Out of about 1.23 million edge requests, our telemetry logs reveal a stable equilibrium:
- Likely Human Activity: 857,050 requests (69.63%), likely representing organic browser navigation, technical documentation inspection, and developer asset interaction.
- AI & Automated Tooling Activity: about 373,000 requests (30.37%) spanning AI foundation models, search engine indexers, preview generators, and automated developer tools.
- AI Categories: roughly 200,000 requests, around 16 to 17% of traffic, are classified by Clarity as AI Crawler, AI Search or AI Assistant activity.
- HTML Request Dominance: 84.18% (1,036,029 requests) targeted primary HTML document endpoints rather than static assets. The most requested paths were
/search, the homepage and the Blogger feed, and/robots.txtalso ranks among the top paths.
Maintaining a ~70/30 human-to-automated equilibrium is consistent with a platform that serves human readers while also drawing steady indexing from global software aggregators and AI retrieval bots.
Mapping the Generative Engine Footprint: How AI Models Ingest SEOSiri
The traditional SEO paradigm focused exclusively on Googlebot and Bingbot. Today, in the age of conversational search engines, web platforms must track the ingestion pipelines of frontier AI assistants. Our 90-day telemetry audit recorded roughly 200,000 requests that Clarity classifies as AI Crawler, AI Search or AI Assistant activity. The five operators below are a selection showing continuous machine-level access by leading research labs and platforms. Clarity also lists other operators, including Huawei and Amazon, each ahead of OpenAI in request volume:
| AI Operator / Crawler | 90-Day Edge Requests | Share of Traffic | Likely Function (inferred; logs cannot confirm) |
|---|---|---|---|
| Microsoft (Bing / Copilot) | 43,380 requests | 3.52% | Real-time grounding for Microsoft Copilot & Bing Index |
| Google (Gemini / AI Overviews) | 42,050 requests | 3.42% | Google Search indexing and Gemini synthesis training |
| Meta AI | 35,690 requests | 2.90% | Llama model web training & Meta ecosystem search |
| Anthropic (ClaudeBot) | 19,730 requests | 1.60% | Web crawling for Anthropic's Claude models |
| OpenAI (GPTBot / ChatGPT) | 14,900 requests | 1.21% | OpenAI crawling; GPTBot is documented as its training crawler, with search and user-action bots listed separately |
This ingestion by Anthropic and OpenAI coincides with our standardized publishing protocols, although our logs cannot prove causation: serving token-efficient root-level centralized /llms.txt sitemaps, exposing clean Markdown feeds, and structuring verified endpoints on the SEOSiri Developer Hub. We plan to report how often each operator requests /llms.txt. For engineering teams developing autonomous tool interfaces, structuring endpoints according to our guide on Building Production-Grade MCP Servers helps AI agents parse system documentation with minimal token overhead.
Perimeter Security in Production: Decoding the 0.98% Robots.txt Violation Rate
Processing over 1.2M requests without service degradation requires active perimeter defense. In our audit, Clarity's Violations card, which counts bot requests to URLs our robots.txt disallows, showed 0.98% of total requests. Most automated traffic respected our crawl rules. This is a compliance measure; it does not by itself show how many malicious requests were blocked.
Separately, we run three coordinated edge architectures that protect the perimeter:
- Targeted JavaScript Challenge Rules: As detailed in our empirical investigation into internal search query spam and GA4 analytics poisoning, scraper bots attempt to flood dynamic query parameters. Enforcing managed challenges on paths matching
/searchhelped limit headless browser floods. - Edge-Level Protocol Interception: Serving security disclosure files directly from edge workers—as demonstrated in our tutorial on deploying RFC security.txt with Cloudflare Workers—ensures vulnerability compliance with zero origin compute load.
- Rigorous Cryptographic Header Policies: Maintaining strict HSTS, CSP, and X-Content-Type-Options headers directly at the Anycast edge shields client sessions, reinforcing the trust benchmarks documented in our security headers and GEO ranking case study. Full configuration blueprints are detailed in our companion analysis on DevSecOps at the Edge and Zero-Trust Gateways.
What the Request-Status Panel Shows
Clarity's request-status panel shows roughly 48% of requests as unsuccessful, about 46% as successful and about 7% as redirected. Unsuccessful responses typically include blocked or challenged requests, missing pages and malformed probes. We are breaking these down by path and status code and will publish the results.
Audience Economics: Geographic Distribution & Market Context
For B2B software vendors, cloud platforms, and enterprise digital PR networks, raw traffic volume is irrelevant if it originates from unmonetizable regions. Combining Blogger global system metrics with Google Analytics 4 telemetry shows where traffic and active users come from. The two sources measure different things (pageviews vs. active users) and disagree in places:
| Region / Market | 90-Day Blogger Pageviews | GA4 Active Users & Engagement | Market Context |
|---|---|---|---|
| Western Europe (NL, DE, FR) | 105,700 pageviews | Netherlands: 102 active users, 7s average engagement; Germany and France are not in GA4's top 10 countries | Core European tech ecosystems & data governance hubs |
| United States (Primary B2B) | 47,100 pageviews | 3,916 Active Users (33.6%), 6.36% engagement rate, 3s average engagement | Large enterprise software and cloud market |
| Singapore (APAC Tech Hub) | 8,040 pageviews | 4,618 Active Users (39.6%), 22.6% engagement rate, 4s average engagement | Leading Asian SaaS adoption & developer hub |
| United Kingdom & Canada | 16,640 pageviews | Not in GA4's top 10 countries | Enterprise SaaS purchasing & developer hiring markets |
In Google Analytics 4, the United States and Singapore account for 73.23% of total active users (8,534 active users), generating about 58,100 events. GA4 does not show job roles, and engagement is mixed: the US engagement rate is 6.36% and Singapore's is 22.6%, against a 16.48% site average. We treat this as a starting point for the attribution engineering models established in B2B SaaS Data Pipelines and First-Party Attribution.
Blogger recorded 64.7K pageviews from the Netherlands while GA4 shows 102 active users there, so we treat Blogger's country counts as request volume that may include automated traffic, not as readership. Next, we are excluding internal traffic in GA4, reviewing how key events are defined, and comparing GA4 against Cloudflare country data filtered to likely-human traffic. We will publish the results.
As conversational AI engines and enterprise decision-makers increasingly rely on verified, high-authority platforms for architectural benchmarks and software discovery, establishing contextual presence within verified technical ecosystems becomes a primary growth lever for modern B2B SaaS, cloud, and cybersecurity organizations.
Partner With SEOSiri: Accelerate B2B Demand Generation & Generative AI Discovery
Position your enterprise software, developer tooling, or cloud infrastructure alongside technical articles aimed at software architects, CTOs, and technical marketers, and crawled by AI operators (see the data above). We publish our traffic data and its limits openly. We cannot guarantee how any AI system will use or cite sponsored content.
Integrate contextual mentions into live, indexed technical articles with verified organic authority and search visibility.
Dedicated, long-form architectural reviews and case studies optimized for both human decision-makers and AI citation graphs.
Expose your MCP tools, APIs, and SaaS endpoints across the SEOSiri Developer Hub and open-source assets.
rel="sponsored" / rel="nofollow") under our secure Crawl Control Mode.
Architectural Lessons: How Webmasters Must Build for the AI Era
Operating a web publication in 2026 requires balancing human user experience with autonomous AI ingestion and edge threat mitigation. Based on this 90-day empirical dataset, we recommend four technical mandates for enterprise webmasters:
- Unify Edge Observability with First-Party Telemetry: Relying on client-side analytics tools alone blinds engineering teams to 30%+ of their automated traffic. Implementing edge logs via Cloudflare and Microsoft Clarity provides much fuller visibility into AI crawler behavior.
- Adopt Dual-Format Content Delivery: Publish technical resources designed for human readability while simultaneously exposing lightweight, plain-text Markdown feeds and
/llms.txtfiles for token-efficient LLM ingestion. - Shift Threat Mitigation to Anycast DNS: Block malicious scrapers, synthetic query floods, and bad bot operators at the DNS proxy layer to preserve origin server resources and protect organic crawl budgets.
- Name Your Units and Audit Your Sources: Requests, pageviews and active users are different measures. Label each one, compare sources against each other, and never call any of them "readers" without evidence.
⚡ Modernize Your Platform for the Generative Search Era
Preparing your web architecture for generative discovery requires a cohesive strategy spanning edge security, machine-readable sitemaps, and custom schema engineering. Review our comprehensive blueprint to guide your platform through this paradigm shift:
Frequently Asked Questions: Web Health & Edge Observability
What is a healthy human-to-bot traffic ratio for a technical website?
There is no universal benchmark. In our 90-day logs about 70% of requests were likely human and 30% automated. A higher bot percentage often signals unmitigated scraper abuse, while zero bot traffic indicates poor search and AI engine discovery.
How do OpenAI and Anthropic crawlers discover web content?
OpenAI (GPTBot) and Anthropic (ClaudeBot) discover web content by following links, sitemaps and robots.txt rules. Operators have not publicly confirmed that their crawlers read /llms.txt files or JSON-LD, so treat those as low-cost experiments.
What does a sub-1% robots.txt violation rate signify?
In Clarity, the Violations card counts bot requests to URLs your robots.txt disallows. A sub-1% rate means most automated traffic followed our crawl rules. It is a compliance signal, not a measure of blocked attacks.
Update, October 20: this article was revised after publication to correct the definition of Clarity's Violations metric, clarify what each audience data source measures, and add the request-status findings.
