How to Diagnose an AI Bot Traffic Surge

An AI bot traffic surge and a CMS performance regression present identically and are diagnosed differently, and teams reach for the wrong one first often enough that it is worth writing down. The signal that separates them is not the degree of latency, but the source and legitimacy of the request volume. Establishing that takes minutes; doing so before profiling the application prevents weeks of wasted work in cases where the origin is not the bottleneck.

The first hour usually goes the other way. A client asks why the site feels slow, editors are complaining, and the room has already decided the CMS regressed. On one estate we manage, the site was serving roughly two million requests a day at that point, and real users accounted for a small fraction of it.

Here is the sequence we run, in the order we run it, and the part most teams get wrong.

How Do You Tell a Bot Surge From a Performance Regression?

A bot surge is a volume event, and a regression is a latency event, so the diagnosis starts with which of the two actually moved. Pull total requests and response time per request for the same window, and read volume first.

If volume climbed with no matching change in the application, the problem sits in front of the application. Time per request is the weaker signal: it holds steady while the origin still has headroom, and rises once the surge consumes it, so a large enough surge produces the same latency curve as a regression.

Signals That Separate a Bot Surge From a CMS Regression

Image

Fix the window first, to the hour if you can get it. Then read that window across three systems that have no reason to agree with each other:

• Application performance monitoring (APM), for response time and throughput at the application layer
• CDN analytics, for requests arriving at the edge, including everything that never reached your origin server
• Hosting alerts, for CPU, memory, and connection saturation

Three independent systems inflecting at the same minute is a much stronger signal than any one of them alone. No single indicator settles the question, and combinations do.

Geography usually gets there fastest. Say a site serves an audience in California. When five thousand visits arrive in an hour from data centres in China or Russia, regions where that site has no readership at all, the question is effectively answered before you open anything else. Look at the paths those requests hit, the frequency, and which data centres they came from.

How Do You Tell a Good Bot From a Bad One?

Identification takes four signals read together, because no single one of them holds up alone. The name is the weakest of the four.

• The declared name. Treat it as a claim. I can build a crawler this afternoon, call it Googlebot, and release it onto the web. When ChatGPT first became widely used, we saw a wave of crawlers arriving under borrowed OpenAI and GPT identities.
• The complete user agent string, not the truncated fragment most log views show you.
• The IP ranges it arrives from. A crawler is distributed across data centres, and inside each data centre across servers. Blocking one address accomplishes nothing when the operator has millions of them.
• Your CDN's own verification. Cloudflare's verified bots classification ships out of the box. We do not do anything clever with it. We switch it on, and it tells us whether this is a recognised crawler or something it has never seen before. You do not need an agency for that step.

The well-behaved crawlers make themselves easy to identify. Search engines, the major LLM providers, and the social platforms all identify themselves honestly, and they read the instructions you give them. Google, Bing, Baidu, Yandex, Facebook: these are not the ones that wake anybody up.

Does robots.txt Actually Stop AI Crawlers?

Robots.txt does not stop anything, and it is worth being precise about why, because a lot of policy gets built on the assumption that it does. The Robots Exclusion Protocol is a set of recommendations.

Technically, we cannot compel a crawler to follow the rules written in that file. Reputable operators comply because compliance is in their interest, and they honour noindex and nofollow directives and the headers you set on a page for the same reason.

The crawlers that cause incidents ignore the file completely and hit every page they can reach, as fast as they can reach it, collecting as much as they can before anyone reacts.

When Is the Slowness Genuinely Your CMS?

Sometimes it genuinely is, and ruling that in or out costs five minutes, so do it before you touch your CDN configuration. A deployment from two hours ago can slow a site. So can a deployment from three months ago whose effect only appeared once a table crossed some threshold. Problems do not always surface at the moment they are introduced.

Sites were slow long before AI crawlers existed. What exhausts resources is traffic volume, and the volume does not care which side of the human line it came from.

Open your deploy log and compare it against the window. If something shipped inside it, start there, and this article is not describing your problem.

Why Is the Crawler Sometimes the Symptom Rather Than the Cause?

My view, and it is not the popular one: in some surges we handle, the crawler is doing nothing wrong. The site created far more crawlable surface than anyone intended, then left part of it outside the protection.

Take one estate we manage where the surge reached roughly two million requests a day. Three things produced it. Only one of the three was hostile.

1. A Page Design That Generated Infinite URLs

A single page with filters produced millions of crawlable addresses, and the crawler could not find its way out of them. Select a filter, and the page generates a new URL. Select a different combination, and another new URL. Follow that outward, and the combinations ran into the millions. The crawler sees a fresh page each time and fetches it, then finds more.

This is what we call a spider trap: a page structure that traps a crawler in an effectively unlimited set of addresses. The bot doing most of that fetching was Facebook's. It was declared, it was respecting robots.txt throughout, and it was following links that should never have existed. The fix for that one was in the CMS, in the design of the page itself. No firewall rule would have touched it.

2. Several Hundred Domains Outside the Edge

A web application firewall (WAF) protects only the traffic that routes through it, and on this estate a large number of domains did not. The organisation had bought many additional domains over the years, and most of them pointed straight at the application rather than resolving through Cloudflare.

Traffic arriving on those aliases never passed the WAF at all. The protection was configured correctly. It simply was not in the path.

We resolved it by redirecting every one of those domains through the main domain, so that no route into the origin bypassed the edge. Inventorying every domain and alias that resolves to your origin, and proving traffic actually traverses the WAF, is its own piece of work.

3. Undeclared Crawlers From a Region Unrelated to the Audience

This is the one everybody pictures when they say bot attack. We blocked it at the Cloudflare layer in an afternoon. It was the easiest of the three.

The spider trap and the unrouted domains were both self-inflicted, and neither is visible from a security dashboard. That is why we treat a surge as a question about the estate rather than a question about the incident. Surface you have not enumerated is surface you cannot defend.

What Does a Bot Surge Actually Cost?

The highest cost is reputation, and in my opinion that is not close. A business can absorb a higher bandwidth bill. What it absorbs badly is a slow site during the hours its audience is looking, an intermittent one, or an outage.

Compute and bandwidth come next, and they arrive quietly. If you run autoscaling, every extra request converts into CPU, into memory, and into an invoice at the end of the month that nobody approved.

The cost teams miss sits in metered third-party services. On the two-million-request estate, the infrastructure held: the site never went down.

The site search product bolted onto it did not hold. It bills per request; its plan was sized for roughly 85,000, and the surge drove it past 7.5 million. Containing the traffic meant a conversation with the search vendor to lift capacity while we worked.

Nothing in the infrastructure monitoring would have raised a flag, because from the infrastructure's point of view nothing was wrong.

Before you assume the blast radius is your servers, list everything you pay for per request. Site search, geocoding, translation, personalisation, recommendation engines. Those fail first.

Why Can Blocking Every Bot Cost More Than the Surge?

Blocking all bot traffic is the most common mistake I see, and for a large class of organisations it does more damage than the surge it was meant to stop.

A great many businesses receive almost no direct traffic. People arrive because they searched on Google, and now increasingly through ChatGPT, Claude, Perplexity, and Copilot. Turn off every crawler, and you have turned off Google. Indexing stops, the index goes stale, and the traffic does not come back the moment you lift the rule. One switch, applied in good faith during an incident, can remove a business from the surfaces its revenue depends on.

Nonprofits, news organisations, and any informative site whose readers arrive through discovery sit at the sharp end of this. So do we. Almost nobody types vardot.com into a browser.

The containment move is temporary rather than permanent. Putting the site into a challenge mode that verifies human visitors buys an hour to find the source. That is a tourniquet, not a policy.

Deciding what to block permanently needs someone who knows where the business gets its traffic, which is rarely the person holding the pager at 3 am. It also needs someone who understands that this doesn't finish.

We build something to stop a technique, the operators adapt, and we adapt again. Cat and mouse, and the mouse is well funded. Treat any configuration as current rather than complete.

What Do Cloudflare's Crawler Categories Change?

Cloudflare has replaced its single block-AI-bots switch with three categories, so blocking automated traffic is no longer an all-or-nothing decision. Announced on 1 July 2026, the controls are available on every plan, including the free tier.

Cloudflare's Three AI Crawler Categories

Two changes land on 15 September 2026, and they affect different readers. The first is a default: Training and Agent crawlers get blocked on pages that carry advertising, while Search stays allowed. If your site does not run ads, that default largely passes you by.

The second reaches much further. Multi-purpose crawlers are evaluated under all of their behaviours, with the most restrictive applicable rule winning, and Googlebot, Applebot, and BingBot each crawl for search indexing and for AI training in a single bot. So any site that blocks Training is also blocking Googlebot, whether it runs ads or not. If you clicked the old block-AI-bots switch at some point, that is the setting to open this month.

What Should You Check Before the Next Surge?

Five checks decide whether your next surge gets diagnosed calmly or under pressure. Each is answerable from information you already have, and none of them needs an incident in progress.

1. Do you know what share of your origin requests are automated right now? At origin, not at the edge. If that takes more than ten minutes to answer, you will be finding it out during an incident instead.
2. Does every domain and alias pointing at your origin resolve through your CDN? Count them before you answer. Estates that have accumulated campaign domains, retired brands, and country variants routinely have routes nobody has examined in years.
3. Does your site generate URLs combinatorially? Faceted filters, calendar views, sort parameters, internal search results. Any of them can turn a well-behaved crawler into a load event.
4. Which of your services bill per request? These break before your servers do, and your infrastructure monitoring will not tell you.
5. Has anyone decided which crawler categories you serve, and does that decision exist anywhere outside the operations team's heads?

Answer no to the first two and your next surge gets diagnosed by whoever happens to be available, which is when the blanket block starts looking sensible. Answer no to the last one and the decision still gets made. It gets made at 3 am, by one person, with nothing to refer to.

Protection is layered. The edge (CDN and WAF), the server (caching rules and IP range blocks), and the application each stop something the other two cannot, so a complete answer covers all three.

What Should You Fix First?

Start with the domain inventory: establish what actually points at your origin, and how much of it sits behind the edge. Teams most often cannot answer that question, and the answer is what makes every other control meaningful. Most of the rest you can do yourself. The Cloudflare verified-bots setting genuinely is a toggle.

At Vardot, we run managed infrastructure for enterprise Drupal estates, and we also audit estates we do not host, so this recommendation does not depend on moving your hosting to us. If the inventory does not exist for your estate, a crawl surface review is a smaller and more useful first step than a hosting conversation.

Vardot is a Drupal Diamond Certified Partner with 200+ platforms launched, a 4.9/5 Clutch rating across verified reviews, a top-20 Drupal contributor worldwide, and a Gold Sponsor of the Drupal AI Initiative.