The robots are reading

Traffic is up. Nobody bought anything, nobody replied to the newsletter, and support is as quiet as it was in December. The graph looks like a win and the month doesn't feel like one.

Before you go hunting for the channel that's underperforming, check who was actually on the site. A decent share of what most analytics tools happily call a visitor is now software.

Two different things, one name

"AI traffic" gets used for two populations that behave nothing alike, and mixing them produces a number that means nothing at all.

The first is training crawlers. GPTBot, ClaudeBot, CCBot, Bytespider, PerplexityBot. They arrive in volume, work through your sitemap methodically, and they do not run JavaScript. That last part is the important one. If your analytics is a script inside the page, those crawlers are invisible to it. They exist in your server logs and nowhere else.

The second is assistant fetches. ChatGPT-User, Claude-User, Perplexity-User. Somebody asked an assistant a question and the assistant went off and read your page in order to answer it. The volume is much lower. The intent behind it is much higher, because there is a real person waiting at the other end, and they might well click through afterwards. These do run JavaScript, so a client-side script can see them.

One is a machine copying you. The other is a person reading you at one remove. Averaging them together tells you nothing useful about either.

What it does to your numbers

Bot requests almost never carry a referrer, so they land in your Direct bucket. Direct traffic climbing steadily with no campaign, no press mention and no launch behind it is the most common early symptom of bot inflation, and it's the one people most often mistake for organic brand growth.

Crawlers also request a page and leave, which is indistinguishable from a human bouncing. Your bounce rate goes up. Your time on page goes down. Neither number changed because of anything a person did.

Then there's the awkward one. If your analytics runs entirely in the browser and reports no bot traffic whatsoever, that is not evidence that you have no bot traffic. It's evidence that your script never ran. A client-side tool cannot see something that never executed JavaScript, so the absence in the dashboard is an absence of measurement, not an absence of crawlers. Server logs are the only place they show up. This is the same reason our cookieless visitor counting has to be explicit about what it can and can't observe.

What Cabin shows you, and what it misses

Cabin has an AI Agents section in the dashboard that splits agent hits out from human ones and breaks them down by operator. It's on the Agency plan, which is $59/mo, alongside the other things agencies tend to need.

Two things are true on every plan, including Free. Agent traffic is never counted in your pageviews, and it is never billed. Your allowance is human visits only, so nobody pays for a crawler.

Now the part a competitor won't tell you. Cabin's tracking script drops any user agent containing bot, spider or crawl in the browser, before the request is ever sent. That was written to keep junk out of human counts, and it does that job well. It also means the high-volume training crawlers never reach our servers to be classified in the first place. So what the AI Agents view mostly shows you today is the assistant traffic, not the crawler flood. The classifier on the server recognises all the big crawler user agents already. They just never arrive.

robots.txt, and what blocking actually costs

If you want crawlers to stop, robots.txt is still the lever:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

Two caveats, and the second one is the expensive one.

Not every crawler respects the file. It's a request, not a firewall, and the ones with the worst manners are precisely the ones ignoring it.

More importantly, blocking training crawlers means your content stops appearing in AI-generated answers. If somebody asks an assistant a question your documentation answers perfectly, and you've locked the door, the assistant recommends whoever left theirs open. For a lot of sites that referral path is growing faster than search. Blocking is a real option, it just isn't a free one, and it's worth deciding deliberately rather than pasting the block in because bots feel like something you should stop.

Which half is it

The interesting question was never how much of your traffic is bots. It's which of the two kinds you're looking at. Crawler volume is a hosting bill and a licensing argument. Assistant fetches are people, arriving through a door that didn't exist three years ago, and they deserve to be measured properly rather than filtered away as noise.

Cabin gives you human numbers you can trust on every plan, no cookie banner needed. If you want the agent breakdown on top of that, it's on Agency.