The classification problem
A request to a web server proves that a URL was fetched. It does not prove that a person saw the page, read it or considered the offer. Search crawlers, AI fetchers, uptime monitors, link-preview bots and security scanners can all create requests that resemble traffic in a raw log.
The reporting error begins when every request is pushed into a single visitor total. A crawler can request many URLs in seconds, ignore JavaScript, never accept consent storage and never submit a genuine enquiry. Counting that activity as sessions inflates reach and makes conversion rates meaningless.
| Event | What it supports | What it does not support |
|---|---|---|
| Bot request | A known or suspected automated agent accessed a URL. | A human visit or commercial interest. |
| Browser session | A browser loaded the site under the analytics rules in use. | That the visitor came from an AI answer. |
| Observable referral | The session retained an approved referrer or campaign marker. | Every earlier research touchpoint. |
| Lead event | A form, call or booking reached the business. | Qualification or revenue until reviewed. |
Build two reporting lanes
The operational lane should retain timestamp, path, status, user agent, IP-verification result when available and a bot class. The commercial lane should retain consented session identifiers, acquisition context, landing page and meaningful user events. The two lanes may share a timestamp and URL, but their totals should never be added together.
A practical classification sequence
- Preserve the raw server request before applying a label.
- Match documented crawler user agents, then verify network identity where the provider publishes a verification method.
- Classify obvious automation such as monitoring and link previews separately from search or AI fetchers.
- Mark unresolved agents as unknown automation instead of forcing them into a known brand.
- Exclude classified automation from human session and conversion denominators.
- Review the rules after new user agents appear or traffic patterns change.
Any requester can copy a crawler name. For important investigations, follow the provider's documented IP or authentication checks rather than trusting the string alone.
A one-day QA exercise
Select one production day and compare three totals: server requests, analytics sessions and verified enquiries. Inspect the largest user-agent groups, unusually fast multi-page sequences and requests that never load normal browser assets. Then document which events enter each report and who owns exceptions.
The useful result is not a dramatic bot number. It is a reproducible rule: crawler activity reports access and discovery operations; human analytics reports observable visits; the CRM reports enquiries and qualification.
Primary reference
Google publishes crawler and fetcher identities together with verification guidance. Use the equivalent first-party documentation for every provider you classify, and retain a dated copy of the rule set used in each report.
Where to go next
Use the Nexus AI Visibility Framework to place this topic inside a complete measurement system, or review the AI Visibility Audit scope.
13 Jul 2026 — Expanded with an independent structure, examples, implementation guidance and primary references.