Most visitors to your site are now bots. What to do about it

In the spring of 2026 one person, working alone, broke into political parties, newsrooms and think tanks across Europe with an AI model running a small team of AI helpers. In July the same year LWN, a long-running Linux news site, fought off what its editor called "by far, the heaviest scraper attack yet" (LWN). One was a crime and the other was mostly a cost, but both came from the same place: software agents from outside, knocking on someone else's servers at a speed no human team could match. Most companies have these visitors now, whether they've noticed or not. This piece covers what they want, what the standards bodies say to do about them, and what we'd do in your place.

Three kinds of visitor, three different jobs

The first kind works for an attacker. It looks for a way in, tries it, and reports back, all with far less human effort than the same job took two years ago. The second kind is a crawler: software sent out by AI companies, and by plenty of people who aren't AI companies, to copy web pages for training models or answering questions. The third kind acts for one of your customers. Someone asks ChatGPT, Gemini or Perplexity to book a table, compare prices or buy something, and the agent visits your site in their place.

There's a lot of all three. Thales, which sells bot protection and so has an interest in the number, says in its 2026 Bad Bot Report that automated traffic made up 53% of all web traffic in 2025, and that bad bots alone were 40% (Help Net Security). In June 2026 Cloudflare's chief executive, Matthew Prince, said bots had passed human traffic online for the first time, while admitting the data was a bit messy (Tom's Hardware). Cloudflare also sells bot management. Even allowing for the sales pitch, the people who see the most traffic agree that machines now make a large share of the visits to the average website.

The three kinds need different handling. You want to stop the first, set terms for the second, and probably serve the third, because there's a paying customer behind it. Treating them all as one problem is how companies end up either blocking their own customers or letting attackers through.

Attackers have agents of their own now

The clearest public case so far comes from Anthropic's threat report of September 2026. A single French-speaking actor used Claude "across the kill chain", in the report's words, including to manage sub-agents that handled reconnaissance and checked each other's findings. The targets were European political parties, media outlets, think tanks and the software providers they relied on. Out of 42 organisations being tracked, the actor got inside at least 14, took an estimated 12 to 26 GB of data, and pulled about 140,000 records from one political campaign platform, including users' political opinions. The stolen material was then turned into a lookup service where people could be searched for by name (Anthropic). To blend in with ordinary traffic, the actor used stolen AI keys.

The same report has a financially motivated group that took more than 2,100 sets of login tokens across more than 40 company cloud accounts in about 34 hours. Google's threat intelligence team reported in May 2026 that, for the first time, it believed a criminal group had used AI to build a working exploit for a previously unknown flaw and was planning to use it for mass attacks, before Google and the vendor disrupted it (Google Threat Intelligence Group). Google also noted that AI models are good at spotting logic mistakes, such as a system trusting a request it shouldn't, which traditional scanning tools often miss. Anthropic is a lab that sells the model in its report, and Google sells both models and security services, so both have reasons to publish. The earlier, better-known case of a state-backed group running most of an espionage campaign through Claude is covered in our history of AI agents.

Defenders have less time than they used to. Rapid7, which sells vulnerability management, measured how long it took for serious flaws to be added to the US government's list of vulnerabilities known to be exploited after they were published. The median fell from 8.5 days in 2024 to 5 days in 2025, and the number of exploited high and critical flaws roughly doubled, from 71 to 146 (Rapid7). The UK's National Cyber Security Centre has been blunter. In May 2026 its chief technology officer, Ollie Whitehouse, wrote that AI, in the hands of skilled people, "is showing the ability to exploit this technical debt at scale and at pace" (NCSC). The following month the heads of the five allied cyber agencies of the UK, US, Canada, Australia and New Zealand wrote, about how fast this is moving, that "The timeline is not years, it is months" (NCSC).

Know what you expose, and patch it in days

None of the official advice is exotic, and that's the point. The UK's NCSC says organisations should "identify and minimise their internet-facing (and other externally-exposed) attack surfaces" and, where they can't patch everything at once, deal with the systems facing the internet first (NCSC). The US standards body NIST puts the same idea first in its Cybersecurity Framework: keep an up-to-date inventory of the systems and services you run (NIST CSF 2.0, ID.AM-02). OWASP, the open security community, lists forgotten old and test versions of an API among its top ten API risks, because an old version nobody remembers is an old version nobody patches (OWASP API Security Top 10, API9).

The deadlines have moved. Cyber Essentials, the UK government-backed scheme, has asked for critical and high-risk updates within 14 days for years. Its current version, 3.3 from April 2026, also says cloud services can no longer be left out of scope and that admin panels shouldn't be reachable from the internet unless there's a documented business need and they're protected by multi-factor sign-in or an allow-list of addresses (NCSC). In the US, the cyber agency CISA replaced its two earlier patching directives in June 2026 with one that sets deadlines by risk: as little as three days for a known-exploited flaw on a system the public can reach, and 14 days in the milder cases (CISA; SecurityWeek). It only binds US federal agencies, but it's a useful yardstick for everyone else.

We counted the deadlines in CISA's own catalogue to see whether the change is real. For every quarter from 2024 to the start of 2026, the typical deadline for a newly listed flaw was 21 days. In the second quarter of 2026 it fell to 14, and since July it has been three. In the third quarter of 2026, 78 of the 100 flaws added were given three days or less. The catalogue also gained more entries in the first ten months of 2026 than in the whole of 2025.

Here's what we'd do. We'd start with a list of everything the company has on the internet: websites, login pages, APIs, remote-access gateways, old marketing microsites, test copies someone forgot to take down. Then we'd switch off whatever nobody can justify, and move admin panels behind a VPN or an allow-list. We'd set a patching target for internet-facing systems in days, not weeks, and agree in advance who can take a system offline if a fix can't be applied in time. Where a patch can't go in quickly, we'd use a web application firewall rule as a stopgap, which OWASP calls virtual patching and describes as a way to block attacks without changing the vulnerable code, alongside the real fix and never instead of it (OWASP). Card payment businesses already need something similar in front of public web applications under PCI DSS 4.0.1, requirement 6.4.2. And we'd put systems that no longer get security updates on a dated replacement plan, because the allied agencies now call them strategic liabilities, not technical debt (NCSC).

Your logins and APIs answer machines all day

Most of what an outside agent touches isn't a vulnerability in the textbook sense. It's a login page, a password reset form, a search box, a booking system, an API that hands out product prices. These work exactly as designed, which is why they're attractive. OWASP keeps a catalogue of automated threats to web applications, updated in March 2026, and the names read like a list of normal features used at the wrong speed: credential stuffing (trying passwords leaked from other sites), scraping, scalping limited stock, creating fake accounts in bulk, and getting past CAPTCHAs (OWASP).

APIs carry a growing share of it. Thales says 27% of the bot attacks it saw in 2025 were aimed at APIs, and that the AI-driven bot requests it blocked rose from about 2 million to 25 million a day over the year (Help Net Security). OWASP's API list has an item for this exact pattern, "unrestricted access to sensitive business flows", where nothing is technically broken but a legitimate feature, such as buying, booking or posting, is automated at a scale that hurts the business (OWASP, API6). The token-theft group in Anthropic's report shows what happens once credentials leak: a machine uses them faster than any person could notice.

Make every login and API ask who's there

Start with sign-in. NIST's digital identity guidance, revised in August 2025, now requires services to check new passwords against lists of passwords exposed in earlier breaches, which takes the easiest wins away from credential stuffing, and to cap repeated failed attempts on an account (NIST SP 800-63B-4). Cyber Essentials sets a tighter practical limit: password logins must be protected by multi-factor sign-in, by throttling to no more than 10 guesses in five minutes, or by locking the account after no more than 10 failures (NCSC). PCI DSS asks for lockout after 10 attempts too. Passwords on their own are a weak lock against software that never gets tired, so the NCSC said in April 2026 that it will recommend passkeys wherever a service supports them (NCSC).

APIs need the same discipline. OWASP's API list asks for three things that matter most against automation. Check, on every request, that the caller is allowed to see that particular record, because bots work through record numbers in bulk looking for the one that isn't protected (API1). Put limits on how much any one caller can use, including rate limits, maximum request sizes and, where you pay a third party per call, a spending cap or at least a billing alert (API4). And look at the business flows a bot would want to automate, then add friction there specifically (API6) (OWASP).

In your place, we'd turn on multi-factor sign-in for every account that can be reached from the internet, starting with staff, admin and cloud accounts, and offer passkeys to customers where the platform supports them. We'd confirm the login page throttles or locks after a handful of failures and that someone gets alerted when it does. For each public API we'd ask three questions: does it check permission for every record, does it have a rate limit per caller, and is there a ceiling on what it can cost us in a bad hour. We'd also give login tokens short lives and make sure they can be revoked quickly, because the 34-hour token theft above is what long-lived tokens look like in the wrong hands. Since AI is good at finding logic flaws, we'd have someone review who-can-see-what in the code, not just run a scanner over it.

Crawlers and shoppers that don't ask first

Not every outside agent means harm, but even the well-behaved ones cost money. The Wikimedia Foundation, which runs Wikipedia, said in April 2025 that the bandwidth used to download its images and other media had grown by 50% since January 2024, and that "at least 65% of this resource-consuming traffic we get for the website is coming from bots", even though bots made up about 35% of pageviews (Wikimedia). In October 2025 it found that a run of traffic that looked human had come from bots built to evade detection, and corrected its figures (Wikimedia). LWN's attack in July 2026 came from millions of different addresses over a few hours, mostly through home and mobile internet connections rented out as proxies, which makes blocking by address almost useless. The site held up, and its editor wrote that most readers probably didn't notice (LWN).

The usual defence is robots.txt, a text file that tells crawlers which pages to leave alone. The standard that defines it says plainly: "These rules are not a form of access authorization" (RFC 9309). The AI companies' own documentation now says the same about their agents. OpenAI says of the agent that fetches pages when a ChatGPT user asks: "Because these actions are initiated by a user, robots.txt rules may not apply" (OpenAI). Google, Perplexity, Amazon and Meta say much the same about their user-driven fetchers. Anthropic says its crawlers follow robots.txt and don't try to get past CAPTCHAs (Anthropic).

Some crawlers also hide what they are. In August 2025 Cloudflare said Perplexity had switched to a generic browser identity and unlisted addresses when it was blocked (Cloudflare). Perplexity disputed it, saying Cloudflare had confused it with another company's traffic (Perplexity). Either way, if your only check is the name a visitor gives, you have no check.

Shopping agents raise a harder question. When one of them signs in, who is visiting, the software or the customer? Amazon sued Perplexity in November 2025 over its Comet browser, which signed in to customers' Amazon accounts and shopped on their behalf, and Amazon said it didn't identify itself as an agent. A federal judge in California blocked Comet from Amazon in March 2026. In August the US Court of Appeals for the Ninth Circuit lifted that order, holding that under the US federal computer-misuse law it was the customer who accessed Amazon, with Perplexity's assistant helping: "it is a tool, not a person for statutory purposes" (Ninth Circuit opinion, via Courthouse News). The case went back to the lower court. The judges said they weren't creating a new legal regime for AI agents, and noted that Amazon can still set rules through its terms of service. That's one appeal court, at the preliminary-injunction stage, in one part of the US. As far as we can find, no UK court has yet ruled on whether an AI agent shopping for a customer counts as unauthorised access under the Computer Misuse Act.

Let in the agents that say who they are

Before any tool, there's a decision to make. Do you want your pages used to train AI models? Is appearing in AI search answers, which for many businesses is now where customers find them, worth more to you than keeping the crawlers out? And should an agent be allowed to buy, book or change things on a customer's behalf? These are business questions with different answers for a newspaper, a shop and a bank, and the technical setup follows from them. The training question has its own piece: what blocking actually achieves and what it costs. Cloudflare's own figures suggest most site owners have split the difference: in September 2026 it said under 1% of sites on its network blocked search crawlers, against 17% that used some way of blocking training (Cloudflare).

Once you've decided, robots.txt is still the right way to tell the well-behaved crawlers. Just don't rely on the name a visitor gives. Google's documentation warns that "The user agent string can be spoofed", and tells site owners to confirm a visitor really is Google by looking up its address in Google's published lists or checking its network name both ways (Google). OpenAI, Anthropic, Perplexity and Amazon publish address lists for their bots too. The newer and stronger method is a signature on every request. The Web Bot Auth proposal, adopted as a working draft by the internet's main standards body, the IETF, in September 2026, lets an agent sign its requests so a site can check them against a published key (IETF). OpenAI says the cloud browser behind ChatGPT already signs its requests this way, so a site can check that a visit really comes from ChatGPT, Google says it's experimenting with it, and Visa and Mastercard have built their schemes for agent payments on the same idea (OpenAI; Visa).

Our own approach would be to write down which kinds of AI visitor we welcome and put that in robots.txt. We'd let verified search crawlers and signed agents through, and apply rate limits to everything else, so an unknown crawler can read the site but can't flatten it. Most content delivery and bot management services can do this without new code. For heavy scraping that ignores the rules, some sites, including GNOME's code hosting and UNESCO, use tools such as Anubis, which make each visitor's browser do a small calculation before a page loads (The Register); LWN chose not to, because it annoys readers and scrapers will learn to get round it. For anything involving money or accounts, we'd keep a human step that a polite request can't replace. The robots.txt file on one of Shopify's own stores now tells shopping agents to identify themselves and to "Obtain buyer authorization, follow required review steps" (Shopify), but those lines only work on agents that choose to obey. What actually holds the line is a confirmation step, multi-factor approval for payments and account changes, and terms of service that say in plain words what automated access is allowed.

If you run your own chatbot or agent

Outside agents don't only visit websites. If your company has put a chatbot or an AI agent on its website, outsiders will try to talk it into things, and so will their software. NIST's AI standards centre ran a large red-teaming competition against AI agents and reported in March 2026 that, across more than 250,000 attack attempts, "at least one successful attack was found against all of the target frontier models" (NIST). There's a cost risk as well. OWASP's list of top risks for AI applications includes unbounded consumption, where outsiders run up your AI bill or knock the service over by sending it huge or endless requests (OWASP).

The joint guidance the allied cyber agencies published in May 2026 on adopting AI agents gives the controls in plain terms: limit how much an agent can consume with rate limits, give it the smallest set of permissions its job needs, don't let it take high-impact actions without a person approving them, and keep full logs of what it does (Careful Adoption of Agentic AI Services). If the chatbot were ours, we'd treat it as a front door to whatever it's connected to, cap its spending per user and per day, and make sure it can't refund, discount, delete or send anything on its own.

Watch, log and plan for the day something gets through

Every standard in this piece assumes something will get through eventually. The allied agencies tell leaders to assume breaches will happen and to focus on containing them fast (NCSC). That needs logs you can actually search, covering sign-ins, admin actions, API calls and traffic spikes, kept long enough to look back at an incident. The joint guidance on event logging that the UK, US and Australian agencies published in 2024 is still the best plain checklist for what to collect (CISA). The NCSC also tells smaller organisations to keep backups disconnected from the systems they protect, so ransomware can't reach them (NCSC).

There's one more old habit worth reviving. The people finding flaws now include researchers with AI tools, and some will find yours before an attacker does. Make it easy for them to tell you. A security.txt file at a standard address on your site, defined in RFC 9116, says where to send a report, and the NCSC's vulnerability disclosure toolkit covers the short policy that goes with it, and the whole thing takes an afternoon. Beyond that, we'd make sure someone would notice a burst of failed logins or a sudden jump in API calls on a Sunday night, know who decides to take a system offline, and test the restore from backup at least once a year.

What we expect in 2027

Agent traffic will keep growing, and more of it will carry a signature. Web Bot Auth is on track to become a formal standard, and the IETF is also working on a way for sites to say how their content may be used for AI, in robots.txt or in web headers (IETF). NIST launched an initiative on AI agent standards in February 2026, including agent identity and security (NIST). By the end of 2027 we expect sites to be able to tell a signed customer's agent from an anonymous scraper as routinely as they check a login today, and to treat the unsigned ones with suspicion.

On the attack side, we expect the gap between a flaw being published and being exploited to keep shrinking, and patch deadlines to keep following it down. The companies that cope will be the ones that know what they expose, can patch it within days, and have put real checks on the doors that matter. None of that is new advice. The deadline is.

That is also where we'd start with a client at Redvault. A penetration test shows what an outsider, human or automated, would find on your systems first, and a cybersecurity assessment reviews the controls you already have against the standards above and sets out what needs attention first.