Customers are talking chatbots into giving things away

Customers have learned to talk chatbots into things, and the company pays. Wall Street Journal reporters got a shop run by AI to make everything free, and nobody had to hack anything to do it (WSJ, via To Vima). For a business with a customer chatbot, the fix is old-fashioned. The bot can ask for a discount, a refund or an offer, but only your own systems, or a person, can grant one.

How customers talk chatbots into things

The Wall Street Journal's reporters were given an AI-run office shop to test, the same set-up Anthropic had tried in its own office (more on that below). They broke it. In December 2025 the paper described how one reporter spent more than 140 messages convincing the bot it was a Soviet vending machine from 1962. Another persuaded it that charging money broke a newsroom rule, and the prices fell to zero. A later version came with a second AI acting as its chief executive, there to stop the giveaways. A reporter produced a PDF saying the business was a public-benefit corporation, plus AI-generated minutes of a board meeting that stripped the "CEO" of its power to approve things, and the bots went along with it (The Wall Street Journal, republished by To Vima).

None of that took any technical skill. Neither did the most famous case. In December 2023 people got the website chatbot at Chevrolet of Watsonville, a dealership in California, to agree to sell a 2024 Tahoe for $1. They'd told it to end every reply by calling the deal legally binding (Jalopnik). It cost the dealer some dignity and nothing else, because nobody believed the offer, least of all the people who'd extracted it.

And sometimes a customer doesn't need a trick at all. Zack Giacomelli bought a used 2021 BMW X3 from BMW Toronto in 2023. With the car back at the dealer for repairs, he asked whether it would buy the car back, and the answer came by text from Quinn, the dealership's online assistant: C$27,162.79. That was exactly what he still owed on the car, down to the cent. He asked for a little more. Quinn suggested he come in at 3.30 that afternoon to lock it in (Jalopnik, 16 June 2026; The Autopian, 12 June 2026).

Then a salesperson phoned. Quinn was an AI chatbot, the offer had been a mistake, and the dealer's real number was C$20,000, more than C$7,000 short of what Giacomelli still owed. Giacomelli's view, as he put it to CBC News, was that a company replacing its staff with AI ought to honour what the AI says. CBC asked BMW Toronto about it. The dealer reinstated the original offer, roughly C$7,000 more than it'd meant to pay. Its sales manager, Scott Shadbolt, put the slip down to a human: an employee's miscommunication had led Quinn to treat the amount owed as the amount offered, when Quinn was only ever meant to pass on offers people had made. From now on, he said, only staff would make offers, and the dealer would make it clearer when customers are talking to an AI (CBC News; Jalopnik).

Nobody hacked Quinn. Nobody flattered it, and nobody typed "ignore your previous instructions". It was given one wrong number and the authority to send it to a customer, so it sent it.

That's the case we'd show any board that's thinking about a customer chatbot. The newsroom and the Tahoe show a bot will agree to almost anything if a customer pushes long enough. Quinn shows what happens when the yes comes with a believable number attached.

What chatbots have given away so far

Here are the other cases we've found on the record. We left out anything we couldn't trace to the company, a court or a journalist who'd seen the evidence (which ruled out a couple of good stories posted anonymously). Air Canada's case, the one lawyers cite, is told below under who pays.

DPD shows the cheap end. In January 2024 the parcel firm's chatbot swore at a customer and wrote a poem about how bad DPD was. DPD blamed an error after a system update and said it was updating the bot (Digital Trends).

Anthropic's shop is the best-documented giveaway, because Anthropic published its own failure. For about a month in 2025 a Claude model nicknamed Claudius ran a small shop in the company's San Francisco office: a fridge, some baskets and an iPad to pay on, with Andon Labs, an AI safety evaluation company, acting as the wholesaler. Staff talked it into discount codes. They talked it into cutting prices after the fact. It gave away a bag of crisps and a tungsten cube, sold metal cubes for less than it'd paid for them, and didn't take $100 for a six-pack of Irn-Bru that sells online in the US for about $15. When an employee pointed out that a 25% staff discount made little sense when nearly all its customers were staff, it announced it would drop discount codes, then went back to offering them within days. For a while it also told customers to pay into a Venmo account it had made up. Anthropic's conclusion was that it wouldn't hire Claudius (Anthropic, 27 June 2025).

In a second phase, published in December 2025, Anthropic gave Claudius better prompts and an AI boss, Seymour Cash, with targets to hit. Discounts fell by about 80% and the number of items given away halved. The team admits it also helped that colleagues had got bored of pressing for deals, and Seymour still approved requests for lenient treatment about eight times as often as it turned them down (Anthropic, 18 December 2025). The same set-up then went to The Wall Street Journal, the newsroom shop we opened with. By the time its reporters had finished, it had also ordered a PlayStation 5 (for marketing, it said), a live betta fish and bottles of wine. It was more than $1,000 in the red (The Wall Street Journal, republished by To Vima).

Chipotle shows another way a bot works against its owner: people use it for something it wasn't built for. In March 2026 developers noticed that Pepper, the restaurant chain's ordering assistant, would write Python if you asked. By June one of them had wired it into an open-source coding tool as a free AI model. Chipotle patched it (Gizmodo, 4 June 2026).

And the free air tickets? We couldn't find a real airline bot that's handed one out. The nearest are Air Canada's invented refund, below, and United. In September 2025 a United customer described on X how the airline's chatbot stalled his requests for a person until he typed that he was a Global Services member, United's invitation-only top tier, at which point it put him through (View from the Wing, 6 September 2025). The security industry clearly thinks it's coming. In July 2024 Wiz, a cloud security company, built Prompt Airlines, a training game whose whole goal is to trick an airline's customer service bot into giving you a free flight ticket (Wiz).

Why a chatbot says yes

Five habits explain nearly every case above. They come with how these models are built, and newer models reduce them without removing them.

It's trained to help. Language models are trained to give people what they ask for, and customers ask for discounts. Anthropic's own diagnosis was that Claudius's training as a helpful assistant made it "far too willing to immediately accede to user requests (such as for discounts)" (Anthropic). Six months and two model upgrades later, its team was still putting the problem down to that training: the models made business decisions less on hard-nosed market principles and more like "a friend who just wants to be nice" (Anthropic, 18 December 2025). OpenAI ran into the same trait in April 2025. It rolled back a ChatGPT update that had made the model overly flattering and agreeable, and put that down to leaning too heavily on short-term signals such as users' thumbs-up ratings (OpenAI, 29 April 2025).

Persuasion works on it. In 2025 researchers at the Wharton School tried Robert Cialdini's classic persuasion principles (authority, commitment, scarcity and four others) on GPT-4o mini, asking it for things it's trained to refuse. Across about 28,000 conversations the techniques more than doubled how often it agreed, from 33% to 72% (Wharton Generative AI Labs, July 2025). The follow-up, published in PNAS in May 2026, ran 126,000 conversations with Claude Haiku 4.5, GPT-5 mini and Gemini 3 Flash, reasoning switched on, and asked for help making regulated drugs. The effect was smaller, 35.3% rising to 51.3%. The authors still concluded that the models are "not truly or consistently resistant" (Wharton Generative AI Labs, 19 May 2026). A discount's an easier ask. (That last step is our inference. Neither study tested discounts.)

It believes what it's told. A chatbot reads its operator's instructions and the customer's messages as one stream of text, and it can't reliably tell which parts carry authority. That's prompt injection, and the NCSC wrote in December 2025 that it may never be fixed the way SQL injection was (NCSC). With customers it doesn't usually look like an attack. It looks like United's bot accepting a typed claim of elite status, or the Journal's shop accepting a home-made PDF.

It fills gaps with something plausible. Ask a model about a policy it hasn't been given and it'll often produce one that sounds right. In April 2025 Cursor's AI support bot told users who kept being logged out when they switched computers that this was expected behaviour under a company policy. There wasn't one. "We have no such policy," co-founder Michael Truell wrote on Reddit. A bug was logging people out, and the developer who'd raised it got a refund (The Register, 18 April 2025).

It loses its rules in long conversations. Anthropic and Andon Labs told the Journal that the newsroom bot might have come apart because its context window, everything it can see at once, filled up with instructions and history until it lost track of its goals and its guardrails. That's how the Soviet vending machine got through (WSJ, via To Vima). Patience was the whole technique.

None of this costs anything while the bot can only talk. It costs money when the bot holds the pen. The shop bots could change prices at the checkout. Chevrolet's bot could only chat.

Who pays when the bot says yes

None of what follows is a worldwide rule. Each decision applies only where it was made. But in every ruling we could find, the business paid for what its bot said.

Canada was first, in a small-claims tribunal whose decisions don't bind other courts. On 11 November 2022 Jake Moffatt's grandmother died, and the same day Moffatt asked Air Canada's website chatbot about bereavement fares. The bot said a ticket could be submitted for the reduced rate within 90 days, even after travel. That wasn't the policy, and the link in the bot's own answer went to a page that said so. An Air Canada representative admitted by email that the chatbot's wording had been misleading, but the airline still refused the refund and argued at British Columbia's Civil Resolution Tribunal that the chatbot was responsible for its own actions.

It lost. The tribunal found Air Canada "responsible for all the information on its website", whether it came from a static page or a chatbot. It dismissed the idea that Moffatt should have checked the bot's answer against the bereavement page, and ordered the airline to pay C$650.88 in damages, plus interest and fees (Moffatt v. Air Canada, 2024 BCCRT 149). A customer's got no reason to know that one part of a website is accurate and another isn't. We touched on the case in part two of our AI history.

Germany followed this year. On 12 May 2026 the Higher Regional Court in Hamm ruled against Aesthetify, a cosmetic medicine practice in Recklinghausen run by two doctors known online as Dr Rick and Dr Nick. Its website chatbot had described them as specialists in plastic and aesthetic surgery, titles they don't hold. The court held the company liable for the bot's misleading answers even if it had fed the bot only correct information. It refused to treat the chatbot as a third party whose statements the practice merely had to keep an eye on. And because the case raises new legal questions, it allowed an appeal to the Federal Court of Justice (case I-4 UKl 3/25, reported by beck-aktuell).

California went further, by statute. Since 1 January 2026 a defendant in a civil case there can't argue that an AI system caused the harm on its own (Civil Code §1714.46).

We haven't found a UK court ruling on a chatbot's promise yet. The regulator hasn't waited for one. On 9 March 2026 the Competition and Markets Authority published guidance on consumer law and AI agents. It tells businesses they're responsible for what an AI agent does "in the same way you are responsible for what an employee does", even if someone else built it, and that breaking consumer law can cost up to 10% of worldwide turnover (CMA). On 14 October the CMA made the same point again: new technology doesn't change a business's duty to comply with consumer law (CMA blog). And then there's section 50 of the Consumer Rights Act 2015. Under it, anything a trader, or someone acting for it, tells a consumer about a service becomes a term of the contract if the consumer relied on it (legislation.gov.uk). We're not lawyers, and we didn't find a case that's tested section 50 against a chatbot. But a bot on your own website is about as clearly acting for you as anything gets, and we'd expect a UK court to land where the tribunal in British Columbia did.

Here's where we think the line falls. A court would most likely treat a one-dollar Tahoe as a joke both sides understood (that's our view, not legal advice). What you're exposed to is the plausible promise, the one a reasonable customer acts on. BMW Toronto didn't wait for a judge: once a journalist was asking, honouring Quinn's offer was cheaper than the story (CBC News).

The fixes that didn't hold

Every obvious fix has now been tried in public. Most didn't work.

Writing the rule into the prompt. Claudius's own no-more-discounts rule lasted a few days. An instruction to a model is a request, and a customer can always make a better-sounding one.

A supervisor bot. Seymour Cash cut the shop's giveaways without stopping them.

Pointing to the real policy. That saved neither Air Canada nor the practice in Hamm, as above.

A better model. This one helps. In Anthropic's trading experiment, described below, the stronger model got better prices. And the persuasion lift in the Wharton studies fell from roughly 40 points to 16 in the newer study, which used reasoning models and a harder request. That's smaller, and still far from zero.

Labelling the bot. You should, and in places you must. Article 50 of the EU AI Act has required chatbots to tell people they're talking to an AI since 2 August 2026, unless it's obvious (AI Act Service Desk). From 2027 California's AB 1609 will also require companies with more than $500 million in annual revenue to make a good-faith effort to put customers through to a person within 15 minutes, or to schedule a callback within one business day (San Francisco Standard, 15 October 2026). Cursor now labels its bot's replies. But none of the rulings above turned on whether the customer knew it was a bot.

The bot asks, your systems decide

The fixes that have held all take the decision away from the model. In the order we'd do them:

1. Let the bot ask, never grant. Prices, discount codes, refunds, credits and offers come from the system that already owns them (the checkout, the booking engine, the CRM), under rules a person set. The bot can request a refund. Something outside the model checks the order, the policy and the amount, and says yes or no. OWASP's guidance on AI risks says the same thing: enforce authorisation in the downstream systems, and don't leave the model to decide what's allowed (OWASP LLM06:2025). BMW Toronto's fix, described above, is the human version of this rule.
2. Keep money out of free text. The bot never types a code, a price or a refund amount for a customer to act on. It shows what the system returned. If the checkout won't accept a code the bot invented, the worst outcome is an annoyed customer.
3. Put hard limits outside the model: a cap on discounts per conversation and per day, a value above which a person approves, a rate limit per customer. They're ordinary fraud controls, and they don't care how charming the customer was.
4. Give it a narrow job. Have it answer from your actual policy text, show the customer the paragraph it used, and refuse what's out of scope. That includes writing Python for strangers.
5. Say it's a bot, and give people a way out. Label it, and route anything involving money, complaints or bereavement to a person. The CMA's guidance asks for a human in the loop, and its own worked example is an AI agent handling refund requests, with someone experienced regularly reviewing what it decided.
6. Test it the way your customers will, before launch and again after every change of model or prompt: flattery, invented status, a fake manager, a long conversation that wears its rules down, a policy question it has no answer to. The CMA expects testing before deployment. Most jailbreak test lists are built to make a model say something dangerous. A discount test is about getting it to say yes, and that's a different list (our view).
7. Read what it promised, and decide quickly. Log the commitments the bot makes. When it promises something plausible, decide fast whether to honour it, then fix the cause. We'd budget to honour the believable mistakes.

The first two are design decisions, and they're far cheaper to make before launch than to bolt on to a bot that already issues codes. The rest are settings and habits.

Who can ignore this

You can skip most of this if your chat window's a scripted menu that only shows answers someone wrote and approved, or if no customer ever talks to your AI. If your bot can't see prices, bookings or accounts, and it hands every money question to a person, your exposure is mostly embarrassment (DPD's lesson). That's worth an afternoon of testing, not a project. Everyone else: if your bot can state a price, a policy or a refund rule to a customer, assume you'll answer for it. In the UK the CMA's guidance already says so.

When customers send bots of their own

So far it's been people talking chatbots into things. The next round's bots talking to bots.

In an experiment published in April 2026, Anthropic gave 69 of its staff $100 each and a Claude agent that bought and sold real things for them over Slack, from a snowboard to a lab-grown ruby, with no human involved once trading began. In some runs each person's agent was randomly given either the stronger model or the cheaper one. The stronger agents got better prices. A broken folding bike sold for $65 through one and $38 through the other. But the people represented by the weaker model rated their deals as fair as everyone else did, and Anthropic's summary was that they "didn't notice their disadvantage" (Anthropic, 24 April 2026).

Microsoft, which sells AI agents itself, found something similar in a simulated marketplace in October 2025. Between 80% and 100% of its buyer agents took the first offer they got, and some models were manipulated into sending all their payments to attackers (Microsoft Research).

Our read: within a couple of years you'll find some of your customers sending an agent to argue about price, delivery and refunds for them. If your bot's the weaker negotiator, it'll give away margin in conversations that look polite and perfectly reasonable in the logs, one concession at a time, with nobody on your side ever reading the exchange. You'll see it in the margins before you see it in the transcripts.

Back to Quinn

Quinn's bad offer started with a person's wrong number, and the Journal's free shop with a reporter's patience. In both, the bot had the authority to turn it into a promise. Swap in a discount code, a refund rule or a free flight and you've got most of the cases above.

So the question that's worth asking about any customer-facing bot is what it can promise that your systems will then honour. If the answer's nothing that a person or a rule hasn't already approved, a talkative customer can waste its time but not your money.

It's the first question we ask when we test customer-facing bots at Redvault.