Chat on US public company websites: methods and results
An October 2026 study of website chats at US public companies: how many offer one, and whether bots answered three basic customer questions.
What did the study find?
In October 2026, about one in eight US public companies with at least $50 million in revenue (13%, with a 95% range of 11-18%) offered a chat a visitor could find on their website. When an AI agent asked the chats of 120 sampled companies three basic customer questions, it found a chat at 102 of them. 31 of the 102 wouldn't take a question until the visitor gave personal details, and 4 of the 68 that opened with a bot answered all three questions. Large companies did no measurably better than mid-market ones.
Who was studied?
The study covers all 2,764 US-based companies listed on the NYSE or Nasdaq that file annual reports (Form 10-K) with the SEC and reported at least $50 million in 2025 revenue, from the SEC's public financial data. Companies incorporated abroad are out even if run from the US, as are private companies, funds and blank checks. Each company's website came from its own 10-K; 2,745 had one. Where Wikidata listed a different official website, often a shopping site rather than the corporate one, both were scanned.
What counts as a chat?
A chat is one a visitor can find on the homepage or a contact or help page, whether a bot, a person or both, including an in-page "Chat with us" button. Chat behind a sign-in doesn't count.
How were the websites scanned?
On October 14-15, 2026, software visited each company's website in a headless Chrome browser that identified itself as a research visitor, honored each site's robots.txt, and never tried to get past a bot-protection page. On each site it loaded the homepage and up to two contact, support or help pages, scrolled once, and waited. It recorded which chat software loaded and whether a chat launcher appeared on screen.
Who checked the results?
An AI agent (Anthropic's Claude) carried out the checks and the chat tests, operating the visible Chromium browser built into the Claude desktop app in separate sessions, each following a written protocol. That browser names Claude in the user agent it sends to every site (checked November 3, 2026), so a site could tell the visitor was using the Claude app. No person visited the sites for the scan, the checks or the chat tests; a later spot check by a person is described below.
A scanner gets some sites wrong: some block automated browsers, some chat software only shows itself to an ordinary browser, and some chats load from a company's own servers where the scanner can't recognize them. So on October 15-16, 2026, the agent checked a random sample of each scan result in a visible (non-headless) browser, 160 companies in all, without seeing the scanner's result for any company. Five of the 40 sampled blocked sites blocked the agent's browser too, so the table below counts 155.
How many companies offer a chat?
An estimated 13.3% of companies offer a chat a visitor can find, with a 95% range of 11.0-18.3%. The estimate covers 2,677 companies; the 68 sites that failed to load or refused the scanner through robots.txt are left out, as are companies with no website. The ranges for the revenue bands overlap, so these figures don't show a difference by company size.
| Revenue | Companies | Found by the scanner alone | Estimate | 95% range |
|---|---|---|---|---|
| $50M-$100M | 220 | 4.1% | 9.4% | 8.2%-14.5% |
| $100M-$1B | 997 | 7.2% | 13.1% | 11.0%-18.0% |
| $1B-$2B | 390 | 6.2% | 13.1% | 11.0%-17.9% |
| $2B-$10B | 690 | 5.7% | 13.6% | 11.0%-18.8% |
| $10B+ | 380 | 7.1% | 15.8% | 12.3%-21.8% |
| $50M-$2B | 1,607 | 6.5% | 12.6% | 10.6%-17.5% |
| All | 2,677 | 6.4% | 13.3% | 11.0%-18.3% |
How was the estimate built?
Each company falls in one scan-result group, and the estimate applies the checked share of each group to that group's size. For example, 16 of 20 checked "chat detected" sites had a chat, so 80% of that group counts. The ranges come from simulating each share's uncertainty (Jeffreys posterior).
| Scan result | Companies | Checked | With a chat |
|---|---|---|---|
| Chat detected | 171 | 20 | 16 of 20 (80%; 58-92%) |
| Chat-like element, vendor unknown | 79 | 30 | 23 of 30 (77%; 59-88%) |
| Chat software loaded, nothing shown | 81 | 30 | 15 of 30 (50%; 33-67%) |
| Site blocked the scanner | 518 | 35 | 8 of 35 (23%; 12-39%) |
| Nothing found | 1,784 | 40 | 0 of 40 (0%; 0-9%) |
| Chat software loaded, sales chat only for targeted visitors | 44 | Counted as no chat | |
| Site failed to load or refused by robots.txt | 68 | Left out |
What were the chats asked?
Each chat got the same three questions, word for word. The companies were drawn at random, by a seeded draw rather than by the agent, from the 213 companies the scan and checks confirmed had a chat: 60 of the 130 with $50 million to $2 billion in revenue (tested October 16, 2026) and 60 of the 83 above $2 billion (tested October 19, 2026). For each, the agent opened the chat and typed:
- "What are your customer service hours?"
- "How do I update my billing or payment details?"
- "How much does [the company's main product or service] cost?"
The protocol kept the test anonymous and cheap for the companies. No personal details were ever given, real or invented; a chat that asked for a name, email, phone or account number before any question was recorded as such and not tested further. The agent never asked for a person, and if a person joined, it thanked them and closed the chat. It asked three questions only, with no follow-ups, in one chat per company.
How were the answers scored?
Each reply got one of five scores against a written rubric:
- Answered: a direct, specific answer.
- Partial: relevant but incomplete, or a short answer plus a link to the right page.
- Deflected: only a generic link, menu or "contact us".
- Handoff: sent to a person, phone number or form instead of an answer.
- Failed: didn't understand, answered something else, or didn't reply within 60 seconds.
What happened when the chats were asked?
About three in ten chats demanded personal details before taking a question, and few bots answered all three questions. The two revenue groups came out about the same. Ranges in parentheses are 95% Wilson intervals.
| Mid-market ($50M-$2B) | Large ($2B+) | |
|---|---|---|
| Companies sampled | 60 | 60 |
| Chat found | 53 | 49 |
| Asked for name, email, phone or account number before any question | 17 of 53 (32%; 21-45%) | 14 of 49 (29%; 18-42%) |
| Opened with a person | 3 | 0 |
| Opened with a bot | 33 | 35 |
| Bot reads free text (generative or mixed) | 24 of 53 (45%; 33-59%) | 22 of 49 (45%; 32-59%) |
| Button-menu bot | 8 of 53 (15%; 8-27%) | 14 of 49 (29%; 18-42%) |
| Bots that answered all three questions | 2 of 33 | 2 of 35 |
Bot answers by question at mid-market companies, for chats that opened with a bot, leaving out questions a bot couldn't be asked:
| Question | Asked | Answered | Partial | Deflected | Handoff | Failed |
|---|---|---|---|---|---|---|
| Hours | 31 | 10 | 9 | 3 | 5 | 4 |
| Billing | 29 | 7 | 5 | 7 | 7 | 3 |
| Price | 26 | 3 | 8 | 6 | 9 | 0 |
The same at large companies:
| Question | Asked | Answered | Partial | Deflected | Handoff | Failed |
|---|---|---|---|---|---|---|
| Hours | 34 | 11 | 2 | 5 | 2 | 14 |
| Billing | 31 | 11 | 2 | 5 | 4 | 9 |
| Price | 30 | 5 | 8 | 2 | 6 | 9 |
Bot answers by bot style, across all three questions, counting questions answered fully or in part:
| Bot style | Mid-market | Large |
|---|---|---|
| Free text | 41 of 69 (59%; 48-70%) | 38 of 66 (58%; 46-69%) |
| Button menu | 1 of 17 (6%; 1-27%) | 1 of 29 (3%; 1-17%) |
Which bots answered all three questions?
Four did: Bandwidth and CarGurus among mid-market companies, and DocuSign and Steve Madden among large companies.
How far do the scores hold?
A second, separate AI agent session, which never saw the first session's answers, repeated 20 of the mid-market chat tests three days later. The two runs agreed on who answers first every time, and on whether a question was answered for 85% of questions.
| Compared | Agreement | Cohen's kappa |
|---|---|---|
| Who answers first (bot, person, form) | 100% of 20 | 1.00 |
| Answered (fully or in part) or not | 85% of 33 questions | 0.67 |
| Bot style | 79% of 14 | 0.72 |
Luke Wilcox, Roiva's founder, also repeated 10 of the chat tests himself in Google Chrome, with the same questions and rules. He and the agent agreed on whether a question was answered (fully or in part) for 26 of 30 questions, and for all 10 they agreed on whether the chat answered all three questions at least partially. The check ran two weeks after the study, so bots may have changed in between.
What are the study's limits?
This is a snapshot: one scan, and one chat session per company on one day. Some chats are staffed or open only during business hours.
- The chat tests are samples of 60 a side. Differences smaller than about 20 points between the two groups can't be told apart from chance, which is why the groups are reported as about the same.
- The scores are an AI agent's judgment against a written rubric. The repeat test measures how far they hold.
- The same three questions went to every company so the results could be compared, and the agent filled in each company's main product from its homepage. Some questions fit some businesses poorly: a grocery chain's shoppers rarely have billing details to update, and "groceries" has no single price. A reply that gave a price range or examples, or explained why a question didn't apply and where to go instead, was scored answered or partial.
- A bot that offered only buttons got one button choice per question. Some menus reach an answer after two or three clicks, so the button-menu results describe the first step, not the best a patient visitor could get.
- The agent's browser named itself as the Claude app, and a site could have treated it differently from a customer's browser. No chat test met a verification or "are you a robot" page, and the spot check by a person found no sign of worse answers, but a quieter difference can't be ruled out.
- The study covers US public companies only.
What doesn't this study show?
It doesn't show how many companies use AI for customer service. A chat may be a bot, a person or both, and the software behind it often can't be identified from outside.
It also doesn't rank chat vendors, industries or individual companies. One session on one day is not a verdict on any company, so no company is named here except the four whose bots answered all three questions, and company-level records aren't public.
Who ran the study?
Roiva ran this study. Roiva is AI portfolio management software that tracks every AI initiative's budget, approvals, cost and ROI, so how well AI performs is a problem the company sells into.
Get the next one
New field notes by email as they're published. Or subscribe to the RSS feed.