Short answer: probably not. But about one in eight small business websites is turning away at least one AI reader, and almost none of those owners chose it.
On September 17, 2026 we fetched the homepage of 400 small business websites - 200 alarm and security companies, 200 dental and chiropractic practices. We asked for each one first as an ordinary web browser, then as each of the four readers the AI companies send out.
On 341 of the 400 the plain browser request came back clean, so we could tell what happened next. 44 of those 341 - 12.9% - refused at least one AI reader. 15 of them, 4.4%, refused the readers that decide whether a business gets named in an answer.
The other 59 we are not counting either way. Something in front of those sites challenged our request before we got a clean read, so the honest verdict is that we could not tell. Every number below is out of the 341.
There are two kinds of AI reader, and only one costs you customers
This gets reported wrong almost everywhere, and it changes what the number means.
Training crawlers have names like GPTBot, ClaudeBot and Google-Extended. Their job is to collect pages to help train an AI model. If your site turns them away, your words stay out of that training. That is the whole effect. ChatGPT can still open your site later to answer a live question.
Answer readers have names like OAI-SearchBot, ChatGPT-User and PerplexityBot. These are what an assistant sends out in the moment. Somebody asks who to call for a leaking water heater, it reads a handful of websites right then, and it names a few businesses. Turning these away is the one that costs you the job.
Of the 44 sites that refused something, 29 refused training crawlers only and 15 refused an answer reader. Refusing the training crawlers on purpose is a fair choice, and some owners make it deliberately. The cost is real but small: your own description of your work stays out of the model, and live lookups still go through.
It clusters by what the website is built on
| Sites we could check | Turned an AI reader away | |
|---|---|---|
| Self-hosted WordPress | 173 | 31 (17.9%) |
| Sitting behind Cloudflare | 49 | 9 (18.4%) |
| Wix | 12 | 0 (0.0%) |
| Squarespace | 15 | 0 (0.0%) |
| Plain Apache or nginx | 35 | 0 (0.0%) |
| Everything else | 57 | 4 (7.0%) |
| All sites we could check | 341 | 44 (12.9%) |
Wix, Squarespace, plain Apache and plain nginx refused nothing at all - not one site out of 62. Self-hosted WordPress and sites behind Cloudflare refused at roughly one in five. The two niches are unrelated and produced nearly the same shape, so this is not a quirk of one trade.
Nobody sat down and decided to keep AI out
That clustering is the whole story. If owners were choosing this, the refusals would be spread evenly across every kind of website. They are not. They land on the two setups that arrive with bot blocking already switched on.
A security plugin on a self-hosted WordPress site comes with a built-in list of robots it treats as bad, and a firewall ships with managed rules that do the same. Those lists exist to stop scrapers and spam, and most were written before any of these AI readers existed.
Your robots.txt file will not tell you
A lot of advice says to open robots.txt and look for AI bot names. We pulled that file on all 400 domains and 338 came back readable. Only 6 of them - 1.8% - name an answer reader and tell it not to read the site. At the server, 4.4% refused one.
The file under-reports by roughly two and a half times, because the refusal usually is not in the file at all. It is in the plugin or the firewall sitting in front of it. Only 43 of the 338 readable files mention an AI reader of any kind.
Why this is worth ten minutes
People who asked an AI assistant to recommend a local business went from 6% to 45% in one year (BrightLocal 2026 Local Consumer Review Survey, n=1,002). A few years ago a customer found you through a search page. Now a good share of them ask an assistant and get three names back.
If your site is one of the 87.1% that read clean, this costs you nothing. If it is in the 12.9%, find out which kind of refusal you have first.
- 1Run your web address through the free checker at aisecurityedge.com/tools/ai-crawler-check. It does what this study did and reports reader by reader.
- 2See whether the names refused are training crawlers (GPTBot, ClaudeBot) or answer readers (OAI-SearchBot, ChatGPT-User, PerplexityBot).
- 3If only training crawlers were refused, decide whether you want your words in AI training. Leaving it alone is defensible.
- 4If an answer reader was refused, send the result to whoever manages your website or firewall. The fix is almost always one allow rule.
- 5Check again in a week. Plugin and firewall rules update themselves.
- ✓About one in eight small business websites turns away at least one AI reader.
- ✓Refusals cluster on self-hosted WordPress and sites behind a firewall, which means defaults, not decisions.
- ✓Turning away training crawlers is a fair trade with a small cost. Turning away answer readers is the expensive one.
- ✓Do not judge this from robots.txt. Check what your server does when an AI reader asks.
- ✓Being readable is the starting line, not the finish.
Fast Digital Marketing does one thing about this for small businesses across the US: Get Recommended, $97 a month, cancel any time, no contract. We work on getting your business named when somebody asks Google or ChatGPT who to hire.