Study Reveals Vast AI Visibility Gap: Few Sites Block Crawlers, Even Fewer Are Cited

New data reveals a stark disconnect between website accessibility and visibility in the age of AI-powered search. An ongoing study tracking small and mid-sized businesses shows that most remain absent from AI-generated answers, even though the majority make their sites technically readable by AI systems. These findings underscore the new challenge for businesses as search shifts from providing links to delivering direct answers.

Key takeaways

• Extreme citation gap: 94.8% of audited websites are never named in an AI assistant’s answer to a buying-intent question about their category.
• Low barrier to entry: Only 8.9% of sites block at least one AI crawler in their robots.txt file, showing most are open to being crawled.
• Focus on training crawlers: Blocking mainly targets bots that harvest data for model training (like GPTBot and ClaudeBot) instead of agents that fetch pages for live user queries.
• Structured data adoption lags: While 54.6% of sites publish some schema.org structured data, only 19.3% use the specific LocalBusiness schema that clearly states a business’s identity and location for machines.
• Assistant performance varies: Gemini named audited businesses in 2.9% of its answers, the highest rate among the four major assistants tested, all of which were asked identical questions.

The visibility challenge

The AI Visibility Study, built from 531 audits across 458 domains, measures two distinct pillars: answer presence and machine readability. The core finding is that presence—being named in an answer—is exceptionally rare. Out of 5,978 assistant answers generated for specific buying-intent queries, only 1.9% named the business in question. When analyzed per site, this translated to just 10 out of 193 sites being named even once across all four major AI platforms: ChatGPT, Claude, Gemini, and Perplexity. Simply being technically accessible is not enough to earn a citation in this new AI-driven answer paradigm.

Readiness versus reality

On the technical readiness side, most websites are taking basic steps. The study found that 79.4% publish a robots.txt file, 73.9% publish a sitemap, and 77.1% have a meta description. Adoption of more advanced, machine-readable signals is lower, however. Only about half (54.6%) use any schema.org structured data on their homepage, leaving assistants with nothing but prose to interpret. The specific LocalBusiness schema, critical for local service providers, appears on just 19.3% of sites. The data also shows a key nuance in how sites block AI: 38 sites block GPTBot and 36 block ClaudeBot (both used for training models), while only 4 block OAI-SearchBot and 3 block Perplexity-User (agents that fetch pages for live answers). This suggests site owners might not fully differentiate between crawlers for training and those for retrieval. Blocking the latter directly harms a site’s citability.

Why it matters

AI assistants are fast becoming the first point of contact for customers seeking recommendations. Invisibility in their answers poses an existential threat to businesses, especially local ones. The study highlights that the problem is not mainly one of access—few sites are actively shutting out AI—but of relevance and clear communication. A site can have perfect markup and still go unnamed. That’s why the study publishes presence and readability metrics side-by-side instead of as a combined score. For business owners, the call to action is clear: beyond basic SEO, they must optimize for how AI systems evaluate and cite sources. They need to ensure their core identity and value proposition are obvious not just to humans, but to the machines now curating information for potential customers.