AI search visibility for advisory firms

Can AI search engines read your firm’s site? A robots.txt check of 982 RIA sites

Fewer than 1 in 100 of the 982 RIA websites we checked block AI search crawlers in robots.txt. Ten questions find the blocks that file cannot show.

Updated 10 min read

Request your free Audit

AI search engines can read most RIA websites, as far as robots.txt shows. Of the 982 RIA websites we checked on November 14, 2026, 7 blocked ChatGPT’s, Perplexity’s or Claude’s search crawler in robots.txt. Another 79 answered our request with “unauthorized” or “forbidden”, so we never saw what their files allow. And the file cannot show the gates outside it, such as your CDN’s bot settings, text that only JavaScript adds and pages the engines have not indexed. The self-check below covers all of them, each with the engine’s own rule.

0.7% of the 982 RIA websites we checked, 7 sites, block ChatGPT’s, Perplexity’s or Claude’s search crawler in robots.txt.

How do you check whether AI search engines can read your site?

Answer the ten questions below about your own site. Each one says what to fix if the answer is no and links to the rule behind it, and your answers are not stored or sent.

The first five questions are about robots.txt, and you can read yours by typing your web address, then /robots.txt, into a browser. A “Disallow: /” line under “User-agent: *”, or under a crawler’s name, blocks that crawler from the whole site, while a Disallow line with a longer path blocks only that path. If there is no file at all, robots.txt blocks nothing.

Order matters when a file names a crawler. A group that names a crawler replaces the * group for that crawler, so a file can shut out every crawler by default and still let a named one in.

Can AI search engines read your firm’s site?

  1. Does your robots.txt allow OAI-SearchBot, ChatGPT’s search crawler?

    If not: Remove any rule that blocks it. OpenAI says sites that opt out are not shown in ChatGPT search answers.

  2. Does it allow PerplexityBot?

    If not: Allow it. Perplexity uses it to surface and link sites in its search results.

  3. Does it allow Claude-SearchBot?

    If not: Allow it. Anthropic says blocking it may reduce your visibility in Claude’s search results.

  4. Does it allow Googlebot?

    If not: Allow it. Google says a page must be indexed in Google Search, and eligible for a snippet, to be linked in AI Overviews or AI Mode.

  5. If you block training crawlers such as GPTBot, are the search crawlers above still allowed?

    If not: Give each crawler its own rule. Blocking GPTBot or Google-Extended does not block search.

  6. Does your CDN or firewall let these crawlers through?

    If not: Ask whoever runs your hosting to check its bot settings. Cloudflare, for one, sets search, agent and training traffic separately.

  7. Is your main text in the HTML your server sends, not added by JavaScript?

    If not: Put the text in the page itself. Most AI crawlers measured so far do not run JavaScript.

  8. Are your key pages free of noindex and nosnippet?

    If not: Remove them from pages you want cited. Google lists both as ways to limit what its AI features show.

  9. Does your robots.txt load without a server error?

    If not: Fix any server error on /robots.txt. The robots.txt standard tells crawlers to treat one as a full block.

  10. Is your site verified in Google Search Console and Bing Webmaster Tools, with a sitemap submitted?

    If not: Verify it in both and submit the sitemap. Bing’s tools also report when its AI answers cite you.

Count your yeses

  • 0 to 4 yes. Several answers are no or not sure. Start with the access questions: an engine cannot cite a page it cannot fetch.
  • 5 to 7 yes. Some basics are in place. Settle each no or not sure, access questions first, then the text and indexing ones.
  • 8 to 9 yes. Nearly there. Fix each remaining no, access questions first.
  • 10 yes. The basics are in place. What decides citations next is how your pages match questions and what other sites say about you.

The access questions come first because they gate everything else. Your pages can match a prospect’s question perfectly and still never be cited if the engine’s crawler was turned away before it read them.

Which AI crawlers should you allow, and which can you block?

Allow each company’s search crawler, and treat its training control as a separate choice, because blocking training keeps your pages out of model training without hiding them from search. Perplexity is the one company in the table below without a separate training control.

Search crawlers and training controls, by company
Search crawlerTraining control
OpenAIOAI-SearchBotGPTBot
AnthropicClaude-SearchBotClaudeBot
PerplexityPerplexityBotNone: Perplexity says PerplexityBot is not used to crawl content for AI foundation models
GoogleGooglebotGoogle-Extended
AppleApplebotApplebot-Extended, a rule only: it does not crawl

Source: OpenAI, Anthropic, Perplexity, Google and Apple, read November 14, 2026

The search crawler is the one to protect. OpenAI states it plainly: “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.” A training block is a different matter, and Google and Apple both say it leaves you in search. Google says Google-Extended “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search.” In Apple’s words, “Webpages that disallow Applebot-Extended can still be included in search results.”

Seven of 982 RIA websites block an AI search crawler in robots.txt

Only seven of the 982 RIA websites we checked on November 14, 2026, used robots.txt to block at least one AI search crawler from the home page. Six of those seven did it with a rule written for all crawlers, one that was never aimed at AI in particular. Training crawlers are blocked more often, by 34 sites, but 20 of those 34 files carry Cloudflare’s managed robots.txt rather than a rule someone wrote by hand.

Fewer than 1 in 100 RIA websites block an AI search crawler in robots.txt

Bar chart. Name any AI crawler: 11.9%. Block an AI training crawler: 3.5%. Block an AI search crawler: 0.7%. Block all crawlers by default: 0.6%. Based on: 982 websites of SEC-registered investment advisers that answered.

Data
Fewer than 1 in 100 RIA websites block an AI search crawler in robots.txt
LabelValueNote
Name any AI crawler11.9%117 of 982
Block an AI training crawler3.5%34 of 982
Block an AI search crawler0.7%7 of 982
Block all crawlers by default0.6%6 of 982

Source: Qubo robots.txt check, November 14, 2026. Based on: 982 websites of SEC-registered investment advisers that answered.

A crawler named in your file is not always blocked, which is why the first bar runs so far past the other three. Of the 117 sites that name an AI crawler, 72 use Squarespace’s standard file, which lists AI crawlers without blocking them from your pages.

How we counted

  • Which sites. 1,000 websites drawn at random from the 14,698 listed by SEC-registered investment advisers in the SEC’s adviser data of November 13, 2026. 982 answered; 17 did not, and 1 returned a server error, which the standard treats as a full block.
  • What we fetched. One fetch of /robots.txt per site on November 14, 2026, following up to five redirects, as QuboResearch/1.0, and nothing else.
  • How we read it. A matcher built to the robots.txt standard, RFC 9309, decided whether each crawler may fetch the home page. 420 of 420 verdicts read by hand agreed with it.
  • Counted as open. 95 sites had no robots.txt, 2 answered with another error and 35 sent a web page instead. Crawlers may read these as no rules.
  • Refused our request. 79 sites answered “unauthorized” or “forbidden”. We counted them as not blocking in robots.txt.
  • What it cannot see. CDN and firewall rules, what crawlers actually do, and websites of state-registered advisers.

With only 7 of 982 files blocking an AI search crawler, your own robots.txt probably lets the search crawlers in, and opening it to confirm takes about a minute. The 79 sites that refused our request outnumber those blockers more than ten to one, and from outside we could not tell what their files would have said. A CDN or firewall can turn away an AI crawler even when robots.txt lets it in, and the file will never show it. The gates worth your time are the ones the next sections cover.

A CDN, a site builder or a broken robots.txt can also block crawlers

Your CDN, your site builder’s settings or a broken robots.txt file can each shut crawlers out without anyone at the firm choosing to. Cloudflare sets Search, Agent and Training traffic separately, and its Training setting also covers crawlers that do both training and search. So blocking training there can also block a crawler that feeds search. Its defaults have changed since July 2025 and differ by the kind of bot, which means the only way to know what yours does is to check your own zone’s setting.

Site builders add switches of their own. On the hosted WordPress.com service, “Prevent third-party sharing” adds AI bots to your robots.txt without its help page saying which ones, so read the file after you turn it on. Squarespace’s “Block known artificial intelligence crawlers” box is off by default, and its list names training crawlers such as GPTBot while leaving out OAI-SearchBot, Claude-SearchBot and PerplexityBot. On Wix, AI blocks are manual lines in its robots.txt editor, so check each name you find there against the table above.

The file itself can fail, too, and when /robots.txt returns a server error, the robots.txt standard tells crawlers to treat it as a full block. Google stops crawling for 12 hours, then uses the last good copy for up to 30 days. One of the 1,000 sites we drew for our check returned a server error, and it is one of the 18 that did not answer.

Text added by JavaScript can be invisible to AI crawlers

Most AI crawlers read only the HTML your server sends, so text that a script adds after the page opens can be invisible to them. In December 2024, Vercel and MERJ reported that “none of the major AI crawlers currently render JavaScript.” They named two exceptions, Google’s Gemini, which uses Googlebot’s infrastructure, and AppleBot. Crawlers change, which makes the finding a reason to check your own pages more than a rule to rely on.

To check, open any one page, view the page source and search for a sentence from your main text. If it is there, the text is in the HTML, and crawlers that skip JavaScript can read it. If it is not, ask your web team to render that text on the server or in the page itself.

An llms.txt file does not help on current evidence

Ahrefs found that 97% of the llms.txt files on sites it tracks got no requests in May 2026, and it called the file “largely decoration” for showing up in ChatGPT, Perplexity or AI Overviews. SE Ranking found “no correlation between AI citations and LLMs.txt.” Our AEO post covers what else doesn’t help.

What this check does not cover

This check covers access, which is only the first gate in AI search optimization (AEO). Once engines can read your site, they still choose which firms to name, and that depends on whether your pages match the question and how other sites describe you. Our ChatGPT post covers both. To see what the engines say about your firm today, check it yourself, or ask for Qubo’s free AI Visibility Audit, which runs that check for you.

Sources

Questions

How long does a robots.txt change take to work?

Allow about a day. OpenAI says ChatGPT search can take about 24 hours to adjust, Perplexity says up to 24 hours, and Google generally caches robots.txt for up to 24 hours.

Do AI tools ignore robots.txt when a user asks them to open a page?

Some do. Perplexity says its user-triggered fetcher, Perplexity-User, generally ignores robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, which does not decide what appears in ChatGPT search.

Should an advisory firm block AI training crawlers?

That is a policy choice for your firm, not a visibility fix. OpenAI, Anthropic, Google and Apple each give training its own control, so you can block training and still be found in search.

Gino Ferrand founded Qubo, an AI-native, full-service marketing agency for US financial services firms.

How Qubo does this

In the free Audit, a Qubo strategist asks five AI engines the questions your clients ask, traces each wrong fact to its source and walks you through what to fix first.