How to Test Whether AI Crawlers Can See Your Content

Your page ranks on Google. You checked, it is there, on the first page for a query you care about. So you assume the content is visible to machines and move on. Then you ask ChatGPT or Perplexity the same question and your page is nowhere in the answer, while a thinner competitor gets cited. The ranking was real. The visibility was not. This tutorial shows you how to test the gap directly, in a few commands, before it quietly costs you the next era of search traffic.

The reason for the gap is worth one paragraph, and no more, because the mechanism is covered in depth elsewhere. Googlebot renders JavaScript. It fetches your page, queues it, runs the scripts, and eventually indexes the finished result. Most AI crawlers do not do this. They fetch the raw HTML your server sends and keep whatever text is already in it. If your content is assembled in the browser by JavaScript, Google can wait for it and AI engines usually cannot. For the full picture of how rendering strategies cause this, read the JavaScript SEO and rendering guide. This post is the other half of that story: not which strategy to choose, but how to measure what you already ship.

What Readable Bytes Actually Means

Every page you serve has two versions. The first is the raw HTML, the exact bytes your server returns before any script runs. The second is the rendered DOM, what the page becomes after JavaScript executes in a browser. A human sees the second version. A non rendering crawler sees only the first.

The useful number is how much real, readable text lives in that first version. Call it the readable content of the page. Not the byte size of the file, which is inflated by scripts, inline styles, and framework boilerplate, but the visible words a reader could actually extract from the raw HTML alone.

A server rendered blog post has almost all of its readable content in the raw HTML. A single page application often has almost none. Its raw HTML is a shell: a <div id="root">, a few <script> tags, and a loading spinner. To a browser that is the start of a page. To an AI crawler that is the entire page, and it contains no article.

The pattern is easiest to see side by side. The same 1,500 word article, served two different ways, produces very different raw HTML:

The test you are about to run answers one question for any URL: if the reader were a machine that does not run JavaScript, how much of my content would survive? Everything below is a way to see that number, first for one page, then for a whole site.

Three Ways to See What a Non Rendering Crawler Sees

You do not need special software for a single page. You need a terminal and a browser. Each method below shows the same raw HTML from a slightly different angle, and using two of them together removes any doubt.

The Raw Fetch

curl fetches a URL and prints exactly what the server sends, with no JavaScript execution at all. That makes it the closest thing to an AI crawler’s view that you already have installed.

curl -s https://example.com/your-page > raw.html

Open raw.html in a browser, or scroll it in the terminal, and look for a sentence you know is in your article. If your opening paragraph is there, good. If all you see is a shell and script tags, that is what the crawler sees too.

One refinement matters. Some sites serve different HTML to bots than to browsers, sometimes on purpose and sometimes by accident. To catch that, fetch with a real AI crawler identity so you see what the bot is actually served:

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
  https://example.com/your-page > raw-as-bot.html

If raw.html and raw-as-bot.html differ in their readable text, you have a serving inconsistency worth understanding before anything else.

The Disabled JavaScript View

The raw fetch shows the bytes. The browser with JavaScript disabled shows the same bytes rendered as a page, which is often easier to judge at a glance.

In Chrome, open DevTools, press the command menu with Control+Shift+P or Command+Shift+P, type “Disable JavaScript”, select it, and reload the page. What remains on screen is, roughly, what a non rendering crawler keeps. If the article body vanishes and you are left with a header and an empty frame, the content depends on JavaScript that the crawler will never run.

This view is worth doing even when the curl output looked fine, because it catches content that is present in the HTML but hidden or replaced by scripts on load.

The Rendered Comparison

To prove the gap rather than suspect it, compare the two states. A rough readable content count from the raw HTML takes one command. This strips scripts and styles, removes the tags, and counts the characters of visible text that remain:

curl -s https://example.com/your-page \
  | perl -0777 -pe 's/<(script|style)[^>]*>.*?<\/\1>//gs; s/<[^>]+>/ /g; s/\s+/ /g' \
  | wc -c

Run it on a page you know is server rendered and note the number. Run it on a page you suspect is client rendered and compare. A content rich article that returns a few hundred characters is telling you its words are not in the raw HTML. The exact figure does not matter. The ratio between a healthy page and a suspect one is what reveals the problem.

Here is how the three methods relate, so you can pick the right one for the moment:

Two of the three together is usually enough. The curl fetch tells you what is served, and the disabled JavaScript view tells you what that looks like to a reader who is really a machine.

Auditing Readable Content Across a Whole Site

Testing one URL by hand is fine for a spot check. It does not scale to a site with hundreds of templates and thousands of pages, and the pages that fail are rarely the ones you would think to test. The homepage is usually fine. The deep product page, the filtered listing, the article loaded through a client side route, those are where content quietly disappears.

For scale, you want a crawler that can fetch every URL twice, once as raw HTML and once with a real browser engine, and report the difference per page. That difference is the audit. Pages where the rendered version has far more readable content than the raw version are your risk list, ranked.

This is exactly the gap Seodisias is built to surface. It crawls with a real browser engine and reports what content was actually extracted, so you can compare what your code produces against what a crawler keeps, across the whole site rather than one page at a time. Its AI Ready analysis looks at the same question from the content side, checking whether the text that does survive is structured in a way AI engines can use. However you run the audit, the principle is the same: never assume a template is safe because one page on it looked fine.

While you are auditing, it helps to know which machines actually visit. Your access logs already list them, and finding AI crawlers in your server logs tells you whether GPTBot, ClaudeBot, and the rest are reaching your pages at all. A page that is invisible in the raw HTML and never crawled has two problems, not one.

Reading the Results Without Overreacting

A failing test is a signal, not a verdict. Before you rewrite an architecture, sort what you found by how much it matters.

Start with intent. A logged in dashboard, an account settings screen, or an internal tool has no reason to be readable by an AI crawler, and client side rendering is a perfectly good choice there. Missing readable content on those pages is not a bug. It is the design working as intended.

Then look at the pages that exist to be found. Articles, guides, product and category pages, documentation, anything that should show up in an answer or a citation. When those come back nearly empty in the raw HTML, that is the real finding, and it deserves attention in rough proportion to the traffic and the citations the page should be earning.

Watch for the partial failures too, because they are easy to miss. A page whose body text is in the raw HTML but whose <title> and meta description are injected by JavaScript after load will be read with a generic site name as its title. The words are visible, the framing is not, and in AI answers the framing is often what gets quoted. A quick way to catch this is to confirm the title and description you expect are present in the curl output, not just the body.

Finally, resist the urge to treat every gap as urgent. A quarterly pass on a sample of pages from each template catches regressions early without turning a measurement habit into a panic. The point of the test is calm awareness of what machines keep, run often enough to trust.

What the Test Does Not Tell You

This is a visibility test, not a citation test, and the difference is worth holding onto. Passing it means an AI crawler can read your words. It does not mean an engine will choose to quote them. Plenty of pages are perfectly readable in the raw HTML and still never appear in an answer, because the content is thin, undifferentiated, or simply not the best source for the question.

Think of readable content as the entry ticket. Without it you are not in the room at all, no matter how good the writing is. With it, you are eligible, and then the usual work decides the outcome: clear structure, genuine expertise, information a reader cannot get in ten other places. That work is the subject of the generative engine optimization playbook, and this test is simply the check you run first, because it is the one failure that makes all the rest irrelevant.

The test is also a snapshot, not a monitor. A page that passes today can regress the next time a component is refactored to fetch its content on the client, and nothing in the ranking will warn you. That is why the habit matters more than any single result. Measure on a schedule, keep the numbers, and watch for the page that used to be full and is suddenly a shell.

Conclusion

Rankings tell you what Google does after it renders your page. They say nothing about what an AI crawler keeps when it does not. The only way to know is to look at the raw HTML directly, and now you have three ways to do it for one page and a way to do it across a whole site. Fetch the page, disable JavaScript, compare the readable content, and sort the results by whether the page was ever meant to be found. When the pages that matter come back empty, the fix is a rendering decision, and the options are worth working through carefully rather than guessing. Getting the words into the first version of the page is the groundwork for everything else, because an engine cannot cite what it never read. If you want to run this audit across a whole site instead of one URL at a time, download Seodisias for free and let it crawl with a real browser engine. Measure first, then fix what the measurement proves.