Why might an AI URL auditor see only a React SPA's raw HTML, and what should developers do?

What a raw-HTML result actually tells you

When a client-side React application is built with a tool such as Vite, the server often sends a small HTML document containing a root element and one or more script tags. The browser downloads the JavaScript, runs the app, fetches data if needed, and changes the DOM. A plain HTTP client sees the response before those browser steps. If the content lives in that later work, the raw response can look nearly empty even though a person with a working browser sees a complete page.

That observation is useful, but bounded. It proves that the tested path did not obtain the post-JavaScript content at that time, with that request context. It does not identify whether the tool is a simple fetcher, a browser with JavaScript disabled, a browser that timed out before data arrived, a fetcher blocked by a login or robots rule, or a product that deliberately returns a simplified extraction. It also says nothing by itself about a search crawler, a training crawler, or a normal browser.

Do not use the phrase “AI crawler” as a technical capability label. It does not tell you whether the system runs JavaScript, waits for network requests, executes a user click, sends cookies, respects a given robots.txt rule, stores an index, or merely retrieves a URL supplied in a conversation. Those are separate properties that must be tested or documented for the particular system.

Four systems that are often called an AI crawler

The names are not hypothetical. Anthropic distinguishes Claude-User, used for user-initiated website access, from Claude-SearchBot, used to improve search results, and it separately documents ClaudeBot for possible model-training use. Anthropic crawler documentation This is why an assistant's link analysis cannot settle what a search index, another vendor's browser, or an accessibility audit sees.

Google gives a concrete counterexample to the claim that all crawlers only read the raw response. It documents that Googlebot crawls first, queues eligible pages for rendering, then uses a headless Chromium renderer and indexes rendered HTML. That is a statement about Google Search, not a promise for other services. Google also says that pre-rendering or server rendering remains a good idea because not all bots can run JavaScript. Google's JavaScript SEO basics

An audit tool is a third category, not a proxy for a crawler. For example, Lighthouse runs audits against a page and can be run in Chrome DevTools, where it can audit an authenticated page. That makes it valuable for observing a browser-loaded page, but it cannot tell you whether a remote chatbot, indexer, or training agent has the same browser, wait budget, cookies, policy, or purpose. Chrome Lighthouse overview

React rendering modes and why the distinction matters

Client-side rendering

With client-side rendering, the server sends an app shell and the browser constructs the meaningful page after JavaScript runs. This is often a sensible design for a signed-in application, a dashboard, or a workflow that has no useful public representation. It becomes fragile for public discovery if the only copy of the title, body text, canonical URL, internal links, or page-specific metadata is created after asynchronous browser work.

Hydration

Hydration is not another word for ordinary client-side rendering. React says hydrateRoot is for a browser DOM node whose HTML was previously generated by react-dom/server. In other words, the visitor first receives meaningful HTML and React then attaches event handlers and continues managing it. React client APIs Hydration allows a public page to be readable before JavaScript has completed while retaining interactive components afterward.

Server-side rendering and static generation

Server-side rendering produces page HTML on the server for a request. Static generation produces it at build time, often for many public routes, and can serve it efficiently from a CDN. The two can coexist with client-side React. MDN notes that SSR sends generated HTML to the client and that static pages are generated at build time rather than request time. MDN on SSR React also provides a prerender API for static HTML generation and expects a client to call hydrateRoot if the result should become interactive. React prerender

Do not equate adding SSR with giving up hooks. Components can still use state and event handlers after hydration. The constraint is narrower: content required in the initial HTML cannot depend solely on a useEffect fetch, because React documents that Effects run only on the client, not during server rendering. Move public data loading into the server or build step, provide it as initial data, and keep browser-only calls such as localStorage, geolocation, or an authenticated user profile behind a client boundary. React useEffect

What should be visible before JavaScript runs

For a public, canonical page, put the things that establish its meaning in the server response whenever practical:

• A unique <title>, a useful meta description, the canonical URL, and language declaration.
• The page's principal heading, explanatory text, and normal <a href> links to other public routes.
• Accurate status behavior, including an actual 404 or 410 for missing public pages rather than a successful app shell for every unknown route.
• Structured data that represents real, visible page content when a supported structured-data type applies. Google generally recommends JSON-LD because it is easier to maintain at scale, but markup is not a substitute for accurate human-readable content. Google structured-data guidance

This is a resilience recommendation, not an instruction to create a bot-only version of a site. Serve the same material public page to people and legitimate automated clients. A screenshot, JSON-LD blob, or page title does not make a page equivalent to having its actual text and navigable links available. For data that changes rapidly, define the freshness contract and make sure the static or server-rendered version obeys it.

Client-side route handling also deserves attention. A server that returns 200 OK and the same shell for an unknown route can produce a soft 404. Google specifically identifies this as a risk for client-side routed SPAs and suggests making the server ultimately return a genuine 404, or adding a noindex rule to the error route. Google's SPA error guidance

Robots controls, authentication, and interaction-only content

robots.txt is a crawler protocol, not a confidentiality control. The IETF Robots Exclusion Protocol specifies a method for service owners to control how automatic crawlers access served content. It does not turn an already-public resource into a private one, and a system that does not honor a policy is not stopped by a text file. Use authentication and authorization for accounts, customer data, previews, and anything that must not be public. RFC 9309

Use page-level noindex or the X-Robots-Tag response header when the goal is to request that search engines not list a page. These rules must be discoverable by an allowed crawler, and Google notes that support can vary among other search engines. Do not block a URL in robots.txt and then expect a crawler to find the noindex instruction inside that blocked page. Google robots-meta documentation

Treat authentication as a separate test dimension. A chatbot fetch, search crawler, and unauthenticated renderer will usually not have a user's session, but do not rely on that assumption for security. Verify that protected responses return an appropriate 401, 403, or login flow and do not contain sensitive records in initial HTML, hydration data, JavaScript bundles, or public APIs. Google also advises using meaningful status codes, including 401 for pages behind a login. Google's JavaScript SEO basics

Content that appears only after a scroll event, a consent choice, a click, a client-side search, or an API call tied to personal state is another boundary. A renderer may not trigger that interaction, may not wait long enough, or may have no permission to make the request. If the content is intended to be publicly found, give it a stable public URL and an initial representation. If it is meant only for an authenticated user, keep it protected and do not judge it by public-crawler visibility.

A reproducible diagnostic procedure

Do the following on representative public routes, not only the home page. Capture a baseline before changing architecture so that a later migration can be evaluated rather than guessed at.

1. Capture the raw response. Save headers and HTML from a clean, unauthenticated request. Check the status code, redirect chain, Content-Type, cache behavior, X-Robots-Tag, canonical link, title, description, main heading, and whether the public body text is present. A harmless baseline command is:

   curl -sSIL https://www.example.com/guide/widget
   curl -sSL https://www.example.com/guide/widget -o raw-widget.html
   

   Inspect raw-widget.html locally. A custom user-agent string can help label your own test, but it does not turn the request into a real search or assistant bot and should not be treated as one.

2. Capture the rendered DOM from a clean browser. In Chrome, compare View Source, which represents the response document, with the Elements panel after a hard reload. Then automate the same check in a fresh browser profile so the result is repeatable. This small Playwright example checks text after the page has finished loading:

   import { chromium } from 'playwright';
   
   const browser = await chromium.launch();
   const page = await browser.newPage();
   await page.goto('https://www.example.com/guide/widget', { waitUntil: 'networkidle' });
   console.log((await page.locator('main').innerText()).slice(0, 500));
   await browser.close();
   

   Record the browser version, viewport, wait condition, console errors, failed requests, and final DOM snapshot. The browser result proves renderability under those chosen conditions. It does not prove that every remote service uses the same conditions.

3. Inspect server, CDN, and application logs. For each test, retain the request time, URL, status, response size, cache outcome, redirect, user agent, IP verification result if available, and failures loading JavaScript or API data. Do not treat a user-agent string alone as bot authentication. Google warns that client-side analytics may not accurately represent Googlebot and its Web Rendering Service activity, and recommends Search Console crawl data for Google-specific monitoring. Google's JavaScript troubleshooting guidance

4. Use the diagnostic for the system being discussed. Google Search Console's URL Inspection and Rich Results Test are appropriate for testing Google rendering. A Lighthouse run is appropriate for its browser audit. A Claude link-analysis result is a sample of that feature under that account and prompt. Keep those results in separate columns rather than combining them into a single pass or fail result.

5. Test state deliberately. Repeat the check without cookies, with a test account when the audit supports authentication, and after clearing caches. Make sure a cached public page never includes a user name, account balance, or private bootstrap data. For a page with geo or consent variants, test a defined baseline region and consent state.

The point is not to simulate every bot. It is to know exactly which representation you are serving and to eliminate accidental dependency on a browser capability when the public page needs to stand on its own.

Architecture choices

For many teams, the hybrid option is the default that best matches the problem. Statically generate or server-render pages that explain the product, answer public questions, and need stable metadata. Run the product workspace as a client-rendered application behind authentication. This is not two different truths. It is one public representation for shared information and one protected representation for individual work.

SSR is not automatically superior. Do not render per-user data into HTML simply because SSR is available. A public product page may be a good SSR candidate, but a personalized inbox is not. Static generation is usually cheaper and more reliable for content that can tolerate a controlled publishing or regeneration delay. A pure SPA remains appropriate where a public page has no real meaning apart from a signed-in session.

Example

Hypothetical example: a Vite React company site serves /guides/secure-backups as a document with <div id="root"></div> and a JavaScript bundle. The route's heading, body copy, title, JSON-LD, and related-guide links appear only after useEffect fetches a content API. A raw capture has no guide text, while a local browser eventually displays it.

The team changes only public guide routes to static generation. The build produces HTML containing the title, canonical URL, <h1>, article body, normal links, and accurate JSON-LD. React hydrates a feedback widget and a save button after load. The private administrative editor remains a client-side app behind login.

The decisive outcome is not a claim about unnamed AI tools. The raw capture and rendered-DOM capture now both contain the guide's main text and route metadata, while the interactive controls still work after hydration. That reduces dependency on rendering capability without exposing private data or requiring a rewrite of the whole product.

Implementation checklist

• Classify every route as public content, public but personalized, authenticated product, preview, or administrative. Assign an owner and intended rendering mode.
• For public canonical routes, compare raw HTML and a fresh rendered DOM in continuous integration. Fail the check when the page title, canonical URL, main heading, or required body text is absent from the representation your policy requires.
• Generate page-specific <title>, meta description, canonical link, and social-sharing metadata from the same content record used for the visible page. Avoid a generic app-shell title across routes.
• Use real HTTP status codes at the server or edge for redirects, missing pages, expired pages, and protected routes. Test deep-linked routes directly, not only client navigation from the home page.
• Put accurate structured data on appropriate pages, validate it, and keep it consistent with visible text. Google recommends JSON-LD as the easiest format to maintain at scale. Google structured-data guidance
• Keep important public links as normal anchors with stable URLs. Do not require a menu click, endless scroll, or a client-side search interaction to reveal the only path to public material.
• Audit robots.txt, robots meta tags, and X-Robots-Tag at deploy time. Make the policy explicit for the named agents that publish documented controls, and retain authentication as the actual privacy boundary.
• Render public content on the server or during the build when it relies on data. Keep browser-only logic in client components and make its initial fallback acceptable.
• Monitor origin and CDN logs for failed assets, API errors, unexpected status codes, and suspicious cache variants. Protect log access because URLs and user agents can be sensitive operational data.
• Re-run representative raw, browser, search-console, and accessibility tests after a routing, rendering, consent, CDN, or authentication change.

Common failure modes and the next test