How to Format Your Website So AI Crawlers Can Actually Read It

When an assistant names a brand, it’s repeating something it read. Before ChatGPT or Google’s AI Overview can recommend you, a crawler has to visit your page, read the text, and store what it found. That crawler is not a person. It doesn’t admire your photography or notice your brand colors. It reads text — and if the thing it needs to know isn’t there as text it can read, that fact does not exist as far as the machine is concerned.

This is the least glamorous part of getting found by AI, and the most decisive. You can write the perfect page and still be skipped because of how it’s built. Here are the six formatting fixes that decide whether a crawler can read you.

1. Put the facts in on-page text, not in an image or a PDF

This is the one that trips up the most brands. Your tasting notes are in a beautiful PDF sell sheet. Your shipping states are baked into a graphic. Your story is a scanned menu. A crawler reads none of it — a picture of text is, to a machine, just a picture. The same goes for a PDF, which many crawlers skip or read poorly. Whatever you need an assistant to know — what you make, where you ship, what it costs — write it in plain HTML text on the page. If a fact only lives in an image or a download, it’s invisible.

2. Use real headings, in order

Headings are how a machine understands the shape of your page. One H1 that names the page, then H2s and H3s that each say what the section answers. This isn’t about font size — text that merely looks big and bold but isn’t marked up as a heading tells the crawler nothing. Proper headings let the machine see “this section answers the shipping question, this one answers the pairing question” and pull the right piece for the right query.

3. Answer the question in the first two sentences

Assistants pull the extractable answer that sits near the top of a section, not the payoff buried in paragraph six. Lead with the plain answer, then explain. “We ship to 42 states. Here’s how it works…” beats three paragraphs of story before the fact a buyer — and the AI on their behalf — came for. (This is the whole idea behind an answer-first page.)

4. Don’t hide your content behind a click or a script

If your key text only appears after someone clicks a tab, expands an accordion, or waits for JavaScript to fetch it from somewhere else, a crawler may never see it. Some do render scripts; many don’t, and none are guaranteed to. The safe rule: the content that matters should be in the page’s HTML when it first loads, not assembled afterward. Fancy interactions are fine as long as the words underneath them are really there.

5. Check that you’re not blocking AI crawlers

Your robots.txt file — and sometimes your host’s firewall or bot protection — can quietly turn AI crawlers away at the door. The crawlers behind the big assistants have names: GPTBot (ChatGPT), Google-Extended (Google’s AI), PerplexityBot, ClaudeBot. If your rules block them, you have opted out of being cited without meaning to. Open the file, confirm you’re not disallowing them, and you’ve cleared the single most common invisible blocker. (More on telling whether AI can see you.)

6. Label your facts with structured data and an llms.txt

Once the text is readable, two additions make it quotable. Structured data (schema.org markup) tags each fact so the machine knows this is the author, this is the price, this is the question and its answer — the difference between a page a machine can read and one it can confidently quote. An llms.txt file is a short plain-text brief that tells an assistant who you are and what your key pages are, in language built for it. Neither is required to be read, but both make you far easier to name correctly.

The test

Here’s a two-minute check. Open your most important page, and use your browser’s “view source” or select-all-and-copy. Can you see — as plain, selectable text — the facts you’d want an assistant to repeat about you? If yes, a crawler can read them. If the important things are missing because they live in an image, a PDF, or a script, you’ve just found exactly why the machine can’t name you — and exactly what to fix first.

Zillah Bahar is the founder of COLAClear and writes about getting found by AI at GetDiscoverable.io.