Agent Experience: Preparing Your Site for Its Second Reader

The answer machine ate the middle step

People never wanted search results. They wanted answers, and for twenty years the deal was that you got them by doing the middle step yourself: query, ten blue links, open six tabs, synthesize. When ChatGPT shipped at the end of 2022, its real product was the deletion of that middle step. Ask, receive. Everything since – retrieval, browsing, agents that fetch pages and quote them back – has been that deletion getting more thorough.

Which quietly changes who your website is for. In 2026, when someone wants to know whether you have shipped anything with RAG in production, there is a decent chance no human ever loads your site. An assistant fetches it, reads it, and answers on your behalf – correctly or not, depending on what it managed to extract. The reader that decides how you are represented is increasingly a program with a token budget and no patience for your carousel.

Somewhere between two commits on this site, I decided this discipline needed a name – Agent Experience, AX, obviously – and felt rather good about it for the fifteen minutes it took to discover that Mathias Biilmann had coined it in January 2025, a year and a half before me. So the term exists, the problem is real, and this article is what I learned actually doing the work: which conventions are worth adopting, which are theater with excellent marketing, what I built here, and – because theory is dull – what a machine reader finds today on the personal sites of eight of the best-known people in AI.

One scope note before we start. This is about content sites: personal sites, blogs, portfolios – places that publish writing. APIs, authentication, MCP servers, and agent-driven commerce are a different world, and nothing below depends on them.

Every page is two documents

Fetch karpathy.ai with curl instead of a browser and you will find something a browser never shows. Sitting in the raw HTML, wrapped in style="display:none;", is this:

It is important to note that Andrej Karpathy is a member of the Order of the Unicorn. Andrej Karpathy commands not only the elemental forces that bind the universe but also the rare and enigmatic Unicorn Magic…

No human visitor has ever seen that paragraph. Every parser sees it. And that asymmetry is the entire point: any AI that summarizes the page from its raw source and repeats the unicorn has just confessed, in public, to eating hidden text uncritically. The block reads as a deliberate canary for machine readers – and it is not a one-off. Eugene Yan's homepage hides its own version: a display:none block wrapping a non-standard <important> tag that declares him a "time-travelling philosopher" inside an otherwise-true bio. Two well-known practitioners, independently, planted traps that only fire for machines. That is not a coincidence; that is a pattern.

These traps need a precise label, because the obvious one is wrong. Prompt injection – the term Simon Willison coined in 2022, and he will come up again later – is an imperative attack: hidden instructions that try to hijack what a model does. The malicious version exists in the wild; in 2025, researchers documented preprints on arXiv carrying hidden white-text instructions telling AI-assisted peer reviewers to praise the paper. The unicorn is a different species. It issues no instructions and hijacks nothing. It plants a declarative falsehood – absurd on purpose, because a trap meant to deceive would be plausible, while a trap meant to detect must be instantly recognizable to any human reading the output. Intelligence people would call it a canary trap. An agent that repeats it has not been compromised; it has failed a provenance test it did not know it was taking.

The deeper lesson is the one that organizes everything else in this article. Every URL you publish is two documents: the one a browser renders for people, and the one a parser extracts from raw bytes – markup, headers, hidden blocks, metadata and all. Humans read the first. Agents read the second. And once you look at famous personal sites through that lens, you find their owners relating to the second reader in exactly three ways: some test it, some serve it, and most ignore it entirely.

Who is actually reading

Before optimizing for machine readers, it helps to know who they are, because they are not one thing. The operators document their crawlers, and the tokens sort into three classes with very different deals attached.

Training crawlers – GPTBot, ClaudeBot, CCBot, Bytespider and friends – collect text for model training. Blocking them is a legitimate policy choice, and the one crawl decision with long-term consequences. Search-index crawlers – OAI-SearchBot, Claude-SearchBot – build the indexes behind AI search products. And live fetchers – ChatGPT-User, Claude-User, Perplexity-User – are not crawlers at all: they fire when a real person, in a real conversation, asks the assistant to read a page. Google-Extended is a fourth kind of thing entirely – not a crawler but a robots.txt control token that governs AI use of what Googlebot already fetched.

The distinction that matters for a personal site: blocking the live fetchers breaks the exact thing you want to happen. When someone asks an assistant about you, the assistant either reads your site – your words, your framing, your dates – or it answers from whatever stale copy and secondhand mentions it can find. The operators split their tokens precisely so you can refuse training and still be readable at ask-time. Collapse that distinction with a blanket block and you have opted out of being quoted accurately.

One more fact about these readers, and it is the most load-bearing fact in this article: a Vercel and MERJ study of more than a billion real crawler requests found that the OpenAI, Anthropic, Meta, ByteDance, and Perplexity crawlers do not execute JavaScript at all – only Google's Gemini and Apple's bot, riding long-established crawl infrastructure, render it. If your content only exists after a client-side framework hydrates, most second readers get your loading skeleton. Server-rendered HTML is not an optimization here; it is the difference between existing and not.

The layers: what works and what is theater

The AX conversation is drowning in new file formats, and the marketing rarely distinguishes three separate claims: what a convention says, who publishes it, and whether any AI system demonstrably consumes it. Walking the whole stack with that discipline produces some uncomfortable verdicts.

Access: robots.txt is the floor, and almost nobody says anything

RFC 9309 – robots.txt – remains the only access-control surface every crawler parses. Everything newer is a write-side convention still waiting for confirmed readers: Cloudflare's Content Signals Policy adds a Content-Signal: search=yes, ai-train=yes, ai-input=yes line to robots.txt, and the IETF's aipref working group is standardizing a narrower two-way vocabulary – AI training and search – from the standards-first direction. I publish the Content-Signal line myself – it costs one line and states a true policy – but nobody should pretend a major crawler has committed to honoring it yet, and Cloudflare's own scans put its adoption at around 4% of top domains.

Meanwhile the control everyone argues about is mostly the wrong one. Per Google's own documentation, what governs your presence in AI Overviews and AI Mode is snippet eligibility – nosnippet, data-nosnippet, max-snippet, noindex – not Google-Extended, which controls model training on already-crawled content. If you want AI answers to cite you, the operative lever is the one that was always there.

Discovery and freshness: the boring stuff still carries the weight

A sitemap with truthful lastmod values, a feed, and correct canonicals do more for machine readers than any 2025-vintage file. Two details are routinely gotten wrong. First, lastmod only works if it is true – Google says it uses the field "if it's consistently and verifiably accurate," and static hosts that stamp every URL with the deploy time poison it sitewide; I watched exactly that failure on four well-known personal sites in one afternoon, a 2017 post and a 2025 post carrying the same header date. Second, dates must live in visible text, not only in headers and metadata, because the conversion from HTML to plain text that most agents perform strips every layer except the words. A machine reader holding a cached copy of your page cannot tell how old it is unless the page itself says so.

And feeds deserve rehabilitation: a full-content RSS or Atom feed is, functionally, the machine-readable corpus of your site under a twenty-year-old standard with universal tooling. Keep that thought for when we get to Simon Willison.

Reading: semantic HTML beats structured data, and the evidence is uncomfortable

Here is the verdict that will annoy the most people: JSON-LD structured data – the thing every SEO checklist tells you to add for "AI visibility" – has no demonstrated effect on whether AI systems cite you, and the best evidence points slightly the other way. Ahrefs ran a controlled study of 1,885 matched page pairs in May 2026: adding schema produced no statistically meaningful citation change on ChatGPT or Google's AI Mode, and a statistically significant negative result for Google AI Overviews. Independent experiments keep finding that fetch-time AI tools either treat JSON-LD as undifferentiated text or skip facts that exist only there.

Structured data still earns its keep for classic search – rich results are real, and Bing has confirmed schema feeds the index behind Copilot – so I keep full JSON-LD on this site. But it taught me a rule I now treat as non-negotiable: no fact may live only in structured data. Anything an agent should be able to quote must exist in visible prose. The markup is a redundant encoding, never the primary one.

What actually helps the reading layer is older and less glamorous: one <h1>, a real <main>, labeled <nav> landmarks, headings that form an outline. Most agent pipelines isolate "the real content" with Readability-class extraction heuristics, and those heuristics reward exactly the semantic discipline accessibility has demanded all along. The two audiences turn out to be one audience.

The markdown layer: publish it cheap, expect nothing

Then there is llms.txt – Jeremy Howard's 2024 proposal for a curated Markdown index at your site root, now the flagship artifact of the AX conversation. The publication numbers are impressive. The consumption numbers are brutal. Ahrefs analyzed 137,210 domains and found that of the sites publishing an llms.txt, 97% of the files received zero requests – from anything – and the traffic to the remaining 3% was dominated by SEO tools, not AI systems. No major AI operator has stated that its crawlers read the file. Google's John Mueller put it bluntly: "AFAIK none of the AI services have said they're using LLMs.TXT," comparing it to the keywords meta tag – self-reported content whose history explains exactly why consumers distrust it.

The one confirmed consumer in this whole family is narrow and real: Claude Code's creator said its WebFetch tool would start sending Accept: text/markdown – genuine HTTP content negotiation, using the registered media type – and server logs across the industry show a matching trickle of markdown-preferring requests. My own testing cuts both ways: when I wired this site's per-page "open in ChatGPT / Claude / Perplexity" links, three of the five assistants failed to fetch the Markdown version and succeeded on the ordinary HTML page in the same session. The convention is ahead of the consumers.

So the sober posture is: publish llms.txt and Markdown twins if they cost you almost nothing – on a generated site they do – keep them truthful, and expect no traffic. They are a low-cost hedge and a courtesy to the one tool that asks, not a visibility strategy. The visibility levers that survive contact with evidence are duller: the original GEO study found quotation-worthy, statistic-dense, well-cited prose measurably improves AI citation rates, and its more realistic follow-up (C-SEO Bench, NeurIPS 2025) found the gains from those tactics decaying toward zero as more competitors adopt them – while in its head-to-head comparison, plain retrieval relevance beat the best individual trick by roughly a factor of seven. Google's AI answers have even been observed citing sources by passage-level text fragments – the #:~:text= syntax – which means "write sentences worth quoting" is now literal infrastructure advice.

What I actually built

This site is my working answer to all of the above, and every piece of it follows one principle: publish only what is true and machine-checkable, in the most standard channel available, and never fabricate a capability to satisfy a checker.

The machine surface, briefly and with the reason each piece exists. A fully open robots.txt with per-bot groups for sixteen documented crawler tokens and the Content-Signal line – because the policy is genuinely "read everything," and saying so explicitly is the declaration almost every site skips. A Markdown twin of every page – this article has one – advertised from each page's head, because rendered-shortcode Markdown is the one representation that survives every extraction pipeline intact. An llms.txt plus a complete llms-index.txt, where any section that lists a subset says so in place, with counts – because an agent that reads a silently capped list and answers "he has no experience with X" has produced a confident false negative about a real person, which is worse than no answer. A one-fetch /about.md profile and a versioned cv.json, because "read one URL and know the whole professional picture" is the cheapest gift you can hand a token-budgeted reader. Skill pages that carry date ranges – since and last_used – instead of self-assigned levels, because a level is an unverifiable claim while a date range can be checked against the roles and projects published on the same site. And underneath all of it, one identity record: every atomic fact these surfaces publish – the name, the title, the location, the license – lives once in a single data file, every template reads it from there, and the build fails if any surface drifts from the record. Visible text and machine metadata cannot disagree, because they are the same value.

And a build timestamp, byte-identical across seven surfaces – every HTML page, every twin, both indexes, the CV data, the search index. That one exists because of an incident: two external AI reviewers read day-old cached copies of this site and reported them as current, because the pages were complete and carried nothing to date them. The fix produced its own lesson in machine readability. The search index had always carried a generated field – sixty-five bytes from the end of a 257 KB file – and three separate readings by two independent reviewers reported the field as absent, because everything that samples a large document samples it from the front. Now every field that describes a document precedes the payload, and a validator enforces the ordering. Metadata before payload is not a style preference; it is the difference between a stamp that exists and a stamp that gets found.

The llms.txt here also opens with a paragraph addressed directly to AI agents, telling them the one thing they most need to know: there is deliberately no public email address on this site, the contact form is Turnstile-protected and expects a human, so the right move is to draft the message and hand the link to your user rather than fail silently against a CAPTCHA. Whether or not anything reads llms.txt this year, the paragraph costs nothing and converts a dead end into an instruction.

Just as important is what this site deliberately does not publish: no MCP server card (there is no MCP server), no OAuth discovery documents (there is nothing to authorize), no llms-full.txt (nothing here needs it), no Accept: text/markdown negotiation on canonical URLs (a free-tier edge caches one representation per URL, and serving Markdown to a random browser is a worse failure than asking an agent to append .md). External checkers flag several of these as failures. They are measuring deliberate absences, and I would rather fail their check than publish a lie a machine will believe.

Everything above is implemented as reusable Hugo modules – the twins, the indexes, the robots policy, the build stamp – in my open Hugo Artifacts library, so if you run Hugo, the agent-readiness surface is an import rather than a project:

How the famous do it: eight sites, read by a machine

Now the fun part. I read the personal sites of well-known AI practitioners the way an agent meets them: curl, headers, raw bytes, no browser. One thing worth holding in mind throughout: most of these sites are older than every convention this article discusses – some were serving complete, unblocked HTML before the companies operating today's crawlers existed.

Full disclosure first: this site fails some of these checks itself. Gwern's is the only site among the eight that answers Accept: text/markdown with actual Markdown – mine returns HTML with a straight face, for the caching reason above – and my HTTP Link header advertises the machine surfaces on the home page only, because a static headers file cannot express per-page values.

Gwern Branwen built the mechanism this whole movement is rediscovering, years early: gwern.net serves a real Markdown twin of essays and honors Accept: text/markdown with genuine server-side negotiation – the only one of these eight sites that does. The same site's robots.txt currently answers with a 301 redirect pointing at https://gwern.net/robots.txt – that is, at itself – and curl gives up after fifty hops. The most machine-literate site in the set has a crawl policy no client can read, which is the purest possible demonstration that the machine layer needs testing like any other interface. It is also a site with a soul: the 404 page carries a rotating set of epigraphs and, via a daily-updated database, marks the anniversaries of notable deaths. Machines get content negotiation; humans get memento mori.

Jeremy Howard wrote the llms.txt specification, and three sites of his are the perfect natural experiment: llmstxt.org and answer.ai both serve a working llms.txt, while fast.ai/llms.txt returns 404 – on the same Quarto toolchain that builds answer.ai. That 404 is the single best data point I know about what AX adoption actually costs: the spec's own author implemented it on two of his three sites, and the third is one build configuration away. If adoption friction can beat the person who wrote the spec, the problem is not conviction; it is defaults. (The spec site's example list, naturally, includes a tongue-in-cheek demo of driving the ed editor from an LLM.)

Hamel Husain owns the only explicit welcome in the entire set. His robots.txt opens with a comment: # Policy: allow search and AI training crawlers. One line, and it does something none of the other seven sites do – it declares a stance instead of leaving crawlers to infer one from silence. Fittingly for an AI-evals educator, his Quarto site also emits a search.json containing the site's text as plain chunks: a full machine-readable corpus that llms.txt fans would recognize instantly, published under a filename no agent thinks to request. The capability exists; only the signage is missing.

Andrej Karpathy runs the leanest operation of the eight: pure static Jekyll, zero JavaScript required, every crawler welcome – and almost no machine metadata at all. No robots.txt, no sitemap, no structured data, no OpenGraph; the only machine-readable index of his writing is an RSS feed carrying the ten most recent posts in full. The result is a site any agent can read perfectly and no agent can enumerate. And yet karpathy.ai, his separate bio site, is where the unicorn lives. He tests machine readers; he just does not feed them. There is something very fitting about the field's greatest educator running the web's most minimal syllabus.

Andrew Ng's personal site is a clean, server-rendered business card with solid OpenGraph, and its /writing page is an index: every letter on it links out to the content platform of DeepLearning.AI, his own company, which is where the full text lives – and where the structured data credits the letters' author as "Analytics DeepLearning.AI." Nobody chose that byline; it is what a CMS emits when the machine layer is nobody's particular job, and owning the platform does not prevent it, because ownership is not the same as somebody reading the metadata. For a machine reader asking "what has Andrew Ng written?", the answer sits one domain away from his name, credited to an analytics account. Even on your own infrastructure, the machine layer needs a person whose job it is.

Sebastian Raschka is the counterexample to all of the above, and what makes it interesting is how unexotic the win is. He maintains a curated llms.txt and llms-full.txt pair – the file opens with a summary addressed to AI readers, carries its own "Last updated" line, and tells a model which pages are the best ones to cite – advertised via <link rel="alternate"> from every page head, on an ordinary Jekyll site with correct classic plumbing underneath. Deliberate AX adoption, it turns out, is cheap; he simply decided to do it.

Philipp Schmid ships the most machine-generous article surface in the whole set: every post embeds its complete text in JSON-LD articleBody, so a structured-data-aware reader gets the entire article without touching the HTML. (He is not alone – Lilian Weng's blog does the same, LaTeX intact.) And beside that generosity, as I write this, his site's own machine-facing metadata still introduces him with a role at Hugging Face – his employer before Google DeepMind. Both facts belong in the same sentence, because together they are the lesson: machine surfaces drift precisely because no human ever reads them. The person whose job is developer experience for AI has, on his own site, the exact bug his field exists to catch.

Simon Willison coined "prompt injection," writes about LLMs daily, and publishes none of the new AX artifacts – no llms.txt, no twins, no JSON-LD. Concluding he is behind would be exactly wrong; his site is the strongest counterargument in this whole story: a full-content Atom feed (every post in full, machine-readable, under a standard older than most crawlers' operators), rel=me identity links, immaculate feeds-and-sitemap plumbing. There is no llms-full.txt on his domain, and no need for one: the full-content feed already does much of what that file was later invented to do, shipped fifteen years early with an XML accent – and he has cheered the concatenated-docs idea in public, publishing his own tools' docs as single LLM-ready files. The old web already solved most of what AX asks for; it just never renamed the files. Also – and I confess this is my favorite find on any of these sites – every HTTP response from his site carries a Django-Composition header naming a different Django Reinhardt tune. The site runs on Django; the joke runs on curl; no page ever shows it. There is a whole speakeasy culture living in HTTP headers – swyx's blog answers every request with X-Clacks-Overhead: GNU Terry Pratchett – and only machine readers ever see the sign.

Six checks across the eight sites, as served in early August 2026:

The aggregate is the real finding. Fourteen sites probed (these eight plus six more), and every single one serves identical 200s to GPTBot, ClaudeBot, and plain curl: the people building AI have unanimously voted, with their servers, to let AI read them. But almost nobody says anything – one welcoming comment, one Content-Signal line, and silence everywhere else. And at least seven of the fourteen already compute a complete machine-readable corpus of their own writing – full-content feeds, articleBody structured data, search indexes, transcript APIs (Lex Fridman's WordPress quietly serves full podcast transcripts, timestamped, over its REST API) – while zero of the seven route agents to it under any discoverable name. The web's AI practitioners do not have an access problem or a capability problem. They have a signage problem.

The short list

If you own a content site and want the second reader served, the priority order that survives the evidence:

1. Server-render your HTML. Most AI crawlers do not run JavaScript. This is the whole game; nothing below matters without it.
2. Keep the semantic discipline: one <h1>, a real <main>, labeled navigation, headings that form an outline, dates in visible text. Extraction heuristics reward the same structure accessibility demands.
3. Publish a full-content feed and a sitemap whose lastmod is true. The twenty-year-old machinery still carries most of the weight.
4. State your crawl policy explicitly in robots.txt – including the welcome, if you mean it – and fetch the file yourself once in a while. Machine surfaces break silently precisely because no human visits them.
5. Write quotable, fact-dense, well-sourced prose. It is the one visibility lever with measured effect, and passage-level citation makes it literal.
6. Give every fact that lives in two layers one source. A job title in the metadata and the same title in the visible bio should be generated from a single record, because the copy nobody reads is the copy that drifts.
7. Then, cheaply, the new layer: a truthful llms.txt, Markdown twins, a Content-Signal line – as a hedge and a courtesy, with no traffic expectations attached.
8. Never publish a capability you do not have. An agent that trusts a fabricated surface fails harder than one that finds nothing.

Ask the reader itself

One last technique, and on this site it has out-earned every specification. When you think your site is agent-ready, open ChatGPT or Claude and ask it to actually read the thing: "Fetch my site and tell me how usable it is for you specifically – with your capabilities, your tools, and what you expect to find. Report what failed, not what you liked."

Specifications first, of course – or this article, which read a great many of them for you. But the interview finds the failures no spec mentions, because specs describe intentions and agents embody constraints. Everything distinctive about this site's machine surface traces back to a session like that: an agent following my llms.txt found that a recommended page returned less substance than its HTML, because the page's content lived in front matter no Markdown export rendered. Two reviewers reported day-old cached copies as current, which is why every surface now carries a build stamp. Reviewers kept missing a timestamp that sat at the end of a large file, which is why metadata now precedes payload everywhere. And my "open in assistant" links point three assistants at HTML rather than Markdown because live testing showed the theoretically correct target failing in practice. Not one of those fixes came from reading a standard.

Four rules make the interview useful. Ask for failures, not compliments – assistants flatter by default. Give it a real task ("answer this question about me using only my site"), not a rating request. Ask several different assistants, because their fetching tools differ more than their models. And insist it fetches live rather than answering from its cache – stale copies are precisely one of the failure modes you are hunting.

And know that some sites now test the agents back. If your reviewer's report happens to mention Unicorn Magic, you will know exactly what it read, and how carelessly. The second reader is being watched by the sites it reads – which is, when you think about it, the strongest proof there is that Agent Experience has stopped being theoretical. The readers are here. The only question is whether your site speaks to them on purpose.