llms.txt, robots.txt and what actually controls an AI crawler

robots.txt

A file at the domain root listing user agents and the paths they may or may not fetch. It is voluntary but honoured by every major crawler, and AI crawlers are named user agents in it like any other.

llms.txt

A proposed convention: a markdown file at the domain root that describes what the site is and links its most useful pages, so a machine reading it does not have to infer the structure from navigation. It is a courtesy, not a control.

Which agent is which

Writing an llms.txt worth reading

The version on this site is generated from the same data the pages render from, so it cannot drift out of date. It states who the person is, lists the case studies with their live domains, lists the capability pages, and says plainly which projects have no public capture and why. That last part is the only unusual thing in it, and it is the part I would keep.

1. Generate it from your real data, never maintain it by hand — a stale llms.txt is worse than none.
2. Say what the site is in the first three lines, in plain language.
3. Link the pages that answer questions, not the pages you want traffic on.
4. State what is absent and why. An honest gap is a stronger signal than a padded list.
5. Serve it as text/plain; charset=utf-8, and do not gate it behind bot protection.

The lever that actually moves

Neither file will make an assistant quote you. Access control decides whether you are eligible; the structure and honesty of your pages decide whether you are chosen. Spend an hour on robots.txt and llms.txt, confirm nothing is accidentally blocked, and then go and write pages that answer one question each.