SEO & AIO/GEO
llms.txt and ai.txt Explained: Should Your Website Have Them?
llms.txt and ai.txt are new, informal files that describe your site to AI systems. They are cheap to add, but it is important to be honest about what they can and cannot do. In this guide we explain what llms.txt, ai.txt and robots.txt are, how they differ, what the evidence says about their impact, and how to set them up sensibly for a business website.
- What is llms.txt?
- What does an llms.txt file look like?
- What is ai.txt?
- robots.txt: the control that actually matters
- Do search engines and AI tools use these files?
- How to create llms.txt step by step
- What to include and what to leave out
- Common mistakes
- How to check that AI crawlers can reach your site
- A realistic example for a service business
- Why consistency matters more than any single file
- A sensible setup for a service business
What is llms.txt?
llms.txt is a plain-text file written in Markdown that sits at the root of your site, for example yourdomain.com/llms.txt. It gives language models a short, human-readable summary of your business and links to your most important pages. The idea is similar to a sitemap, but written for AI readers: instead of listing every URL, it highlights the ones that matter, with a sentence explaining each.
What does an llms.txt file look like?
A simple version has a title, a one-sentence summary and a few short sections of links.
# NRY Media
> Digital marketing agency for US businesses: SEO, AI search (GEO), Google Ads, Meta Ads and web design.
## Services
- [SEO services](https://nrymedia.com/services/seo-aio-geo/): Technical SEO, content and AI-search optimization.
- [Google Ads](https://nrymedia.com/services/google-ads/): Search and Performance Max management.
## Company
- [About](https://nrymedia.com/about/): Founders and experience.
- [Contact](https://nrymedia.com/contact/): Phone, email and form.Keep it short, factual and current. It should read like a clear introduction, not a sales pitch.
What is ai.txt?
ai.txt is a proposed file for telling AI systems how they may use your content. Unlike robots.txt, it is not a recognized standard, there is no agreed format, and crawlers are not required to read or obey it. Some site owners use it to state preferences, but you should not rely on it to enforce anything. If you use one, keep it consistent with your robots.txt and your terms.
robots.txt: the control that actually matters
robots.txt is the long-established way to tell crawlers which parts of your site they may access. Many AI companies publish the names of their crawlers so that you can allow or block them. Examples include GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot and Claude-SearchBot from Anthropic, PerplexityBot from Perplexity, and Google-Extended and Applebot-Extended, which control certain AI uses of your content.
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://nrymedia.com/sitemap.xmlIf you want AI assistants to be able to cite your pages, make sure you are not blocking them by accident through a security plugin, CDN setting or an old robots.txt rule. If you prefer to opt out of some uses, you can disallow specific crawlers, but remember that blocking can reduce your visibility in the related products.
Do search engines and AI tools use these files?
Here is the honest answer: there is no confirmation that major search engines use llms.txt to rank pages, and Google has indicated that it does not use it. Some AI tools and developer workflows can read the file, and it can help when a person or tool fetches your site as context. Treat it as a low-effort extra that may help in some situations, not as a ranking factor or a guarantee of being cited.
What does influence AI visibility is the same foundation as SEO: crawlable pages, clear answers, structured data, consistent entity information and credibility from independent sources.
How to create llms.txt step by step
- List your most important pages: services, about, contact, key guides and pricing information.
- Write a one- or two-sentence summary of what your business does and who it serves.
- Group links under short headings and add one clear sentence per link.
- Save the file as plain text named llms.txt and upload it to your site root so it loads at yourdomain.com/llms.txt.
- Check that the file is reachable, uses https and shows correctly in a browser.
- Review it whenever you add or remove important pages.
What to include and what to leave out
- Include your name, location or service area, services, founders or team, contact details and your best educational resources.
- Leave out thin pages, duplicate content, private or staging URLs and anything you would not want quoted.
- Do not stuff keywords or add promotional claims you cannot support. AI systems and people both penalize exaggeration.
Common mistakes
- Expecting llms.txt to boost rankings or guarantee citations.
- Forgetting to update the file after a redesign.
- Blocking AI crawlers in robots.txt while publishing an llms.txt that invites them.
- Listing every URL instead of curating the important ones.
- Publishing an ai.txt that contradicts your robots.txt or terms.
How to check that AI crawlers can reach your site
Start with the basics: open yourdomain.com/robots.txt and yourdomain.com/llms.txt in a browser and confirm that both load. Then review your hosting, CDN and security plugin settings, because some tools block unfamiliar bots by default. Check your server logs or analytics for visits from the crawlers you expect, and test a few important URLs with a fetch tool to confirm they return normal pages instead of challenges or errors. If something is blocked unintentionally, fix it before spending time on anything else.
A realistic example for a service business
Imagine a local accounting firm. Its llms.txt names the firm, the city it serves, its core services and its two most useful guides, such as a small-business tax checklist. Its robots.txt allows reputable crawlers and lists the sitemap. Its pages open with clear answers, carry Organization and Service schema and show real author bios. The llms.txt file is the smallest piece of that work; the real gains come from clear, trustworthy pages that anyone, human or machine, can understand quickly.
Why consistency matters more than any single file
Whichever files you publish, make sure they tell the same story as your site. If your llms.txt says you serve the United States, your pages, schema and profiles should say the same. Conflicting details confuse people and models alike. A short monthly review of your business details, key pages and crawler access keeps everything aligned as your site grows.
A sensible setup for a service business
Make sure robots.txt allows the crawlers you want and points to your XML sitemap. Add a concise llms.txt with your key pages. Spend the real effort on answer-first content, schema, reviews and consistent business details. If you want help, our AI-search optimization service covers all of this.