llms.txt, robots.txt and structured data for AI search
Three technical jobs that often get mixed up: access, discovery and meaning.
In this article
Short answer
Robots.txt gives crawling instructions. Links and XML sitemaps help systems discover URLs. Structured data describes visible information. These are separate jobs. OAI-SearchBot and GPTBot have different purposes, and Google does not require llms.txt or special AI schema.
Key points
- Search crawling and model training are separate choices.
- Check firewalls and server responses as well as robots.txt.
- Robots.txt does not protect private information.
- Use structured data to describe visible facts accurately.
- Treat llms.txt as an optional proposal with a specific purpose, rather than a ranking requirement.
An extra file cannot repair a blocked page, and more schema cannot turn thin content into a reliable source. Start by identifying the technical problem you actually need to solve.
This technical layer needs quality content and an established reputation alongside it.
Guide to ChatGPT SEOAccess, discovery, meaning and selection
| Layer | Question | Useful controls |
|---|---|---|
| Access | Can the crawler fetch the page? | Robots rules, server responses, authentication and firewall rules |
| Discovery | Can the system find the URL? | Internal links, sitemaps and relevant external links |
| Meaning | What does the visible information describe? | Clear HTML and suitable structured data |
| Selection | Is it useful for this question? | Relevance, evidence, freshness and wider context |
OAI-SearchBot and GPTBot do different jobs
OpenAI describes OAI-SearchBot as its search crawler and GPTBot as a crawler used in connection with potential model training. Those choices can be managed independently. Allowing the search crawler does not require allowing the training crawler.
A site can allow OAI-SearchBot while disallowing GPTBot. Check the current official documentation and make a deliberate policy choice. Do not change a customer's training policy as an incidental SEO fix.
Also check the CDN, firewall and actual server response. A robots allowance cannot help when another layer blocks the request. Use current published crawler information rather than trusting a user-agent name alone.
Read robots rules in context
A separate user-agent group can allow OAI-SearchBot with 'Allow: /', while a GPTBot group can use 'Disallow: /'. This is an example of two independent choices, not a recommendation to replace your whole robots.txt.
Keep any existing protections and relevant exclusions. Check the file actually served after a change and ensure the public sitemap URL is correct. Do not accidentally block assets needed to understand the public page.
Robots.txt is public crawling guidance. Protect private, account and administration areas with real authentication and server permissions. A disallow line is not access control.
Perplexity and user-triggered requests
Perplexity publishes guidance for PerplexityBot and its user-triggered fetcher. OpenAI also distinguishes its search and training crawlers from ChatGPT-User requests. Read each platform's current documentation before assuming all of its traffic follows the same rules.
Avoid a blanket 'all AI bots' rule unless its effects are understood. It may affect search discovery as well as training-related crawling. Record the purpose of each change and verify the response without weakening authentication or firewall protection.
What llms.txt can and cannot do
llms.txt is a proposed way to present selected documentation and links for language-model workflows. It may have a practical purpose for a system that actually supports it. It is not a generally accepted ranking standard.
Google says llms.txt is not required for its generative search features and is treated neutrally. It does not replace normal page access, internal linking or useful content.
If you use the file, know which system uses it, who maintains it and why those URLs were selected. Keep it public, accurate and consistent with the visible website. Do not put secrets, private URLs or special claims in it. A successful internal agent workflow is not evidence of improved public search rankings.
Use structured data for accurate meaning
Choose existing types that match visible content, such as an organisation, person, service or article. Keep the names, prices, dates and author details consistent with the page. Do not add unsupported reviews, invisible claims or a fictional location.
Valid JSON is only one part of the check. The meaning also needs to be correct. Use canonical absolute URLs and real publication or modification dates. Remove markup when the visible information no longer supports it.
Google does not require a special AI schema. Correct markup can help describe information, but it does not guarantee an AI citation, rich result or recommendation.
Check the foundation in a useful order
First check that the intended public URL returns the right page without a login or server error. Then check crawling and indexing directives, the canonical URL and relevant assets. Review internal links and the XML sitemap next.
Validate suitable structured data and make sure the visible page answers the intended question. Keep the findings separate: accessible does not mean indexed, indexed does not mean selected, and selected does not mean clicked.
For the wider context, read what content AI search can cite. Technical access gives useful content a chance to be reached. It does not replace the content itself.
Conclusion
Use each tool for its real purpose: crawling instructions for access, links and sitemaps for discovery, and structured data for meaning. A GEO audit checks those foundations alongside the actual answers and sources.
Common questions
Must I allow GPTBot to appear in ChatGPT Search?
No. OpenAI separates OAI-SearchBot for search from GPTBot for potential model training. Search access still does not guarantee a mention.
Does Google require llms.txt?
No. Google does not require it for generative search and describes its treatment as neutral.
Can robots.txt secure private pages?
No. Use authentication and appropriate server permissions for private information.
Does a sitemap guarantee an AI citation?
No. It helps discovery but cannot guarantee crawling, indexing or source selection.
Sources
This article uses the following external sources.
Author
Timothy Hartog, SEO specialist and web designer at Hartog Growth.