The race is decided before the content loads
Most conversations about AI search visibility start with content, and most visibility failures we audit start somewhere else entirely: the machine never got a fair look at the page. A crawler that receives an empty HTML shell, a blocked bot, or a sitemap returning errors does not rank you lower. It does not see you at all.
I lead the technology side at MEDITR, and before any SEO, GEO, or AEO work starts we run the same infrastructure checks on every site. This article is that checklist, in plain language, so you can hold any supplier to it, including us.
Rendering: your content must exist as HTML
Modern sites are often built as JavaScript applications. The browser downloads a mostly empty page, runs code, and only then does the content appear. People never notice. Crawlers do. Googlebot can execute JavaScript, at a cost and with delays, but many AI crawlers fetch pages the simple way and read whatever HTML comes back. If your service descriptions only exist after JavaScript runs, then for a meaningful share of the machines deciding your visibility, they do not exist.
The fix is server-side rendering or static generation: the server sends finished HTML, and any script on top is enhancement rather than load-bearing structure. This site works that way, and every site we build does. The test takes thirty seconds: view the page source, not the browser inspector, and search for a sentence from your homepage. If it is not in the source, that is the first thing to fix, before a single word of new content is written.
Crawler access: robots.txt is now a commercial document
robots.txt used to be a file nobody touched after launch. It now quietly decides which answer engines can consider you. OpenAI's OAI-SearchBot, Anthropic's crawler, PerplexityBot, and Google's crawlers all check it, and we still find sites whose old configuration blocks them wholesale, usually left over from a developer blocking everything on a staging server and forgetting.
Our position: allow the AI search crawlers deliberately, block the paths that should stay private, and write it down as policy rather than accident. Alongside it, the basics that were always true still hold. A sitemap that returns real URLs, redirects that preserve every address the engines already know, and clean indexation are the plumbing everything else flows through. We have audited sites paying for monthly SEO while their sitemap returned a 404, which meant the engines had no working map of the site at all.
Structured data: speaking to machines in their format
Everything else on your site requires a machine to infer what you mean. Structured data lets you state it. JSON-LD blocks in the page head declare, in a format every major crawler parses, that this is an organisation with this name, this address, these services, these authors, and these frequently asked questions.
For a service business we implement four types as standard.
None of this is visible to visitors, which is exactly why it is still rare outside the biggest brands, and why it remains a differentiator. When an AI assistant needs to say who you are with confidence, schema is the difference between a verified claim and a guess.
- Organization, with the real legal and contact details that let an engine confirm you exist
- Service, one per offering, stating plainly what you do and where you operate
- FAQPage, only where genuine questions and answers already live on the page
- Person, for named authors linked to their public profiles, which supports the expertise signals AI systems increasingly weigh
The emerging layer: llms.txt and agent-readable pages
A newer file, llms.txt, sits at your site root and gives language models a curated map: what the site is, what matters, where the important pages are. The honest status report is that some AI crawlers read it, none punish its absence, and Google has stated it has no effect on Search or AI Overviews. We add it to every build because it costs minutes and removes one friction point, and we would never sell it as the thing that moves this quarter's numbers.
The same logic applies to the wider agent-readable web: exposing services and prices in formats software agents can consume. It is where things are heading, and being early is cheap. Being early is not the same as being carried by it, and a supplier who leads their pitch with llms.txt is usually avoiding harder conversations about rendering and indexation.
Speed, stability, and the checklist to hold us to
Fast pages win twice: visitors convert better, and crawlers with limited budgets fetch more of your site per visit. Core Web Vitals, image discipline, and stable hosting are unglamorous, permanent work.
Held together, the checklist is short. Real HTML in the page source. AI and search crawlers allowed in robots.txt. A sitemap that works and redirects that preserve old URLs. JSON-LD for organisation, services, FAQs, and authors. Fast pages. llms.txt as cheap insurance. Nothing on that list requires a big budget, and everything on it decides whether the content you pay for can ever be found, extracted, and quoted. Infrastructure does not win the race by itself. It decides who is allowed to run.