Content Delivery

LLMify provides two ways to automatically serve Markdown instead of HTML:

1. Content Negotiation (autoServeMarkdown): Serves Markdown when a request contains an Accept: text/markdown header.
2. AI Bot Detection (enableBotDetection): Automatically detects known AI crawlers by their user agent and serves Markdown to them.

Content Negotiation is enabled by default. AI Bot Detection is disabled by default and has to be enabled deliberately: it forces Markdown on crawlers that did not ask for it, which some search engines consider cloaking.

Content Negotiation

When autoServeMarkdown is enabled, LLMify uses content negotiation to serve Markdown instead of HTML. If a request contains an Accept: text/markdown header, the same URL that normally returns HTML will return clean Markdown instead.

This approach is inspired by Cloudflare's Markdown for Agents. It allows AI agents to request your existing page URLs and receive structured Markdown without needing separate <slug>.md endpoints.

This works well in combination with the isRealUrlLlm setting. When both are enabled, llms.txt will point to your real page URLs, and AI agents can request those URLs with the Accept: text/markdown header to get the Markdown version.

AI Bot Detection

When enableBotDetection is enabled (disabled by default), LLMify detects known AI crawlers by their user agent and automatically serves Markdown to them.

The built-in bot list is sourced from the community-maintained ai-robots-txt/ai.robots.txt project and ships bundled with the plugin. It currently covers 141 crawlers (GPTBot, ClaudeBot, ChatGPT-User, PerplexityBot, Bytespider, CCBot, and many more) and is refreshed on each plugin release.

To detect bots that aren't in the bundled list (yet), use the additionalBotUserAgents config option.

Discovery Tag

When autoInjectDiscoveryTag is enabled (default), LLMify injects two <link> tags into the HTML head, as recommended by the llms.txt spec:

<link rel="describedby" href="https://your-site.com/llms.txt">
<link rel="alternate" type="text/markdown" href="https://your-site.com/about.md">

• rel="describedby" points every page to the site's llms.txt. It is only added when llms.txt is enabled for the site.
• rel="alternate" points to the Markdown version of the current page. It is left out on pages whose section or entry is disabled, and on pages known to have no Markdown.

This allows AI agents to discover the Markdown version of a page and the site's llms.txt without needing to know about content negotiation.

How It Works

When a request hits a page:

1. LLMify checks if the request has an Accept: text/markdown header or if the user agent matches a known AI bot.
2. If either condition is met, it returns the cached Markdown of the requested page.
3. If the page is not cached yet, LLMify renders the Markdown in the same request and caches it.
4. If the page has no Markdown (e.g. its template has no {% llmify %} blocks), the normal HTML page is returned instead.

Response Headers

All Markdown responses include the following headers:

Auto-served Markdown is returned at the page's own URL, so no canonical header is needed: the Vary header already signals that HTML and Markdown are representations of the same resource. Instead, the Link header also points to the dedicated .md URL with rel="alternate"; type="text/markdown". The dedicated <slug>.md URLs add a rel="canonical" link to the page URL, since the Markdown lives at a separate URL there.

Testing

You can test both auto-serve methods with curl:

curl -H "Accept: text/markdown" https://your-site.com/about

curl -A "GPTBot/1.0" https://your-site.com/about

curl -v -H "Accept: text/markdown" https://your-site.com/about 2>&1 | grep "< "

Blitz Integration

LLMify works with Blitz out of the box when Blitz serves its cache through PHP. LLMify tells Blitz to skip its cache for Markdown requests (content negotiation and detected AI bots), so they are passed through to LLMify. The Markdown itself is cached by LLMify, so these requests stay fast.

Blitz with Server Rewrites

If you use Blitz with Nginx server rewrites, Nginx serves cached pages directly from the file system without ever hitting PHP. In this case, LLMify cannot intercept the request.

To fix this, add a condition to your Nginx config that skips the Blitz cache when the Accept header contains text/markdown:

set $cache_path false;
if ($request_method = GET) {
    set $cache_path /cache/blitz/$host/$uri/index.html;
}
if ($args ~ "token=") {
    set $cache_path false;
}
# Skip Blitz cache for text/markdown requests (LLMify auto-serve)
if ($http_accept ~ "text/markdown") {
    set $cache_path false;
}

location / {
    try_files $cache_path $uri $uri/ /index.php?$query_string;
}