AI-Friendly Sites: Making Your Site AI/Agent-Friendly with Astro and Cloudflare Pages

Run this command:

curl -H 'accept:text/markdown' https://vercel.com/docs

You get raw markdown, not HTML. The response starts with # Vercel Documentation followed by clean text. No <div> tags, no navigation elements, no JavaScript. Just the content you need.

Agents like Claude Code do this by default. They save tokens and bandwidth by requesting machine-readable format. That is powerful.

Implementing this on Cloudflare Pages is tricky. Architecture differs from Vercel. The obvious solution fails. After hours of debugging, deployment failures, and documentation dives, I discovered why.

The Problem

Content negotiation via HTTP headers:

Client sends: Accept: text/markdown
Server checks header, returns appropriate content type
Both human and AI get what they need

Simple, standard, elegant.

On Cloudflare Pages with Astro, there is a fundamental architectural limitation:

Static files serve directly from edge cache, bypassing middleware.

Set export const prerender = true for performance. Astro generates HTML at build time. Cloudflare caches globally and serves directly. Middleware never runs.

I tried three approaches. All failed.

1. Middleware redirect: Detect Accept: text/markdown, redirect to API endpoint.

Problem: Static files bypass middleware
Result: Returns HTML, ignores redirect

2. Link header in API response: Add HTTP Link headers to API responses.

Problem: Cloudflare strips or ignores these headers
Result: Headers never appear

3. Response type modification in middleware: Check headers after context.next(), modify response.

Problem: Same as above, middleware never runs

The core issue: Cloudflare Pages prioritizes static edge caching over server-side logic for prerendered content. This is by design and essential for performance, but it breaks traditional content negotiation patterns.

The Solution

After research and testing: separate API endpoints combined with HTTP Link headers in HTML.

Architecture

HTML pages include <link rel="alternate" type="text/markdown" href="/api/markdown/posts/slug"> in the <head>.

Agents fetch /api/markdown/posts/slug directly and receive text/markdown.

How It Works

For Humans:

Visit /posts/my-article
Receive full HTML page
See <link rel="alternate" type="text/markdown" href="/api/markdown/posts/my-article"> in <head>
Agent crawlers discover the link

For AI Agents:

Fetch /api/markdown/posts/my-article directly
Receive Content-Type: text/markdown; charset=utf-8
Get raw markdown content
Parse and use

This uses HTTP Link headers, a standard mechanism for content negotiation.

Implementation

Step 1: Create API Endpoint

Create src/pages/api/markdown/[...path].ts:

import type { APIContext } from "astro";
import { getCollection, type CollectionEntry } from "astro:content";
import { getPath } from "@/utils/getPath";

export async function GET({ request }: APIContext) {
  const url = new URL(request.url);
  const pathname = url.pathname;

  let post: CollectionEntry<"blog" | "deepPosts"> | null = null;

  if (
    pathname.startsWith("/api/markdown/posts/") ||
    pathname.startsWith("/api/markdown/posts/drafts/")
  ) {
    const slug = pathname
      .replace(/^\/api\/markdown\/posts\//, "")
      .replace(/^drafts\//, "");
    const allPosts = await getCollection("blog");
    post =
      allPosts.find(p => getPath(p.id, p.filePath, false) === slug) || null;
  } else if (
    pathname.startsWith("/api/markdown/deep-posts/") ||
    pathname.startsWith("/api/markdown/deep-posts/drafts/")
  ) {
    const slug = pathname
      .replace(/^\/api\/markdown\/deep-posts\//, "")
      .replace(/^drafts\//, "");
    const allPosts = await getCollection("deepPosts");
    post =
      allPosts.find(p => getPath(p.id, p.filePath, false) === slug) || null;
  }

  if (!post) {
    return new Response("Post not found", {
      status: 404,
      headers: { "Content-Type": "text/plain" },
    });
  }

  const body = post.body;

  return new Response(body, {
    status: 200,
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "Cache-Control": "public, max-age=3600",
    },
  });
}

Key points:

Handles blog and deep posts collections
Uses Astro’s getCollection for content access
Returns raw markdown with correct content type
Includes cache headers for performance

Step 2: Add Link Header to Layout

Modify src/layouts/Layout.astro:

export interface Props {
  // ... existing props
  markdownURL?: string;
}

// In the <head> section:
{markdownURL && <link rel="alternate" type="text/markdown" href={markdownURL} />}

Pages using Layout include the alternate link when markdownURL is provided.

Step 3: Pass Markdown URL in PostDetails

Update src/layouts/PostDetails.astro:

const layoutProps = {
  // ... existing props
  markdownURL: `/api/markdown/${collection === "deepPosts" ? "deep-posts" : "posts"}/${getPath(post.id, post.filePath, false)}`,
};

Blog posts point to /api/markdown/posts/slug. Deep posts point to /api/markdown/deep-posts/slug.

Step 4: Handle Edge Cases

Default markdownURL to undefined in Layout:

const {
  title = SITE.title,
  // ... other props
  markdownURL = undefined, // Default for pages without it
} = Astro.props;

Conditional rendering:

{
  markdownURL && (
    <link rel="alternate" type="text/markdown" href={markdownURL} />
  )
}

Prevents build errors on pages without markdownURL.

Why This Works

1. Respects Cloudflare Pages Architecture

Keep prerender = true for blog posts:

Static HTML served from edge cache (fast, cheap)
No server computation per request
Global CDN benefits

API runs only when requested. No performance sacrifice.

2. Uses HTTP Standards

Link headers are standard (RFC 5988). Agents discover markdown automatically. No special configuration needed.

3. Simple and Maintainable

No middleware logic, no header manipulation, no edge cases:

API returns markdown
HTML includes link to API
Both work reliably

4. Tested in Production

Deployed and verified:

# Test API endpoint
curl -I https://slavakurilyak.com/api/markdown/posts/agent-stories
# Returns: Content-Type: text/markdown; charset=utf-8

# Test HTML with alternate link
curl -s https://slavakurilyak.com/posts/agent-stories | grep alternate
# Returns: <link rel="alternate" type="text/markdown" href="/api/markdown/posts/agent-stories">

Best Practices

Provide Clean, Structured Content

Agents work best with:

Well-structured markdown (headings, lists, code blocks)
Clear frontmatter (title, description, tags, dates)
Semantic HTML (even when consuming markdown)

Document Your Content Structure

Explain:

Collections available
URL structure
API endpoint locations
Example requests and responses

Consider Caching Strategy

API includes Cache-Control: public, max-age=3600. Cloudflare caches for an hour:

Reduces server load
Faster repeated requests

Monitor Agent Traffic

Watch analytics for:

Accept: text/markdown header requests
Direct API endpoint access
User-agent patterns

What Does Not Work

Vercel-Style Middleware Redirect

Why not redirect Accept: text/markdown to the API endpoint?

Does not work on Cloudflare Pages. Static files bypass middleware. HTML serves from cache before middleware runs.

Response Type Modification in Middleware

Check headers after context.next(), modify response.

Fails for the same reason: middleware never runs for static files.

Dynamic Prerendering

Check headers at request time, choose output format.

Every request hits a server function. Eliminates static caching benefits.

The Trade-Off

Every architecture has trade-offs:

Approach	Performance	Complexity	Standards-Compliant	Works on CF Pages
Middleware content negotiation	Degrades	High	Yes	No (Static files)
API + Link headers	Maintains	Low	Yes	Yes

For Cloudflare Pages, API + Link headers preserves performance while using standard HTTP mechanisms.

Implement It Yourself

If using Astro + Cloudflare Pages:

Create the API endpoint
Add markdownURL prop to Layout interface
Include conditional Link tag in <head>
Set default markdownURL to undefined
Deploy and test

Code is production-tested and ready to copy.

Verify it works:

# Test API endpoint
curl -I https://slavakurilyak.com/api/markdown/posts/agent-stories

# Test HTML with alternate link
curl -s https://slavakurilyak.com/posts/agent-stories | grep "alternate.*markdown"

Agents discover the markdown version through the Link header automatically.