Run this command: ```bash curl -H 'accept:text/markdown' https://vercel.com/docs ``` You get raw markdown, not HTML. The response starts with `# Vercel Documentation` followed by clean text. No `
` tags, no navigation elements, no JavaScript. Just the content you need. Agents like Claude Code do this by default. They save tokens and bandwidth by requesting machine-readable format. That is powerful. Implementing this on Cloudflare Pages is tricky. Architecture differs from Vercel. The obvious solution fails. After hours of debugging, deployment failures, and documentation dives, I discovered why. ## The Problem Content negotiation via HTTP headers: 1. Client sends: `Accept: text/markdown` 2. Server checks header, returns appropriate content type 3. Both human and AI get what they need Simple, standard, elegant. On Cloudflare Pages with Astro, there is a fundamental architectural limitation: **Static files serve directly from edge cache, bypassing middleware.** Set `export const prerender = true` for performance. Astro generates HTML at build time. Cloudflare caches globally and serves directly. Middleware never runs. I tried three approaches. All failed. **1. Middleware redirect:** Detect `Accept: text/markdown`, redirect to API endpoint. - Problem: Static files bypass middleware - Result: Returns HTML, ignores redirect **2. Link header in API response:** Add HTTP Link headers to API responses. - Problem: Cloudflare strips or ignores these headers - Result: Headers never appear **3. Response type modification in middleware:** Check headers after `context.next()`, modify response. - Problem: Same as above, middleware never runs The core issue: Cloudflare Pages prioritizes static edge caching over server-side logic for prerendered content. This is by design and essential for performance, but it breaks traditional content negotiation patterns. ## The Solution After research and testing: **separate API endpoints combined with HTTP Link headers in HTML.** ### Architecture HTML pages include `` in the ``. Agents fetch `/api/markdown/posts/slug` directly and receive `text/markdown`. ### How It Works **For Humans:** 1. Visit `/posts/my-article` 2. Receive full HTML page 3. See `` in `` 4. Agent crawlers discover the link **For AI Agents:** 1. Fetch `/api/markdown/posts/my-article` directly 2. Receive `Content-Type: text/markdown; charset=utf-8` 3. Get raw markdown content 4. Parse and use This uses **HTTP Link headers**, a standard mechanism for content negotiation. ## Implementation ### Step 1: Create API Endpoint Create `src/pages/api/markdown/[...path].ts`: ```typescript import type { APIContext } from "astro"; import { getCollection, type CollectionEntry } from "astro:content"; import { getPath } from "@/utils/getPath"; export async function GET({ request }: APIContext) { const url = new URL(request.url); const pathname = url.pathname; let post: CollectionEntry<"blog" | "deepPosts"> | null = null; if ( pathname.startsWith("/api/markdown/posts/") || pathname.startsWith("/api/markdown/posts/drafts/") ) { const slug = pathname .replace(/^\/api\/markdown\/posts\//, "") .replace(/^drafts\//, ""); const allPosts = await getCollection("blog"); post = allPosts.find(p => getPath(p.id, p.filePath, false) === slug) || null; } else if ( pathname.startsWith("/api/markdown/deep-posts/") || pathname.startsWith("/api/markdown/deep-posts/drafts/") ) { const slug = pathname .replace(/^\/api\/markdown\/deep-posts\//, "") .replace(/^drafts\//, ""); const allPosts = await getCollection("deepPosts"); post = allPosts.find(p => getPath(p.id, p.filePath, false) === slug) || null; } if (!post) { return new Response("Post not found", { status: 404, headers: { "Content-Type": "text/plain" }, }); } const body = post.body; return new Response(body, { status: 200, headers: { "Content-Type": "text/markdown; charset=utf-8", "Cache-Control": "public, max-age=3600", }, }); } ``` Key points: - Handles blog and deep posts collections - Uses Astro's `getCollection` for content access - Returns raw markdown with correct content type - Includes cache headers for performance ### Step 2: Add Link Header to Layout Modify `src/layouts/Layout.astro`: ```typescript export interface Props { // ... existing props markdownURL?: string; } // In the section: {markdownURL && } ``` Pages using Layout include the alternate link when `markdownURL` is provided. ### Step 3: Pass Markdown URL in PostDetails Update `src/layouts/PostDetails.astro`: ```typescript const layoutProps = { // ... existing props markdownURL: `/api/markdown/${collection === "deepPosts" ? "deep-posts" : "posts"}/${getPath(post.id, post.filePath, false)}`, }; ``` Blog posts point to `/api/markdown/posts/slug`. Deep posts point to `/api/markdown/deep-posts/slug`. ### Step 4: Handle Edge Cases Default `markdownURL` to `undefined` in Layout: ```typescript const { title = SITE.title, // ... other props markdownURL = undefined, // Default for pages without it } = Astro.props; ``` Conditional rendering: ```astro { markdownURL && ( ) } ``` Prevents build errors on pages without `markdownURL`. ## Why This Works ### 1. Respects Cloudflare Pages Architecture Keep `prerender = true` for blog posts: - Static HTML served from edge cache (fast, cheap) - No server computation per request - Global CDN benefits API runs only when requested. No performance sacrifice. ### 2. Uses HTTP Standards Link headers are standard (RFC 5988). Agents discover markdown automatically. No special configuration needed. ### 3. Simple and Maintainable No middleware logic, no header manipulation, no edge cases: - API returns markdown - HTML includes link to API - Both work reliably ### 4. Tested in Production Deployed and verified: ```bash # Test API endpoint curl -I https://slavakurilyak.com/api/markdown/posts/agent-stories # Returns: Content-Type: text/markdown; charset=utf-8 # Test HTML with alternate link curl -s https://slavakurilyak.com/posts/agent-stories | grep alternate # Returns: ``` ## Best Practices ### Provide Clean, Structured Content Agents work best with: - Well-structured markdown (headings, lists, code blocks) - Clear frontmatter (title, description, tags, dates) - Semantic HTML (even when consuming markdown) ### Document Your Content Structure Explain: - Collections available - URL structure - API endpoint locations - Example requests and responses ### Consider Caching Strategy API includes `Cache-Control: public, max-age=3600`. Cloudflare caches for an hour: - Reduces server load - Faster repeated requests ### Monitor Agent Traffic Watch analytics for: - `Accept: text/markdown` header requests - Direct API endpoint access - User-agent patterns ## What Does Not Work ### Vercel-Style Middleware Redirect Why not redirect `Accept: text/markdown` to the API endpoint? Does not work on Cloudflare Pages. Static files bypass middleware. HTML serves from cache before middleware runs. ### Response Type Modification in Middleware Check headers after `context.next()`, modify response. Fails for the same reason: middleware never runs for static files. ### Dynamic Prerendering Check headers at request time, choose output format. Every request hits a server function. Eliminates static caching benefits. ## The Trade-Off Every architecture has trade-offs: | Approach | Performance | Complexity | Standards-Compliant | Works on CF Pages | | ------------------------------ | ----------- | ---------- | ------------------- | ----------------- | | Middleware content negotiation | Degrades | High | Yes | No (Static files) | | API + Link headers | Maintains | Low | Yes | Yes | For Cloudflare Pages, API + Link headers preserves performance while using standard HTTP mechanisms. ## Implement It Yourself If using Astro + Cloudflare Pages: 1. Create the API endpoint 2. Add `markdownURL` prop to Layout interface 3. Include conditional Link tag in `` 4. Set default `markdownURL` to `undefined` 5. Deploy and test Code is production-tested and ready to copy. --- **Verify it works:** ```bash # Test API endpoint curl -I https://slavakurilyak.com/api/markdown/posts/agent-stories # Test HTML with alternate link curl -s https://slavakurilyak.com/posts/agent-stories | grep "alternate.*markdown" ``` Agents discover the markdown version through the Link header automatically.