Should a Markdown-first content site turn on Block AI Bots?

For most personal blogs and small Markdown-driven publications, yes - Block AI Bots is a sensible default. The 2025 launch data showed that sites which turned it on lost virtually no meaningful human traffic but shed a large share of crawler load, and the 2026 retrospective confirms that publishers who opted in to the marketplace saw real revenue per crawl. The exception is sites whose business model depends on being cited inside AI answers (developer docs, public APIs, open datasets) - for those sites, staying crawlable is the whole strategy, and Pay-Per-Crawl lets you opt in with a price instead of an opt-out. The toggle is one-click in the Cloudflare dashboard; you can flip it back if the economics change.

How do robots.txt rules compare to Cloudflare's network-level Block AI Bots?

robots.txt is voluntary and easy to ignore - GPTBot, ClaudeBot, and the dozens of less-scrupulous AI crawlers in the wild frequently ignore User-agent blocks, ignore Crawl-delay, and rotate user-agent strings to bypass the rules. Cloudflare's network-level enforcement sits in front of every HTTP request to your origin, sees the actual TCP fingerprint, and can block or meter it regardless of what the crawler claims in its User-agent header. The Cloudflare 2026 report frames robots.txt as a 'voluntary standard' that 'lacked enforcement' while their network-level controls became the default enforcement mechanism. For a Markdown publisher, the practical difference is that robots.txt is documentation; Block AI Bots is execution.

Does blocking AI bots hurt my SEO on Google or Bing?

No. Block AI Bots targets AI training crawlers (GPTBot, ClaudeBot, CCBot, common-crawl, Bytespider, and similar) plus the explicit scrapers named in Cloudflare's verified AI Agents/Bots list. It does not block Googlebot, Bingbot, DuckDuckBot, or any of the legitimate search engine crawlers that drive your human traffic. The 2025 launch confirmed that opted-in sites kept their Google rankings; the 2026 retrospective confirms the same after a full year. If you want to be cited in Google's AI Overviews or in Perplexity's answer engine, that is governed by separate rules (Google's AI Overviews follow the standard Googlebot crawl; PerplexityBot has its own robots token) and is independent of the AI training crawler block.

Where does a client-side Markdown converter like md2rich fit in this AI bot landscape?

md2rich is on the opposite end of the spectrum from the AI crawler question - it is the part of your publishing workflow that intentionally keeps content out of any third-party server, including any AI infrastructure. When you draft in Markdown and convert to rich text in md2rich.com, the draft never leaves your browser. There is no upload, no cloud conversion step, no API call to an LLM for reformatting, no log that captures what you pasted. The 2026 Content Independence Day thesis is that creators should control who sees their content and on what terms; md2rich enforces that at the rendering step, before the content reaches the platform. Pair it with Block AI Bots on your own site and the full chain stays under your control: draft in Markdown locally, render to rich text locally, paste into the platform you trust, and the AI bots never see anything until you publish.

Cloudflare Content Independence Day 2026: AI Bot Rules

Q: What did Cloudflare actually launch on Content Independence Day in 2025, and what changed in 2026?

On July 1, 2025, Cloudflare launched Content Independence Day with two pieces: a one-click 'Block AI Bots' option that defaulted new Cloudflare domains to blocking AI training crawlers unless owners opted in, and a Pay-Per-Crawl marketplace that let publishers charge AI companies per crawl request. The 2026 one-year retrospective (published July 1, 2026) reports that 52% of all crawler requests now target AI training, up from 22% in Spring 2025, and that 50% of all internet traffic is non-human. The 2026 additions are the Monetization Gateway (charge for any resource behind Cloudflare via the x402 protocol), Pay-Per-Use evolution of Pay-Per-Crawl, and Attribution Business Insights that surface which AI services consume your content. Block AI Bots is now the default for new domains and available as a one-click toggle for existing domains.

On July 1, 2026, Cloudflare published its one-year retrospective on Content Independence Day. The headline numbers: 52% of all crawler requests on the Cloudflare network now target AI training, up from 22% in Spring 2025, and roughly 50% of all internet traffic is non-human. The retrospective also announced the next stage - a Monetization Gateway built on the x402 protocol, a Pay-Per-Use evolution of Pay-Per-Crawl, and Attribution Business Insights that show which AI services are consuming your content. For a creator who drafts in Markdown and publishes through client-side tools, this is the inflection point where AI crawler policy stops being a search-engine problem and becomes a publishing-stack design problem.

What Changed Between 2025 and 2026

Content Independence Day launched on July 1, 2025 with two pieces: a one-click "Block AI Bots" toggle that defaulted new Cloudflare domains to blocking AI training crawlers unless owners opted in, and a Pay-Per-Crawl marketplace where publishers could name a price per crawl request and AI companies could pay it (or walk away). The framing at launch was that robots.txt - the voluntary, decades-old standard for telling crawlers what they may not fetch - had failed because the largest AI training crawlers simply ignored it. Cloudflare positioned itself as the network-level enforcement layer that robots.txt had never had.

A year later, the retrospective confirms the strategy worked at the operational level. Crawlers that respected robots.txt before still respect it; crawlers that ignored it before are now blocked at the edge by default on every new Cloudflare domain. The crawler mix has shifted: AI training crawlers now drive 52% of crawler traffic (up from 22%), search crawlers are a declining share, and mixed-use crawlers that both index and train sit at 36%. The retrospective also notes that 2.5 billion people - roughly 30% of humanity - adopted generative AI in 3.5 years, which Cloudflare frames as a platform shift happening at more than 2x the speed of smartphone adoption.

Three new pieces arrived with the 2026 update. The Monetization Gateway lets any Cloudflare customer charge for any resource behind their domain using the x402 HTTP-based payment protocol - effectively a programmable paywall at the edge. Pay-Per-Use evolves Pay-Per-Crawl from per-request pricing to per-actual-use pricing (the AI company pays only when the fetched content is surfaced inside an answer, not when it is fetched speculatively). Attribution Business Insights surfaces which AI services consumed which pages, so publishers can see whether their content ended up in ChatGPT, Claude, Perplexity, or a long tail of smaller models.

What Block AI Bots Actually Blocks

The Cloudflare-verified AI Bots and Agents list is the source of truth. As of July 2026 it covers GPTBot (OpenAI training crawlers), ClaudeBot and anthropic-ai (Anthropic training crawlers), CCBot (Common Crawl, which feeds many training pipelines), Bytespider (ByteDance), Meta-ExternalAgent and meta-webindexer (Meta), Amazonbot (Amazon, when used for training), and a long tail of named scrapers including Diffbot, ImagesiftBot, Omgilibot, PetalBot, and TikTokSpider. The list is updated when a new AI crawler announces itself or when Cloudflare's traffic analysis identifies one.

The block does NOT cover Googlebot, Bingbot, DuckDuckBot, Applebot, or any of the search engine crawlers that drive human referral traffic. It also does not cover crawlers used purely for search indexing - the AI Overviews pipeline at Google, for example, reuses the standard Googlebot crawl, so blocking AI bots does not exclude you from AI Overviews. PerplexityBot and ChatGPT-User (the fetchers used when a user actively asks an AI to look at a page) are governed separately and are not on the Block AI Bots list by default.

For a Markdown-first blog or documentation site, the practical implication is that Block AI Bots cuts the bulk of speculative training crawlers without affecting the crawlers that bring human readers. The Cloudflare 2025 data showed that opted-in sites lost essentially no Google or Bing traffic; the 2026 retrospective confirms the same finding a year later.

Why robots.txt Is Not Enough

The Cloudflare report is unusually blunt about robots.txt: voluntary standards lack enforcement, and the AI training crawlers in particular have shown a pattern of ignoring User-agent blocks, ignoring Crawl-delay directives, and rotating user-agent strings when one identity gets rate-limited. The 2025 launch cited specific crawlers that went from respecting robots.txt to ignoring it as the AI training market heated up; the 2026 update frames network-level controls as the new default, with robots.txt relegated to documentation rather than enforcement.

For a Markdown publisher, the practical distinction is between what your robots.txt says and what the network actually allows through. A site with a strict robots.txt can still be hammered by GPTBot if GPTBot ignores the rules; a site with Cloudflare's Block AI Bots enabled will not see a single GPTBot request regardless of what its robots.txt says. The two layers are complementary - robots.txt is still the polite signal you send to crawlers that do respect it - but the enforcement now lives at the edge.

The Monetization Gateway and What It Means for Tool Builders

The new Monetization Gateway is the most interesting piece for anyone running a content site or a content tool. It exposes the x402 HTTP payment protocol as a first-class Cloudflare feature: any URL behind Cloudflare can return a 402 Payment Required response with a price, a payment address, and a currency, and the request is fulfilled only after the payment settles. The protocol is content-type-agnostic - it works for HTML pages, JSON API responses, Markdown files, even image bytes - which means a publisher can charge per page view, per API call, per image download, or per Markdown fetch.

The practical effect for a Markdown publishing tool is that the same x402 envelope can wrap a draft fetch, a published article fetch, or a tool-API response. An AI agent that wants to summarize your article pays per fetch; a developer who wants to call your converter API pays per call; a reader who wants to read the full article without a subscription pays per view. The Pay-Per-Use evolution of Pay-Per-Crawl sits on top of this - the AI company pays only when the fetched content is actually surfaced in an answer, not when it is fetched speculatively during retrieval. That distinction is important because most AI retrieval pipelines today fetch many more pages than they actually use.

Where md2rich Sits in This Picture

md2rich is intentionally on the opposite end of the spectrum from the AI crawler question. It is the part of a publishing workflow where the content never leaves the creator's browser. Drafts in Markdown are converted to rich text on the client side, in your tab, with no upload, no cloud conversion step, no API call to any LLM for reformatting, no server log that captures what you pasted. The same architecture that made it the obvious answer to the AI writing assistant question (covered here on July 5) makes it the obvious complement to Block AI Bots: the tool that renders your draft is also the tool that never sees the draft.

Pair the two and a creator gets a full chain they can reason about. Draft in Markdown in your editor of choice (Obsidian, iA Writer, VS Code, Zettlr, or the md2rich editor itself). Render to rich text in md2rich.com - the draft never leaves your browser. Paste into the platform you trust (X Articles, LinkedIn, Medium, Substack, Notion, Bluesky long-form). On the destination platform, your published content is governed by the platform's AI bot policy, which most major platforms have not yet set - so you may also want to enable Block AI Bots on your own domain if you syndicate anywhere. The Markdown source stays under your control from first draft to final paste; the AI crawler question is something you decide for each surface where your content lands.

The Content Independence Day thesis is that creators should control who sees their content and on what terms. md2rich has enforced that at the rendering step since launch - the conversion happens locally, the preview is exactly what the platform receives, and there is no third-party server in the loop. The 2026 Cloudflare retrospective confirms the thesis at the network layer. Both pieces point the same direction: the next era of publishing is built on local-first tools and explicit per-request economics, not on cloud-mediated defaults that assume permission.

What to Do This Week

Three concrete actions for a Markdown publisher who wants to align with the 2026 Content Independence Day direction.

Turn on Block AI Bots if your site is on Cloudflare. The toggle is one click in the Cloudflare dashboard under Security > Bots. New domains have it on by default since the 2025 launch; existing domains can opt in without losing search rankings (Cloudflare confirmed this for both 2025 and 2026). The exception is sites whose strategy depends on being cited inside AI answers - developer docs, public APIs, open datasets - where staying crawlable is the whole point and Pay-Per-Crawl is a better fit.

Update your robots.txt to match. Even with network-level enforcement, keep your robots.txt in sync. List the AI crawlers you have blocked at the edge so legitimate crawlers that respect robots.txt see the policy too. Include a sitemap directive pointing at https://yourdomain.com/sitemap.xml so crawlers can find the pages you do want indexed. Cross-reference with the Cloudflare verified AI Bots list rather than guessing.

Keep the rendering step local. Drafts that pass through a cloud converter before being pasted into a platform are visible to that converter, even if briefly. md2rich runs the conversion in your browser, so the unpublished draft never leaves your machine. That makes it a natural fit for any workflow that takes the Content Independence Day thesis seriously - the part of the pipeline where the content is most sensitive is also the part where you have the most control.

Try the Workflow

Cloudflare's Content Independence Day retrospective went live on July 1, 2026 and is the canonical reference for the 2025 launch and the 2026 update. The Block AI Bots toggle, the Pay-Per-Crawl marketplace, the Monetization Gateway, and the Attribution Business Insights dashboard are all available to any Cloudflare customer regardless of plan tier. The rendering step that keeps your draft out of every server until you choose where it lands has been live at md2rich.com since launch - 100% client-side, no upload, no account, no third-party server. Draft once in Markdown, render once in your browser, paste into the platforms you trust, and let the AI bot policy follow from there.