---
title: Changelog · Cloudflare AI Crawl Control docs
description: AI Crawl Control now includes a Robots.txt tab that provides
  insights into how AI crawlers interact with your robots.txt files.
lastUpdated: 2025-08-27T16:00:59.000Z
chatbotDeprioritize: false
source_url:
  html: https://developers.cloudflare.com/ai-crawl-control/changelog/
  md: https://developers.cloudflare.com/ai-crawl-control/changelog/index.md
---

[Subscribe to RSS](https://developers.cloudflare.com/changelog/rss/ai-crawl-control.xml)

## 2025-10-21

**New Robots.txt tab for tracking crawler compliance**

AI Crawl Control now includes a **Robots.txt** tab that provides insights into how AI crawlers interact with your `robots.txt` files.

#### What's new

The Robots.txt tab allows you to:

* Monitor the health status of `robots.txt` files across all your hostnames, including HTTP status codes, and identify hostnames that need a `robots.txt` file.
* Track the total number of requests to each `robots.txt` file, with breakdowns of successful versus unsuccessful requests.
* Check whether your `robots.txt` files contain [Content Signals](https://contentsignals.org/) directives for AI training, search, and AI input.
* Identify crawlers that request paths explicitly disallowed by your `robots.txt` directives, including the crawler name, operator, violated path, specific directive, and violation count.
* Filter `robots.txt` request data by crawler, operator, category, and custom time ranges.

#### Take action

When you identify non-compliant crawlers, you can:

* Block the crawler in the [Crawlers tab](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/)
* Create custom [WAF rules](https://developers.cloudflare.com/waf/) for path-specific security
* Use [Redirect Rules](https://developers.cloudflare.com/rules/url-forwarding/) to guide crawlers to appropriate areas of your site

To get started, go to **AI Crawl Control** > **Robots.txt** in the Cloudflare dashboard. Learn more in the [Track robots.txt documentation](https://developers.cloudflare.com/ai-crawl-control/features/track-robots-txt/).

## 2025-10-14

**Enhanced AI Crawl Control metrics with new drilldowns and filters**

AI Crawl Control now provides enhanced metrics and CSV data exports to help you better understand AI crawler activity across your sites.

#### What's new

#### Track crawler requests over time

Visualize crawler activity patterns over time, and group data by different dimensions:

* **By Crawler** — Track activity from individual AI crawlers (GPTBot, ClaudeBot, Bytespider)
* **By Category** — Analyze crawler purpose or type
* **By Operator** — Discover which companies (OpenAI, Anthropic, ByteDance) are crawling your site
* **By Host** — Break down activity across multiple subdomains
* **By Status Code** — Monitor HTTP response codes to crawlers (200s, 300s, 400s, 500s)

![AI Crawl Control requests over time chart with grouping tabs](https://developers.cloudflare.com/_astro/ai-crawl-control-requests-over-time.BtRyz0OT_Z17gWIC.webp)

#### Analyze referrer data (Paid plans)

Identify traffic sources with referrer analytics:

* View top referrers driving traffic to your site
* Understand discovery patterns and content popularity from AI operators

![AI Crawl Control top referrers breakdown](https://developers.cloudflare.com/_astro/ai-crawl-control-top-referrers.CEUAwpd8_Ssmak.webp)

#### Export data

Download your filtered view as a CSV:

* Includes all applied filters and groupings
* Useful for custom reporting and deeper analysis

#### Get started

1. Log in to the Cloudflare dashboard, and select your account and domain.
2. Go to **AI Crawl Control** > **Metrics**.
3. Use the grouping tabs to explore different views of your data.
4. Apply filters to focus on specific crawlers, time ranges, or response codes.
5. Select **Download CSV** to export your filtered data for further analysis.

Learn more about [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control).

## 2025-08-27

**Enhanced crawler insights and custom 402 responses**

We improved AI crawler management with detailed analytics and introduced custom HTTP 402 responses for blocked crawlers. AI Audit has been renamed to AI Crawl Control and is now generally available.

**Enhanced Crawlers tab:**

* View total allowed and blocked requests for each AI crawler
* Trend charts show crawler activity over your selected time range per crawler

![Updated AI Crawl Control table showing request counts and trend charts](https://developers.cloudflare.com/_astro/ai-crawl-control-table.BDr0Qd-5_Z28yWEm.webp)

**Custom block responses (paid plans):** You can now return HTTP 402 "Payment Required" responses when blocking AI crawlers, enabling direct communication with crawler operators about licensing terms.

For users on paid plans, when blocking AI crawlers you can configure:

* **Response code:** Choose between 403 Forbidden or 402 Payment Required
* **Response body:** Add a custom message with your licensing contact information

![AI Crawl Control block response configuration interface](https://developers.cloudflare.com/_astro/ai-crawl-control-block-response.L4duQj7-_ZjWQYT.webp)

Example 402 response:

```http
HTTP 402 Payment Required
Date: Mon, 24 Aug 2025 12:56:49 GMT
Content-type: application/json
Server: cloudflare
Cf-Ray: 967e8da599d0c3fa-EWR
Cf-Team: 2902f6db750000c3fa1e2ef400000001


{
  "message": "Please contact the site owner for access."
}
```

## 2025-07-01

**Introducing Pay Per Crawl (private beta)**

We are introducing a new feature of [AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/) — Pay Per Crawl. [Pay Per Crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/) enables site owners to require payment from AI crawlers every time the crawlers access their content, thereby fostering a fairer Internet by enabling site owners to control and monetize how their content gets used by AI.

![Pay per crawl](https://developers.cloudflare.com/_astro/pay-per-crawl.B5bv2nwT_Z1qUoVQ.webp)

**For Site Owners:**

* Set pricing and select which crawlers to charge for content access
* Manage payments via Stripe
* Monitor analytics on successful content deliveries

**For AI Crawler Owners:**

* Use HTTP headers to request and accept pricing
* Receive clear confirmations on charges for accessed content

Learn more in the [Pay Per Crawl documentation](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/).

## 2025-07-01

**AI Crawl Control refresh**

We redesigned the AI Crawl Control dashboard to provide more intuitive and granular control over AI crawlers.

* From the new **AI Crawlers** tab: block specific AI crawlers.
* From the new **Metrics** tab: view AI Crawl Control metrics.

![Block AI crawlers](https://developers.cloudflare.com/_astro/manage-ai-crawlers.6UgS8dSG_12BsLg.webp) ![Analyze AI crawler activity](https://developers.cloudflare.com/_astro/analyze-metrics.C52pJZVg_Z1IvdY4.webp)

To get started, explore:

* [Manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/).
* [Analyze AI traffic](https://developers.cloudflare.com/ai-crawl-control/features/analyze-ai-traffic/).

## 2024-09-23

**AI Crawl Control**

Every site on Cloudflare now has access to [**AI Audit**](https://developers.cloudflare.com/ai-crawl-control/), which summarizes the crawling behavior of popular and known AI services.

You can use this data to:

* Understand how and how often crawlers access your site (and which content is the most popular).
* Block specific AI bots accessing your site.
* Use Cloudflare to enforce your `robots.txt` policy via an automatic WAF rule.

![View AI bot activity with AI Audit](https://developers.cloudflare.com/_astro/ai-crawl-control-overview.Dr_yGKOC_2kcoTJ.webp)

To get started, explore [AI audit](https://developers.cloudflare.com/ai-crawl-control/).
