Last week, content delivery network Cloudflare announced a partnership with GoDaddy for Cloudflare’s AI Crawl Control product which gives publishers the ability to “gate” access for AI agents.
Cloudflare chief strategy officer Stephanie Cohen said at the time, “By putting tools like AI Crawl Control and open standards into the hands of website owners, we are providing essential underpinnings for a new Internet business model.”
GoDaddy said it will support Agent Name Service and Web Bot Auth for the “verifiable identity layer needed for a functional digital marketplace.
Read: “Cloudflare and GoDaddy Partner to Help Enable an Open Agentic Web” (April 7) – press release
James Smith, Senior Director of Product Management, Cloudflare, answered a selection of follow-up questions from tipsheet via email.
tipsheet: What still needs to fall into place before publishers can meaningfully monetize AI agent access to their content?
James Smith: Publishers are already meaningfully monetizing their content. The industry has seen a significant increase in deals done over the last year.
At Cloudflare, we are helping to enable the necessary components for success, first by giving publishers the ability to protect their content and drive meaningful scarcity. The LLMs are not going to pay for content they are getting for free, so offering publishers protection is step one. People Inc is an example of this.
Next, we are giving publishers full transparency and understanding into which bots and models are trying to access their content. In doing direct negotiations with any of the LLMs, it’s incredibly helpful to know how much OpenAI, for example, is trying to access their content, how much each of their competitors are trying to get access, what their scrape to referral metrics are and what the publishers most trafficked URLs are. All these data points help publishers start to understand the value of their content and combat some of the asymmetry at the negotiating table. We’ve actually created a dashboard specifically designed for business and partnership teams as publishers to help do smarter, data based direct deals.
For the long tail of both publishers and models, we are starting to see more of a marketplace emerge. As we move to a world where new and more specialized models emerge, and large enterprise industries like law and fintech begin to build their models, it’s becoming increasingly important that content for training is rights cleared. Standardizing deals at scale via protocols and machine readable terms is directionally where we’re headed.
How should publishers think about the timeline for moving from controlling bots to actually monetizing them?
We think of this as a three step process with transparency and control as the necessary steps to monetization. And these will continue to be important signals in the agentic era – optimizing your content for maximum discovery, understanding which content is the most valuable to the network of LLMs is a data based approach for publishers to move forward.
How do you see emerging agentic advertising protocols, like those from IAB Tech Lab and AgenticAdvertising.org, fitting into Cloudflare’s approach to managing AI agent traffic?
Emerging agentic advertising protocols, such as those from the IAB Tech Lab and AgenticAdvertising.org, fit perfectly into Cloudflare’s vision. We need to support a broad range of protocols for this maturing ecosystem. Because Cloudflare powers such a significant portion of the Internet, our role is to serve as both the enablement and verification layer for these standards. We provide the platform that authenticates an agent’s identity, ensures it respects a publisher’s specific preferences in real-time, and mitigates the technical friction of these handshakes. By providing the pipes that allow these micro-negotiations to happen at scale, we help move the industry away from simple blocking and toward a sophisticated, revenue-generating value exchange.
Does Cloudflare see an opportunity for creating its own marketplace where AI companies can buy publisher content on a pay-per-crawl or pay-per-inference (or pay-per-whatever) basis? Currently, smaller publishers and many large publishers do not have access to the relationships and deals that certain publishers (such as People Inc. which you identified) do with AI companies.
Cloudflare announced pay per crawl last year (in beta), which gives publishers and creators the ability to monetize their content per crawl. You can find more information in this blog.