IETF | Enabling Publishers to Express Preferences for AI Crawlers: An Update on the AIPREF Working Group

The IETF works on a broad range of networking technologies that provide the foundation for the Internet's growth and evolution.
While most IETF work takes place online, meetings and other events such as Hackathons provide in-person opportunities to advance work within the IETF.
The IETF is the premier Internet standards organization. It follows open and well-documented processes for setting these standards. Once published, those standards are made freely available.
The global IETF community works together in many different ways to produce high quality, relevant technical documents that influence the way people design, use, and manage the Internet.
By supporting the IETF today, you are ensuring a free and open Internet exists tomorrow.
In October 2026, the Internet Architecture Board (IAB) held a two-day workshop in Prague with people who are deploying post-quantum authentication. These are the first impressions from the co-chairs, ahead of the full workshop report.
Email is the oldest widely used application on the Internet, but it continues to evolve. At IETF 127 four working groups will meet to improve the security and usability of mail, and two Birds-of-a-Feather sessions propose improvements in mail operations and the way users sign up for mail subscriptions.
The IETF Administration LLC has prepared its draft budget for 2027 and now seeks community feedback.
These IETF 127 meeting sessions are likely to include discussions and proposals that are accessible to a broad range of Internet technologists whether they are new to the IETF or long-time participants.
Several new features implemented on RFC-Editor.org aim to make the site more useful and engaging for RFC consumers.
In October 2026, the Internet Architecture Board (IAB) held a two-day workshop in Prague with people who are deploying post-quantum authentication. These are the first impressions from the co-chairs, ahead of the full workshop report.
Email is the oldest widely used application on the Internet, but it continues to evolve. At IETF 127 four working groups will meet to improve the security and usability of mail, and two Birds-of-a-Feather sessions propose improvements in mail operations and the way users sign up for mail subscriptions.
The IETF Administration LLC has prepared its draft budget for 2027 and now seeks community feedback.
These IETF 127 meeting sessions are likely to include discussions and proposals that are accessible to a broad range of Internet technologists whether they are new to the IETF or long-time participants.
Several new features implemented on RFC-Editor.org aim to make the site more useful and engaging for RFC consumers.
In January, the IETF chartered the AI Preferences (AIPREF) Working Group to make it easier to express how AI models should use Internet content.
By June, we reported that the group was near completion of its chartered goal to deliver specifications by September this year.
Accordingly, we made a Working Group Last Call on the documents in September. The response was not one we anticipated: several participants raised fundamental issues about the nature of the Internet-Drafts (I-Ds), forcing the group to reconsider its approach.
Although a variety of issues were raised, a particularly relevant observation was that AI is at its heart a technique, not a purpose. So, stating preferences about whether AI can be used with a given online asset is similar to stating a preference as to whether that content can be used with (for example) the Python programming language, or a linked list.
This observation leads to the realisation that it isn’t terribly useful to let publishers state a preference for what tools a programmer might use—especially as AI becomes pervasive throughout computing, for purposes such as summarisation, accessibility, and translation. This is especially true for online search engines, which use AI pervasively throughout many parts of their "pipeline" for data, regardless of how that data is presented to end users.
However, it is still very useful and relevant to give publishers input into what those tools are used for—the purpose of the processing. That purpose might be the training of an AI model, or various presentations to users in response to a search query. Giving publishers ways to express preferences about how their assets are used in these activities allows them to achieve their goals without unnecessarily linking those preference statements to a particular technology. That helps to avoid unintentional limits on innovation—developers will be able to select the best tools for the job while still respecting publisher preferences.
The result (after extensive discussion and collaboration at our Zürich interim meeting) is an updated vocabulary that defines preferences for:
Together, these preferences are intended to give publishers the ability to state how they’d like their assets to be used. In particular, they should allow a publisher to state that they don’t want their assets used to train foundation models (which in itself is a distinct purpose), but still want to show up in search experiences. And, if they don’t want their assets to be summarised or otherwise transformed, they can use the "search" preference to indicate this.
See the draft vocabulary for more details. Note also that there are several other open issues on the draft, and this is only a proposal, albeit one informed by discussion to date – it does not have consensus, doesn’t address some known problems, and hasn’t yet been widely reviewed.
This approach is currently a proposal to the Working Group; the editors have proposed it in a draft for consideration, but it does not yet have consensus. Accordingly, we’re looking for input regarding it – in particular, from publishers, AI vendors, and others who might be affected. If you have feedback, please give it on the mailing list or at our upcoming meeting during IETF 124 Montreal.
In October 2026, the Internet Architecture Board (IAB) held a two-day