Reflections on AI Explain: A postmortem

HTML: Markup language
CSS: Styling language
JavaScript: Scripting language
Web APIs: Programming interfaces
Learn web development
Get to know MDN better
We recently launched two new AI experiences — AI Explain and AI Help. Thanks to feedback from our community, we realized that AI Explain needs more work. We have disabled it and will be working on it to make it a better experience.
In this blog post, we look into the story behind AI Explain: its development, launch, and the reasons that led us to press the pause button. We greatly value our community's input, which has played a significant part in this decision. Looking forward, we aim to resume our work on this feature in 2024, giving us ample time to address the feedback received and improve the feature in a way that helps everyone who wants to use it.
In preparation for the launch of MDN Plus in March 2022, we spent the majority of 2021 doing user research to try to understand the main pain points people have on MDN. A topic that came up a lot was the discoverability of content for people who are newer to software development. Experienced developers can be more productive by effectively searching for the information they need, but new developers often still need to build this skill.
A recurring complaint we heard from our readers was that MDN is excellent when you know what to look for. If you don't, it's not easy to discover the information you need and extract meaningful knowledge from the appropriate documentation. Developers with less experience find it harder to navigate through MDN and often turn to other places to find the information they need. Even more, we have a high dependency on search engines, with more than 90% of our traffic coming from them. We knew improving site search would help discoverability and saw an opportunity to combine search improvements with novel ways of interacting with the site.
We've noted that it's not always trivial to understand code examples on MDN, because they often combine features, not all of which are explained in the current article. However, fully explaining every detail—like defining what a CSS selector is on every page containing a CSS example—would be impractical. Through AI Explain we could enable readers to better understand embedded code examples in MDN documentation, without overwhelming the core content.
In January 2023, we began experimenting with OpenAI's GPT-3.5 API to determine how we could enhance our documentation platform and make our website more user-friendly. We saw potential for uses such as summarizing documentation pages, explaining existing code samples, generating examples for pages lacking them, and generating unique code examples that combined different technologies according to user queries.
We also saw that other developer-focused companies were investing significantly in and building products on top of the technology. With MDN's documentation being publicly available under an open Creative Commons license, and considered best-in-class, it's reasonable to assume most models have been trained on our content, and products started to be announced which explicitly allowed the consumption of our documentation. This led us to understand that, irrespective of our personal feelings, our users already access our content through generative AI. With the support of our community, we are uniquely positioned to learn how AI can augment documentation and how developers use it. Our extensive experience, a willingness to learn, and our network of contributors and subject matter experts place us in a great position to refine this functionality through iteration and feedback.
In March 2023, we focused development on two features that were the most promising given the current state of the technology:
AI Explain - A way for readers to explore and understand code examples embedded in MDN documentation pages, describing the purpose and behavior of the code or parts of the example.
AI Help - A conversational interface that offers concise answers using MDN articles related to the user's questions for contextual help and provides the MDN pages it used to answer the questions as sources. This prototype was built in collaboration with Supabase for the purpose of getting responses tailored to MDN content. We decided to keep this feature behind a login which would require readers to have an MDN Plus account.
We recognized an opportunity with these features to better assist less experienced developers with the intent to be helpful rather than matching our reference documentation's exact quality and precision. Beginners are new to navigating reference documentation and often turn to other, potentially less accurate, information sources. Learning isn't an immediate jump to a correct conclusion but an iterative process where we consider incorrect information and discard it as our understanding solidifies. Just as a human might (as an expert or a peer in a learning community) give an incorrect response, it is still ultimately useful as it unblocks, gives our users ideas, and points them to something relevant.
Let's zoom in on the AI Explain launch.
We first tested AI Explain with the OpenAI models gpt-3.5-turbo-0301, followed by gpt-3.5-turbo-0613, as this was the default model from June 27, 2023 onwards. Due to the reduced capacity for the new model, we reverted to gpt-3.5-turbo-0301. We ran all 25,820 unique code examples on MDN against gpt-3.5-turbo-0301 and used a combination of automated and manual testing to validate the explained responses.
As beginners are the most likely audience to use this feature, we took common programming concepts and examples that readers would run against AI Explain and corroborated responses individually. Contrasting this audience, we also chose edge cases and new web technologies that we thought the model might struggle to explain.
With regard to automated testing, a test suite is a challenge to build for a feature of this type due to the non-deterministic output