EleutherAI

We build open-source AI infrastructure, pioneer interpretability and open-weight safety, and help researchers without frontier-lab resources do frontier work.
Open research on large models requires open training data, open training and evaluation code, and models whose full development is public. We have built all three since 2020.
Our datasets include the Pile and the Common Pile. Our training framework is GPT-NeoX and our evaluation framework is the LM Evaluation Harness. Model suites such as Pythia are released with their training data, code, and intermediate checkpoints. All of it is openly licensed and in use across the research community.
Most interpretability research studies a finished model. We study how model internals develop over the course of training, and we have argued that a science of AI must study training dynamics rather than only the final artifact.
The Pythia suite was built for this purpose. It provides checkpoints throughout training together with the exact data each checkpoint had seen, and it is now widely used in research on learning dynamics, memorization, and how capabilities form.
Once model weights are public, safeguards cannot rely on API monitoring or on post-training methods that fine-tuning can remove. We work on safety methods that remain effective when anyone can inspect and modify the model, which in practice means intervening in the training data and the training process.
Deep Ignorance is one such method. It filters hazardous material out of pretraining data so that the model never learns it, and the resulting safeguard is substantially more resistant to adversarial fine-tuning than post-training alternatives. With FAR.AI we co-organize the Open-Weight Pretraining Safety Accelerator, which gives researchers access to the pretraining runs needed to test interventions of this kind at scale.
We maintain the Language Model Evaluation Harness, which is the framework behind the Open LLM Leaderboard and the standard tool for evaluating open models. Maintaining it has shown us how much reported numbers depend on details of prompting, scoring, and implementation.
We also study evaluation itself. This includes when benchmarks saturate and stop distinguishing between models, how the age of a benchmark affects comparisons across models, and how results should be reported so that they can be compared across groups. Through the EvalEval Coalition, which we co-founded with Hugging Face and the University of Edinburgh, we work on shared evaluation infrastructure with other institutions.
EMNLP System Demonstrations · October 24, 2026
ICML · July 10, 2026
ICLR · April 23, 2026
NeurIPS Datasets and Benchmarks · December 2, 2025