Harness adapters: Safe RL for efficient agentic development

Small LoRA adapters, trained with reinforcement learning inside pi, opencode and hermes, all served from one model, with every tool call the model makes inspected by safeagents before it runs. Follow the training runs on the live dashboard.
The same model behaves differently in different agent harnesses, where each one has its own tools, prompt and control flow. The ultimate guide to multi-harness RL showed how to train a model inside the harnesses themselves, without modifying them.
Here, we take a different approach, we train a small LoRA adapter per harness, on top of a base model. People can then swap the model behind their local agent for the adapter trained in that agent.
We first released initial harness adapters for LFM2.5-24B-A2B only, trained under pi, hermes and claude-code. They gave positive signals, both in tool-calling efficiency and in accuracy on held-out tasks, and given their adoption by the community, we decided to push further in this direction and add new components to the RL training, to make our initial work more complete.
LoRA adapters are cost-efficient for training and evaluation, since one server can host every harness's policy at once. At inference, anyone can load the adapter for the harness of their choice, to make the base model more efficient on their use case. We detail in One base model, many adapters how exactly we leverage LoRA adapters for training and evaluation.
We increased the diversity and difficulty of our previous RL tasks by generating synthetic RL environments, with open-source agent trajectories (a trajectory being the full logs of an agent session: the user requests, the model replies and every tool call with its output) as seed data. We derived four types of RL environments:
This animated demo shows what each type of task looks like:
How these environments are built, checked and scored is in Environments.
RL also lets the model hack the reward, taking shortcuts on the task it's trying to solve. So we plug safeagents into the RL training stage. It inspects the model's tool calls and blocks the ones that are clearly attempts at reward hacking. A blocked call is replaced by a text explaining why it was blocked, which invites the model to explore a different path. How the checks work, and a call they blocked, is in Saying 🙅 to agent shortcuts.
After feedback from the community, we extended the set of models we train to LFM2.5-24B-A2B and LFM3-1.7B, plus TwIL-8B as an experiment. The model card states it was not trained on agentic data, and we want to know whether such a model gives positive signals and can produce harness adapters too. The models and their settings are in Trained models.
We train our models with the GRPO algorithm, using TRL's AsyncGRPOTrainer together with tree attention for efficient training. Our harness adapters are LoRA adapters with rank 32, attached on every linear layer (all-linear). The models perform 8 rollouts per task, and we use a learning rate of 1e-5 for all our trainings.
Each run mixes SmolDataEnvs tasks, weighted toward medium and hard ones, with 230 of the 270 Repo2RLEnvs tasks. The other 40 Repo2RLEnvs tasks, and 150 SmolDataEnvs tasks, are held out for evaluation and never trained on.
Some shared parameters are the following: sampling temperature 0.8, up to 32,768 output tokens per turn, rollouts at most 4 weight updates stale, 4 gradient-accumulation steps, and a checkpoint every 10 steps, each one evaluated. TwIL gets 30 agent steps per session and 20 minutes timeout as it needs more steps to tasks (since not trained on agentic data).
We train on HF Jobs: 2×H200 for LFM3-1.7B, and 4×H200 for the other runs. The total cost of each training run is shown on the live dashboard of the training jobs.
Adapters pay off in two places, both shown below: switch between the training and evaluation modes. The sections below break each one down.
Each model is served by one vLLM. Its three trainers, one per harness, all send rollouts to it, each request naming its own adapter, so the base weights are loaded once for all three. After a gradient step, the trainer loads its new adapter into vLLM by name, next to the current one. Requests already running on the old version finish on it, so nothing pauses.
For LFM3-1.7B, the three trainers even share one GPU: an agent rollout spends most of its time in the sandbox, not generating, so the trainers mostly wait. In earlier runs a trainer spent about 35 s per step in forward and backward passes, against several minutes waiting for rollouts. Three of them take turns instead of idling on their own.
Training is asynchronous, where the rollouts and gradient steps don't wait for each other. Each trainer keeps up to 48 agent sessions running at once, about 6 tasks with 8 rollouts each (12 sessions for TwIL-8B, whose long reasoning fills vLLM's memory sooner). A task joins the training queue once all 8 of its rollouts are scored, since GRPO compares them with each other.
A gradient step packs finished rollouts into 4 micro-batches, each up to a token budget: 131,072 tokens for LFM2.5-24B-A2B, 65,536 for LFM3-1.7B and TwIL-8B. So the number of tasks per step isn't fixed: short sessions fit many, long ones few. Our earlier LFM2.5-24B-A2B runs trained on about 8 tasks (64 rollouts) per step, TwIL-8B's smoke run on about one. After the step, the new adapter version is loaded into vLLM, and rollouts started more than 4 versions earlier are dropped, so the model never learns from a policy too far behind it.
The dashboard logs both numbers for every run: batch/groups_per_step (tasks) and batch/samples_per_step (rollouts). They also set how often a task comes back: at 8 tasks per step, LFM2.5-24B-A2B's 200 steps go over its 530 tasks about 3 times.
The same trick evaluates every checkpoint at once. One vLLM loads the base weights, then each checkpoint as an adapter, with one sandbox server per checkpoint running its held-out tasks in parallel. Nine checkpoints of LFM2.5-24B-A2B, 1,350 agent