Method

We train open models to solve real-world tasks inside popular agent harnesses (pi, opencode, hermes). Each harness runs unmodified, and we train a small LoRA adapter on top of each model. Previous iterations of this protocol gave encouraging signals for LFM2.5-24B-A2B. The question these runs answer: does reinforcement learning on verifiable tasks, inside a given harness, make the model better at those tasks under that same harness?

Models and training

• LiquidAI/LFM2.5-24B-A2B, a mixture of experts with 2B active parameters, and LiquidAI/LFM3-1.7B. Plus one experiment, webAI-Official/TwIL-8B (see below).
• A LoRA adapter (rank 32, every linear layer), one per model and harness, trained with TRL's AsyncGRPOTrainer: 8 rollouts per task, learning rate 1e-5, weights synced to vLLM after every step.
• The reward is the task verifier's correctness, plus a small bonus for solving with fewer tool calls: 0.1 × (1 − calls / 15), only on correct rollouts. We picked the weight in an ablation. At 0.3 the bonus cut tool calls but cost accuracy (−5.3 points held-out). At 0.1 accuracy held (+2.7 ± 4.8 over the base) while tool calls fell 11 to 17%.
• Each model is served by a single vLLM on GPU 0. Its three trainers, one per harness, all send rollouts to it, each under its own adapter name, so the base weights are loaded once for all three.

Thinking

All three models reason before they act (a <think> block), and train with thinking on, under every harness. Each checkpoint is evaluated the same way, so a score measures the model as it was trained.

Reasoning from earlier turns stays in the context. That keeps each turn an exact extension of the tokens already generated, which training needs, at the cost of longer contexts: TwIL's long reasoning is why it's the slowest of the three per step.

Experiment: TwIL-8B

webAI-Official/TwIL-8B is a dense 8.8B reasoning model, post-trained from IBM's Granite 4.2 for math, code and instruction following. Its model card says it was never trained for tool use. It runs here as an experiment, next to the two LFM models, to ask one question: can harness RL turn a strong reasoning model into a strong agent?

• Why it's interesting: before any RL, it already solves more held-out tasks than LFM3-1.7B under all three harnesses. That comes from a first, partial eval (about 35 tasks per harness), so treat it as a first signal.
• Why it's an experiment: it starts close to the top on our easier tasks, so it trains on hard ones only. Its long reasoning makes it the costliest of the three per step. And its license, the webAI Non-Commercial License, covers our adapters too: they're for research only.
• One fix to serve it: its chat template adds a newline to past reasoning, so a past turn re-rendered as different tokens from the ones generated. A one-line change fixes that.

Three harnesses on one GPU (LFM3-1.7B)

For LFM3-1.7B, all three trainers (opencode, pi and hermes) run on the same GPU, next to the vLLM GPU: two GPUs for three runs, where LFM2.5-24B-A2B uses one trainer GPU per harness. Two facts make this work:

• A trainer is mostly waiting. An agent rollout spends most of its time in the sandbox, running tools and setting up the environment, and very little of it generating. In earlier runs the trainer spent about 35 s per step in forward and backward passes, against several minutes waiting for rollouts. Three trainers on one GPU take turns instead of each idling on its own.
• The model and its adapter are small. 1.7B parameters in bf16 plus a rank-32 adapter leave plenty of memory. Each trainer caps its packed rows at 65,536 tokens so that three backward passes at once still fit.

LFM3 also needed one fix to train this way. Training packs the rollouts of a group as a tree: their shared prompt is computed once, and each continuation branches off it. LFM3's width-7 short convolution read across branches there, so we route it along each sample's own path. On a small LFM3, a parity test confirms packed and per-sample losses and gradients now match exactly.

Harnesses

opencode, pi and hermes run in their own sandbox per rollout, through OpenEnv's Harbor integration. Their model requests are routed to the trainer's vLLM, and the token ids and logprobs of every response are captured. Training uses exactly the tokens the harness saw, with no re-tokenization.

Environments

• SmolDataEnvs: data-analysis questions from public Kaggle notebooks. The agent gets the data files in its sandbox and writes one answer, which is graded by exact or numeric match. 400 medium and 600 hard tasks.
• traj2envs tasks, built from real agent trajectories. A trajectory becomes a task when it contains a verifiable unit of work: repair (a check that failed, then passed after edits), build (a project finished from one request), extend (a later change whose checks pass) or qa (a fact computed from public data). Each task is validated by running its oracle in its own container image, and kept only if a reference agent solves it sometimes but not always (solve rate 0.1 to 0.7).

traj2envs is moving to Repo2RLEnv, where its development continues. The traj2envs repository will be archived.

The traj2envs tasks come from 1,950 environments built from open agent trajectories that people published on the Hugging Face Hub, from pi, Claude Code, Codex and other harnesses:

safeagents

RL finds shortcuts, so every tool call is checked before it runs, by safeagents (live on the safeagents tab). The rules are atif-scan's built-in checks plus a task pack written for these environments:

• high, blocked: fetching a task's published answer or grader (the public dataset, gold_answer, task.toml, grader.py), and reaching for a Hugging Face credential.
• medium, logged: looking up the task's data source or staging script.
• traj2envs tasks get their own pack. Each one is rebuilt from a public trajectory whose later steps are the solution, so fetching any of the source datasets or trajectory files is blocked, and any other Hub download is logged. An offline scan of 213 sessions of ordinary build work matched none of its rules.

A blocked call leaves its notice in the transcript and the rollout scores 0, so the policy learns not to make it. Sandboxes get no Hugging Face token at all: the input data is public.

In a 50-step ablation (pi, LFM2.5-24B-A2B), turning the guard on changed nothing. Held-out pass@1 at step 50 was +0.0 ± 4.1 points against the same run without it. It blocked no calls, and an offline scan of 750 eval trajectories found no high-severity behaviour.

Evaluation

A scheduled job picks up every new checkpoint and evaluates it under the harness it was trained in, on held-out tasks never trained on. Each checkpoint is compared task by task with the base model under the same harness.

Because we train LoRA adapters, one GPU evaluates every checkpoint at once. The diagram above, in its evaluation mode, shows how. A checkpoint is an adapter, about 75 MB for LFM2.5-24B-A2B at rank 32, not a copy of the model. A single vLLM loads the base weights once and serves every adapter side by side, each by its own name, with one sandbox server per checkpoint running its tasks in parallel. The usual approach, merging each checkpoint into full weights and starting one server per checkpoint, pays for the model load and a GPU for every checkpoint.

• The numbers: nine checkpoints of LFM2.5-24B-A2B, across three runs and two harnesses, 1,350 agent sessions in all, were evaluated in a single h200 job in under an hour, for about $5 of GPU time. One merged checkpoint per job takes about 1.5 hours and $10 each, so roughly $90 for the same nine.
• It is cheap because the GPU is otherwise idle. As in training, agent sessions are dominated by sandbox time, so a single vLLM keeps up with every checkpoint's sessions at once. More checkpoints cost more sandboxes, but no more GPUs.
• It measures exactly what was trained. Evaluation serves the adapter the same way training rollouts did, with nothing merged or converted, so a score belongs to the checkpoint as it was trained.

What earlier runs showed

• pi on SmolDataEnvs: +6.0 ± 5.6 points over the base at step 25 and +4.0 ± 5.5 at step 50 (held-out, 150 tasks).
• hermes dips first (−7.3 at step 25), then recovers (+1.3 at step 50).
• On tasks the base model already solves about 80% of the time, most groups of rollouts agree and carry no learning signal. That is why these runs weight hard tasks and add traj2envs tasks in the 0.1 to 0.7 solve-rate band.