Leveraging training and search for better software engineering agents

SWE-bench¹ is the leading benchmark for evaluating the capabilities of autonomous software engineering agents in addressing real-world software issues. It consists of 2,294 issue-pull request pairs sourced from 12 popular Python repositories on GitHub. The goal of the system being tested is to generate a patch based on the issue description that passes the relevant tests, while having access to a containerized environment with the repository where it can execute commands and interact with the code. The SWE-bench data preparation process involved:

• Issue selection: identifying resolved issues from selected repositories that include both the problem description and the corresponding solution.

• Data pairing: associating each issue with its respective pull request, which contains the code changes implemented to address the issue.

• Test integration: incorporating unit tests from the pull requests to verify the correctness of the solutions.

There also exists a curated subset of SWE-bench, SWE-bench Verified, where experts have confirmed that the included issues can indeed be solved based on their descriptions and that the solutions are correctly validated using the provided tests.

SWE-bench has driven significant progress in the development of software engineering agents, with many researchers actively working to enhance their systems' performance. However, the benchmark is still far from being fully mastered, indicating substantial room for further improvement. While no benchmark is perfect, SWE-bench provides a meaningful and reliable assessment of progress in building better software engineering agents. It captures the key challenges and capabilities of these systems, making it an invaluable tool for measuring advancements.

The bitter lesson and agentic systemsThe bitter lesson and agentic systems

Recent research on LLM-based agentic systems largely focuses on building better scaffolding — more sophisticated frameworks and logical structures to coordinate an agent’s actions. However, this emphasis contradicts what has come to be known as the bitter lesson: historically, approaches that scale well with compute, such as search and learning, have consistently outperformed those relying on carefully engineered structures. While complex scaffolding may yield short-term improvements, simple and general methods that capitalize on computational scale ultimately prevail in the long run.

Moreover, when scaffolding is the main focus, valuable trajectory data generated by the agent is often underutilized. We believe in fully leveraging the data through learning from it, rather than merely using it to guide the process of imposing constraints.

Top performing agents leverage frontier modelsTop performing agents leverage frontier models

Top-performing software engineering agents mostly utilize frontier models, e.g. GPT-4o or Claude 3.5 Sonnet, to generate action proposals. To improve on such systems, one could attempt to train an action generator narrowly focused on the problem of agentic software development, hoping that this specialization would enable it to outperform generalist models, which must excel across many domains. However the reality is that frontier models are exceptionally good in their target domains due to an extensive amount of resources dedicated to their development. These models are also specifically trained to write code and reason about its correctness, making them well suited for powering software engineering agents. Even if a specialised model briefly outperforms them, frontier model performance constantly advances, likely rendering such efforts redundant. This is why we are more interested in approaches that can potentially work alongside frontier models, rather than replace them, while still benefiting from compute scaling.

An alternative approach: guided search with critic modelsAn alternative approach: guided search with critic models

In reasoning-heavy domains such as mathematical problem solving or software engineering, there’s often a significant gap between the number of problems that current top systems can solve reliably and those they can only solve occasionally. For instance, the plot below illustrates the performance of a reasonably capable gpt-4o-based agent on SWE-bench Verified under two scenarios:

• best-of-N: the agent runs on each problem instance N times, and a problem is considered solved if it is successfully solved in at least one of these runs.

• random: a problem is considered solved if a solution, randomly selected from N runs, is correct. This serves as a Monte Carlo estimate of the success rate using N samples.

The dataset we trained the model on consists of SWE-agent trajectories that successfully solved various SWE-bench-like problems. These problems come from either the SWE-bench dev set or an additional set of 3.3k issues from 1.3k repositories with a permissive licence, which we collected using SWE-bench recipe to expand our training data. We took only those issues where we could set up the environment and successfully run tests with golden patches applied without manual intervention. We ensured that no mirrors or forks of repositories from the SWE-bench test set were included to avoid data leakage. We gathered these trajectories by running SWE-agent with instruct models or earlier versions of action generators we trained, a process similar in spirit to expert iteration¹⁰.

We collected a large number of successful trajectories for easier problems but significantly fewer for harder ones, which required filtering and rebalancing the dataset. We first filtered the trajectories to exclude those with clearly incorrect actions, such as malformed LLM responses, tool misuses, or linter-triggering errors. We then removed duplicate trajectories for each problem, retaining only unique action sequences, and kept only the 30 shortest trajectories for each problem. The resulting training set contained 2.7k trajectories.

The action generator trained using the described setup performs significantly better than its parent model, Qwen-2.5-72B Instruct, but still falls slightly short of gpt-4o (2024-08-06) performance. Nevertheless, the performance gap is small enough for the in-house action generator to serve as a reasonable proxy for frontier models in our experiments.

Rerun until submittedRerun until submitted

SWE-agent terminates the problem-solving process when one of two conditions is met: either the agent issues a “submit” command, indicating the problem is considered solved, or it encounters an unrecoverable error (e.g. the LLM runs out of context). If the latter occurs, the changes accumulated in the working copy are treated as the generated patch. However, if the LLM exhausts its context, it’s often because the agent made an unrecoverable mistake, leading to an incorrect trajectory.

One way to mitigate this, which can be considered a simple search strategy on its own, is to rerun the agent until it either successfully submits a solution or exhausts the maximum number of attempts:

def run_agent_until_submitted(llm, issue_id, run_agent_fn, max_retries):
　trajectory = None
　for attempt_idx in range(1 + max_retries):
　　trajectory = run_agent_fn(llm, issue_id)
　　if exit_reason(trajectory) == "submitted":
　　　break
　return trajectory

As shown below, the default submission rate of our action generator is around 50%, but allowing for up to three retries increases it to 80%, and up to nine retries — to 90%. Most trajectories require significantly fewer attempts, so this strategy doesn’t drastically increase compute costs. For instance, achieving an 80% submission rate typically requires only one extra attempt per problem on average, while reaching 90% requires about two attempts. As we will show later, these numbers further decrease as agent policy improves.

What are the benefits of improving submission rate? Submitted trajectories are more likely to be successful than random ones, so conditioning on the fact of a submission improves agent performance, fully closing the gap to gpt-4o in our case. One major downside of our action generator compared to gpt-4o is its weaker ability to recover from mistakes — a limitation that re-running until submission essentially circumvents. We’ve also found that this strategy generally reduces inter-run performance variance, making evaluations less noisy (although it was not the case for the evaluation run shown below).

Process supervision is often considered more powerful and sample-efficient, but obtaining per-step target data is challenging. It requires either a costly and labour-intensive manual annotation process³ or significant compute resources to calculate Monte Carlo estimates of action value⁴˒ ⁵. Outcome supervision is simpler, requiring only positive and negative trajectory examples. However, to train a critic capable of per-step decision-making using outcome supervision, an additional model is needed to break down the outcome into per-step scores.

After experimenting with both strategies, we settled on a hybrid approach. Our critic model is trained as an approximator for both the value and action-value functions, using discounted rewards-to-go as targets. Rewards come from two sources: a large terminal reward computed by evaluating the trajectory as in the regular outcome supervision regime, and smaller per-step rewards produced by gpt-4o prompted to recognize clearly poor actions. While these per-step rewards are not strictly necessary, they significantly improve sample efficiency of training. We also tried using bootstrapped targets but found no advantages over rewards-to-go.

More formally, our training dataset D consists of trajectories

with each trajectory composed of alternating observations, actions and rewards. We train the critic model to predict

using single-sample Monte-Carlo estimates of the corresponding expectations as training targets. Each step’s reward is either 0 or 1, except for the terminal step, where the reward is either 0 or 20.

The training dataset consists of 8.3k positive and 12.7k negative trajectories collected using the same problem instances we used for training the action generator. To maximize the utility of the trajectory data collected over the course of this project, our dataset consists of trajectories collected by running SWE-agent with multiple LLMs having drastically varying performance levels. To help the critic model account for these performance variations when estimating value functions, we additionally prompt it with a string identifying the LLM used to produce the trajectory being evaluated, such as swe-agent-llama-3.1-70b-instruct. This string allows the critic model to adjust its predictions to the current policy.

We train the critic for 3 epochs with a batch size of 128, using LLaMa 3.1 70B Base as the starting point. When calculating discounted rewards-to-go, we use a relatively small discount factor of 0.85, ensuring there’s no incentive to delay issuing the “submit” command in favor of collecting additional intermediate rewards. Both value and action-value predictions use the L2 loss with equal weighting. We experimented with cross-entropy losses for value prediction (e.g. using 2-hot targets) and with predicting action advantages instead of values but found no consistent benefits.

A critic model trained using the described setup can estimate the values of individual states, the values and advantages of actions, and the correctness of the overall trajectory (by looking at the action-value of the final action, such as “submit”). In the next section, we’ll discuss how to use these predictions for various types of guided search.

1-step lookahead1-step lookahead

A straightforward search strategy that leverages our critic model is to generate multiple action candidates at each step, then select the action with the highest predicted value or advantage:

def run_agent_1_step_lookahead(llm, critic_llm, issue_id, num_action_candidates):
　env = init_environment(issue_id)
　trajectory = init_trajectory_from_issue_description(issue_id)
　while not finished(trajectory):
　　action_candidates = [
　　　generate_next_action(llm, trajectory)
　　　for _ in range(num_action_candidates)
　　]
　　action = select_most_valuable_action(critic_llm, trajectory, action_candidates)
　　observation = execute_action(env, action)
　　trajectory = update_trajectory(trajectory, action, observation)
　return trajectory

This strategy has some clear benefits:

• It’s easy to integrate into an existing agentic system.

• It has minimal impact on action generator inference costs, as it only affects the number of output tokens produced, while it is input tokens that mostly determine agentic inference costs, especially when dealing with long trajectories.

However, it’s important to note that 1-step lookahead is a local, greedy search strategy that cannot fully explore the search space, potentially limiting its impact. Despite this, applying 1-step lookahead to our agent yields clear improvements in performance and reduces the number of retries needed to achieve submission.

Given the local nature of this search strategy, we hypothesize that increasing the diversity of action candidates can improve performance by allowing a larger fraction of the search space to be explored. To achieve that we can use higher temperatures when sampling action candidates. However, we find that action quality quickly deteriorates as the temperature raises, limiting the potential to generate diverse actions and reducing performance. There appears to be an optimal point around T=0.9.

Another way to improve action diversity is by increasing the number of action candidates. However, we don’t observe further benefits beyond 4 candidates at T=0.9.

We’ve also experimented with prompting gpt-4o to judge action candidates to serve as a baseline for the learned critic, but didn’t observe any benefits compared to no lookahead.

In summary, this simple search strategy is relatively inexpensive in terms of additional compute and delivers substantial performance gains, boosting our agent’s resolved rate by approximately 1.5x.

Trajectory selectionTrajectory selection

Another simple way to search within trajectory space using a trained critic is to generate multiple complete solution trajectories for each problem and then use the one with the highest estimated success probability:

def run_agent_trajectory_selection(
　llm, critic_llm, issue_id, run_agent_fn, num_runs
):
　trajectory_candidates = [
　　run_agent_fn(llm, issue_id)
　　for _ in range(num_runs)
　]
　return select_trajectory_with_max_prob_of_success(
　　critic_llm, trajectory_candidates
　)

This strategy directly attempts to close the gap between the average and best-of-N performance discussed earlier. Essentially, it approximates the best-of-N selection process without access to evaluation results. It offers several practical benefits:

• The strategy is agnostic to the policy used to generate the trajectory candidates, allowing for easy integration with other methods.

• Multiple trajectories can be generated in parallel, meaning that although trajectory selection requires additional compute, it does not increase latency.

Applying trajectory selection on top of our action generator yields clear performance benefits and substantially reduces the resolved rate gap to best-of-N. Interestingly, performance doesn’t monotonically improve as the number of runs increases. This effect is caused by occasional out-of-distribution (OOD) trajectories that our critic model mistakenly assigns high scores to; as the number of runs grows, the likelihood of encountering such trajectories also rises.

Trajectory selection performs even better when applied on top of solutions generated with 1-step lookahead, merging the benefits of both search methods. Interestingly, the performance now monotonically depends on the number of runs, suggesting that fewer OOD trajectories are produced with lookahead. When using 10 runs, the combined approach reduces the performance gap with best-of-N more than 2x and achieves 48% resolved rate on verified-50.

We also ran this setup on the full SWE-Bench Verified dataset using the original SWE-bench code for evaluation, where it scores 40.6%. Performance reduction relative to 48% achieved on verified-50 is caused by several factors:

• A number of environment fixes that we’ve made in our fork of the evaluation codebase.

• Slightly overfitting to verified-50 when selecting the temperature and the number of action candidates.

Nevertheless, at the time of writing, this result is to the best of our knowledge state-of-the-art among approaches relying solely on open-weight models, and the best result achieved using SWE-agent scaffolding.

In summary, trajectory selection provides a substantial performance improvement, especially when layered with 1-step lookahead, enhancing our agent’s success rate and narrowing the gap to best-of-N performance.

Combining search with frontier modelsCombining search with frontier models

To demonstrate that the benefits of critic-guided search extend to more powerful generators, we first compare the performance of SWE-agent using gpt-4o (2024-08-06) with and without 1-step lookahead. We use the best setup identified for our action generator: sampling 4 candidates with a temperature of T=0.9. The plot below demonstrates that our critic, though based on a weaker model, can meaningfully enhance gpt-4o’s performance through guided search. Additionally, running the model until submission further boosts performance. Interestingly, this setup does not outperform the same configuration applied to our in-house action generator, suggesting that search acts as an equalizing factor, increasing the reliability of weaker models to reach a quality level comparable to frontier models.

We can further enhance performance by applying trajectory selection. To keep costs down, we tested up to N=5 runs. As the number of runs grows, the resolved rates of the in-house generator and gpt-4o remain closely aligned, suggesting that their performances are comparable in both mean and variance (e.g. gpt-4o does not generate a more diverse set of candidate solutions).

Another promising avenue is creating an iterative process where the policy induced by guided search is distilled back into the action generator. This process could involve retraining the critic to produce an even better search policy, then refining the action generator again, and so on. This approach—combining search and learning—has been highly effective in previous works, notably AlphaZero⁹, and could be directly applicable to agent-based domains, though scaling it efficiently presents new challenges.

ConclusionConclusion

Modern LLM-based agentic systems, particularly those designed for automating software engineering tasks, can reliably solve simple problems but remain prone to failure as complexity increases. Our work demonstrates that applying search guided by a critic model can significantly enhance these systems' reliability, bridging the gap between best-case and average-case performance. Specifically, we illustrated how to train a critic model on agent trajectory data, how to leverage it within various trajectory search strategies, and how to combine these strategies for even greater performance gains. Our critic-based approach adds value not only on top of open weight models like LLaMa or Qwen, but also when paired with frontier LLMs. Notably, search also helps to close the performance gap between these model families.

By focusing on scalable approaches, such as learning and guided search, rather than overly intricate scaffolding, we’ve taken steps toward creating more robust and adaptive agentic systems. We are excited to continue exploring whether more sophisticated search methods can further enhance these systems’ performance.