How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

arXiv:2607.10856v2 Announce Type: replace-cross Abstract: The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing work in requirements, coordination, and deployment becomes more visible, while reviewing generated code and evaluating agent behavior become new and increasingly central forms of work. We characterize a seven-stage workflow and five process shifts, including a move toward evaluation-driven development, in which evaluation is increasingly defined early and steers iteration, and the emergence of specifications as first-class artifacts that teams test and version alongside code. We further identify six challenges that teams face, together with 12 corresponding practices they use or propose to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.

Related

• Comparing LLM-Based Conversational and Graphical Interfaces for Industrial Decision Tasks: An Exploratory Mixed-Methods Study
• Effective Strategies for Asynchronous Software Engineering Agents
• Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application

Source: arXiv cs.AI | 2026-07-28