Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents

arXiv:2609.00549v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.

Related

• Dynamic Skill Lifecycle Management for Agentic Reinforcement Learning
• SkillGraph: Skill-Augmented Reinforcement Learning for Agents via Evolving Skill Graphs
• Skill-RAG: Failure-State-Aware Retrieval Augmentation via Hidden-State Probing and Skill Routing
• Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents
• Formal Skill: Programmable Runtime Skills for Efficient and Accurate LLM Agents

Source: arXiv cs.CL | 2026-09-02