Your Agent Aced the Task. Will It Do It Again?
The article argues that average benchmark accuracy hides agent unreliability: a ReAct agent on GPT-4.1 scored 77.4% Mean@5 on AppWorld but only 53.0% Pass^5 (succeeding on every run). Their Consistency Analyzer resamples decision points from a single recorded trajectory to find flip-prone steps, and the resulting injected guidelines halved the gap (24.4pp to 12.0pp) without costing average accuracy. Tradeoffs: diagnosis needs one extra LLM call per decision step, and consistency is orthogonal to capability, so a bigger model won't fix it.
Hugging FaceEvelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar, Ayhan Sebin, Michał UlewiczSep 16, 20269 minadvancedUnread