LogicSkills: A Structured Benchmark for Formal Reasoning in Large Language Models

arXiv:2602.06533v3 Announce Type: replace Abstract: Large language models perform well on many logical reasoning benchmarks, but it remains unclear which core logical skills they truly master. To address this, we introduce LogicSkills, a benchmark that isolates three fundamental logical skills: (i) extit{formal symbolization}nicode{x2014}{}translating premises into first-order logic; (ii) extit{countermodel construction}nicode{x2014}showing that an argument is logically invalid by constructing a finite countermodel; and (iii) extit{validity assessment}nicode{x2014}determining whether a conclusion follows from a set of premises. Items are drawn from the two-variable fragment of first-order logic without identity and are presented in both English and a Carrollian nonce-word language. All instances are solver-verified with Z3 for correctness and non-triviality. Across conventional instruction-tuned LLMs, performance is high on extit{validity assessment} but substantially lower on extit{formal symbolization} and extit{countermodel construction}, highlighting that high task-level accuracy can mask weaknesses in core logical skills. In contrast, recent reasoning-tuned models perform strongly across all three tasks, suggesting a more systematic logical skill profile.

Related

• Kinship Data Benchmark for Multi-hop Reasoning
• MathlibLemma: Folklore Lemma Generation and Benchmark for Formal Mathematics
• CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean

Source: arXiv cs.AI | 2026-09-10