Model Releases
MineCEraft: Evaluating Language Models as Construction Engineers in the World of Minecraft
arXiv:2608.28884v1 Announce Type: new Abstract: We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to s
arXiv:2608.28884v1 Announce Type: new Abstract: We introduce MineCEraft (Minecraft Construction Engineering Benchmark, pronounced mine-see-ee-raft), an easy-to-use, open-source benchmark designed to systematically evaluate the reliability and limitations of LLMs for construction tasks in Minecraft. The MineCEraft benchmark comprises 723 domain-expert hand-crafted natural-language instructions with programmatically verifiable evaluation, spanning 17 distinct task categories, providing a safe and controllable experimental environment for assessing LLMs' ability to perform realistic construction engineering tasks. With this benchmark, we conduct an in-depth evaluation of state-of-the-art LLMs and perform a detailed error analysis, revealing key failure modes and practical challenges in applying LLMs to construction engineering tasks.
Related
- EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving
- GVGAI-LLM: Evaluating Large Language Model Agents with Infinite Games
- Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
- Do VLMs Reason Like Engineers? A Benchmark and a Stage-wise Evaluation
Source: arXiv cs.AI | 2026-09-01