Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Following

arXiv:2505.02763v2 Announce Type: replace-cross Abstract: One of the central promises of legal AI is to automate drudgery -- the formal, repetitive tasks of lawyers' work that consume time without calling for much discretion. Yet it remains an open question how well AI models actually perform on such tasks. This article presents the first empirical examination of AI performance on perhaps the most ubiquitous and lamented form of legal drudgery: citation formatting under the Bluebook. We make four contributions. First, we develop a new benchmark of 2,058 Bluebook queries and show that, on average, frontier language models produce a fully compliant legal citation only 42.6% of the time in a zero-shot setting. Second, we conduct an experiment with five top law reviews and show that even a "reasoning" model falls far below the average score of the human candidates in these journals' annual editor-selection competitions. Third, we show that simply providing the models with the rules offers only modest improvements, calling into question the ability of retrieval-augmented generation (RAG) to ensure rule-following alone. Finally, we develop an approach that does meaningfully improve compliance: a neuro-symbolic system that first uses a model to parse natural language into structured citation elements, and then delegates the formatting to a deterministic rule-execution engine. This approach achieves an average accuracy increase of 32.4 percentage points and total accuracy of up to 85.5% on our benchmark. These results point toward a reorientation for legal AI. The original promise of automating drudgery still remains out of reach for even frontier language models on their own -- but pairing them with symbolic rule engines may offer a tractable path forward.

Source: arXiv cs.AI | 2026-08-18