StreamReason-Bench: Can Large Language Models Reason about Event-Time Stream-Processing Semantics?

arXiv:2608.12348v1 Announce Type: cross Abstract: Streaming systems increasingly hand work to large language models (LLMs) -- writing pipelines, triaging alerts, reading logs -- and all of it assumes the model knows how event-time stream processing behaves. We test that assumption head-on. StreamReason-Bench asks a model to stand in for an event-time stream processor: given a windowed query and a stream of out-of-order events, it has to report which windows fire (with their aggregates) and which events are dropped as late. The answer key comes from a small reference implementation of Dataflow-model semantics, so we can grade exactly, and with a partial-credit row-F1, without running an engine. On 600 generated items covering tumbling, hopping, session, and processing-time windows, the models do poorly on event time. Told to answer directly, no model that actually follows the instruction clears 34% exact match; chain-of-thought (CoT) roughly doubles that for several of them (GPT-4o goes from 0.34 to 0.48), and only one frontier model that reasons by default comes near solving the set (0.85). A processing-time control, with no watermarks and nothing late, is almost solved by every capable model. That gap points to event-time and late-data handling, not windowing or arithmetic, as the hard part. Sorting errors by window type tells the same story: late-data mistakes dominate the event-time windows and vanish on the control, and session windows mostly fail on where the session boundaries fall.

Related

• Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding
• LLMs Lean on Priors, Not Programming Language Semantics
• Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory

Source: arXiv cs.AI | 2026-08-14