Scaffold CoT: A CoT dataset built around the failures of small model (>5B Params) free form thinking. Hope its useful to you guys!

TL;DR - A ~4M example, ~3B token CoT dataset designed around helping small models think more concisely, accurately and reliably. Hi all! For the past few months I have been working on a dataset designed around improving small model performance through a structured framework (or Scaffold) for thinking. When using small models (Under 5B parameters), I noticed freeform CoT does not really add much in terms of capability, and usually results in more confusing, poorly structured and inaccurate responses. This dataset is built to remediate that via: A structured framework that the model can lean on to help structure reasoning and responses A large range of example categories for generation to improve topic-specific performance and facilitate specialization into a specific topic or task A maximum example size of 2048 tokens to ensure reliable good output and facilitate training on consumer machines The framework itself Every single example uses the same three sections, in the same order, no exceptions: Inventory: what I have - tools, inputs, constraints, what is known Interaction: how those things affect each other Execution: working through it to an answer ...the answer... The idea is that the format becomes a constant, so the model spends its capacity on the actual content instead of re-deciding how to lay out a thought every time, reducing the number of points of failure and increasing coherence. A nice side effect is that you can validate a fine-tuned model's output with a regex, since a broken scaffold is machine detectable in a way that freeform CoT never is. What varies is depth, not structure. Examples are generated across four depth tiers with roughly a 5.6x spread between the deepest and the shallowest, so the model learns when to think longer rather than always running in one gear. Categories Every example carries an exact domain and a non-empty subdomain in its metadata, so you can filter down to precisely the slice you want to train on. 18 domains, 798 subdomains, and no unlabelled remainder. The counts below sum to the entire dataset. code - 669,517 across 75 subdomains (tracing, complexity, debugging, API design, data structure choice) general - 643,561 across 56 subdomains (factual, definitional, comparison, troubleshooting, advice) antihal - 502,151 across 9 subdomains (knowledge boundaries, calibrated answers, premise auditing, refusing to fabricate) science - 277,072 across 35 subdomains (scale intuition, data interpretation, mechanism reasoning) logic - 258,697 across 21 subdomains (fallacy ID, causal reasoning, counterfactuals, missing information) strategy - 223,119 across 141 subdomains (second order thinking, negotiation, long term positioning) business - 142,343 across 36 subdomains (applied commercial decisions and tradeoffs) creative - 141,739 across 54 subdomains (open ended generation and idea work) tool_use - 141,389 across 48 subdomains (function calling, single and multi step) tool_use_complex - 137,687 across 210 subdomains (long tool chains, including 87k really executed ones) selfcheck - 129,303 across 12 subdomains (catching your own error mid reasoning and correcting it) writing_tone - 124,985 across 41 subdomains (register, audience and voice control) longdoc - 98,039 across 12 subdomains (long input handling) mathqual - 60,002 across 16 subdomains (quantitative reasoning) format_strict - 59,640 across 18 subdomains (hitting an exact output format) storytelling - 58,713 across 29 subdomains (narrative construction) steelman - 51,047 across 12 subdomains (arguing the strongest opposing case) antihal_kb_refined - 49,902 across 10 subdomains (refined knowledge boundary cases) Full subdomain counts are in manifest.json under domain_subdomain. Filtering is just: ds = load_dataset("Specific-Labs/Scaffold-CoT", split="train") tools = ds.filter(lambda r: r["metadata"]["domain"] == "tool_use_complex") 3.07M examples are single turn, 698k are multi turn. The thing I would point at first is the calibration block. antihal plus antihal_kb_refined plus selfcheck is roughly 681k examples of the model saying where its knowledge runs out, auditing a bad premise, or catching its own mistake halfway through and fixing it. Small models are especially prone to this failure mode so a lot of effort was invested into these sections. selfcheck breaks down into 12 specific error types rather than one generic bucket: catch_logical_jump, catch_wrong_assumption, catch_terminology_confusion, catch_orientation_error, catch_correlation_causation, catch_unit_error, catch_missing_constraint, catch_arithmetic_slip, catch_overgeneralization, catch_off_by_one, catch_premise_misread, catch_scope_creep The tool calling chains are not simulated 279k examples are tool calling across tool_use and tool_use_complex. Of those, 87,162 were produced by a harness that actually ran every tool and pasted the real result back into the transcript. The model only ever wrote the reasoning around results it was handed, which means fabricated tool output is structurally impossible in those examples rather than just discouraged. 4.2 tool calls per example on average 100% of them contain a real failure and a recovery from it, not a clean happy path Filter on metadata.subdomain == "multi_step_real_execution" if you want just those. The honest caveat is that the tool result text itself is a bit repetitive, since a fixed set of tools emitting structured output produces structurally similar text and no amount of prompt variety fixes that. The prompts and the reasoning are varied though, so should work well for training anyway. Why break down the examples via topic and subtopic Despite not being the standard practice for CoT (Usually math and logic focused, not topic focused), this provides several advantages at the small LLM scale: Improved task adaptation for small models, which in my experience are less capable at generalizing topic specific CoT into other topics Helps users select specific subtopics and topics for training to address specific points of failure Allows for better domain and task specialization, particularly important for small model deployments What one example actually looks like Taken verbatim from the dataset, domain: code, subdomain: code_tracing, difficulty medium. The prompt Stuck with an Express.js middleware chain that needs to run a time-sensitive health check against a Pulumi-provisioned endpoint, and the regulator just changed the SLA rules mid-quarter, now we need 3 nines of uptime measured in a 4-minute rolling window. I'm running this service out of the NYC Department of Health field office and I just hit a daylight saving time bug that only fires on the last Sunday of October. Here's the function I'm tracing. What does it return? I had good coffee this morning so I'm sharp, but the DST edge case has me second-guessing every timestamp comparison in the stack. The response Inventory: I have the following tools and inputs available: * the function checkStatus takes two arguments: codes (array of HTTP status codes) and window (minimum ratio threshold) * input array: [200, 200, 500, 200, 200, 200, 200, 200, 200, 200] * window argument: 0.9 * the function counts how many codes are in the range [200, 400), divides by total length, and returns 'PASS' if ratio >= window, else 'FAIL' Interaction: I can deduce these items interact in the following way: * the loop iterates over all 10 elements in the codes array * each element is checked: if >= 200 and < 400, the up counter increments * 200 is >= 200 and < 400, so every 200 counts toward up * 500 is not < 400, so it does not count toward up * in the array, there are nine 200s and one 500, so up ends at 9 * ratio = 9 / 10 = 0.9 * the comparison is ratio >= window, which is 0.9 >= 0.9, which is true Execution: I now work through the plan to produce the answer: * count the passing codes: nine 200s pass, one 500 fails -> up = 9 * compute ratio: 9 / 10 = 0.9 * compare: 0.9 >= 0.9 evaluates to true * the function returns 'PASS' The function returns 'PASS'. The array has 10 status codes: nine 200s (successful responses) and one 500 (server error). The success ratio is 9/10 = 0.9. Since the threshold is 0.9 and the comparison uses >=, the function returns 'PASS'. Two things worth noticing The prompt is deliberately noisy. The DST bug, the regulator, the field office and the coffee are all irrelevant to the question, and none of them appear in the Inventory. Real questions arrive wrapped in context that does not matter, and a typical failure mode of small models, and need to learn to drop it. The answer is also checkable. Nine 200s out of ten, ratio 0.9, comparison is >=, so it returns PASS. That is the point of a fixed scaffold, the reasoning is laid out in a form where you can actually follow it and catch it being wrong, instead of a paragraph that sounds plausible. On quality control Every rule is enforced by throwing away examples that to not adhere to them (pruned somewhere between 30-50% of generations to reach the current version): 2048 token cap, actually measured rather than estimated Thoroughly deduped, mechanically and semantically Full LLM Judge pass for corrections and best-example selection Scaffold structure has to be intact, three headers, bulleted body Topic repetition capped at 10 per topic Opener share capped at 2% against the running distribution, so you do not end up with 40% of examples starting the same way Behavioural checks, a selfcheck example has to actually contain a correction, an antihal example has to actually articulate a boundary I also ran a leakage sweep over the training text and pulled 11,293 examples (0.31%) that had broken scaffolds, unclosed think blocks, generator artifacts or template placeholders in them. Being upfront about the limitations The scaffold is a commitment. A model trained on this will reach for Inventory/Interaction/Execution. If you want format flexible reasoning this is the wrong dataset. English only. There is a long tail of 97 junk domain labels covering 542 examples. Realized while writing the post. A bit too lazy to clean it up lmao, but will eventually. Link https://huggingface.co/datasets/Specific-Labs/Scaffold-CoT CC-BY-4.0, 74 JSONL shards, around 15GB. Happy to answer questions. I am especially interested in hearing from anyone who has tried structured vs freeform CoT on small models, and in what categories you think are missing. Would love to hear more about any training runs you guys do. Small recommendation: If you are going to use the dataset for training, I recommend using base models instead of already SFT'ed versions. I have seen significantly better results like that. Overall, initial testing runs (LFM, Gemma 4) are highly encouraging, with performance improvements in benchmarks, though I also notice a significant improvement in perceptible 'vibes'. Posts around those soon. Genuinely hope this dataset proves useful to the community! submitted by /u/Saraozte01 [link] [comments]

Source: r/LocalLLaMA | 2026-08-25

Related