thoughts after doing a bunch of synthetic data gen for eval + environment building - LLMs are incredible projections of the world bundled in…

thoughts after doing a bunch of synthetic data gen for eval + environment building - LLMs are incredible projections of the world bundled into a set of weights - but doing targeted extraction of certain distributions from those weight is incredibly difficult to do at scale. this is especially true for distributions that are less represented in the training corpus like long sequences of data - rubric design + tons of compute on data generation and more importantly validation is a good recipe that can help you build datasets beyond the natural tendencies of models today. these are things like programattic tests on length, verifiers running with ground truth, and judges evaluating each stage of generation - synthetic data design can be framed as an ultra long horizon agentic task. this is helpful framing because it lets us use the most powerful tools we have today…coding agents - it’s unclear to me the exact deviations between frontier models for large scale synthetic data production. anecdotally Opus seemed to follow diversity stipulations better but it’s unclear how this holds at scale - the ideal scenario is distilling the verification, instruction following, and diversity primitives into frontier open models where token costs are 20x less - it’s better to have 3 good examples than 50 that are hard to understand will be doing a lot more of this! it’s actually very fun and required at scale to test many types of agents

Source: Harrison Chase (X) | 2026-05-12