it almost never makes sense to use real api's for your evals. with how good coding agents have become, i will pretty much always opt to crea…

it almost never makes sense to use real api's for your evals. with how good coding agents have become, i will pretty much always opt to create a fake mock server for my agent to hit. the workflow is usually - fetch tasks for each eval - mock the endpoints that i need my agents to hit - create fastapi server for this - pre-fetch the real data i need - have the stub server return that pre-fetched data from start -> fully fledged mock server in under 30 minutes

Source: Harrison Chase (X) | 2026-05-15