Dan Luu

An empirical eval of 26 prompt conditions (TDD, fuzzing, property-based testing, formal methods, and testing skills) given to coding agents implementing Zstd in Rust, ~80 runs each. Nothing beat the default no-instructions baseline; agents apply techniques superficially—vacuous proofs, trivial random tests—while TDD and popular testing skills underperformed. A brief hand-written skill nudging risky-area checks and structured randomization scored best, suggesting expert guidance matters more than naming techniques.

The author argues that 'brain-off' LLM use—having agents write code and just relaying failures back (being a 'meat proxy')—still produces poor-quality software, failing on out-of-distribution problems and overfitting to tests or metrics. Even if it eventually yields good software, it won't benefit the employee, since a company could run the LLM loop without them and lay them off. He contends the present is the worst time to disengage, as companies are eager to cut low-contribution roles.