LLMs are terrible at designing AI systems

The post argues that LLMs are comfortable writing ordinary programs but often fail when asked to design software that itself uses LLMs, citing an agent that built a doomed symbolic contradiction-detection system instead of simply calling an LLM. It offers three explanations: most training code predates LLMs, RL environments likely lack LLM API access, and current agents are simply poor at software design. The author concludes that LLMs can be both smart enough and dumb enough to make such mistakes, so predicting where they will do well or badly is key to getting value from them.