Let LLMs remember what mutation testing is
01 Apr 2026
A colleague sent me Sushant Singhal’s Gartner webinar “Use AI Mutation Testing to Improve Software Quality”, and while I’ve only had a chance to skim the slides, not listen in full, I have a few thoughts:
- Whaddya know, mutation testing is exactly as cognitively overwhelming for software developers to understand and attempt as it was when I learned about it a decade and a half ago. 😆
- I like Sushant’s idea that that makes it a great idea to delegate to an LLM that’s been trained on software engineering concepts and does, therefore, understand mutation testing.
- Reminds me a lot of Simon Willison encouraging developers to add a simple 5-word prompt to their generative AI coding assistants: “Always do red/green TDD.” And to let the AI worry about what on earth “red-green test-driven development (‘TDD’)” means.
- That said, based on Sushant’s many slides, it does look like prompting an LLM to do mutation testing well involves a bit more setup than adding 5 words. Which isn’t suprising – mutation testing is a more complex concept than red-green TDD.
- Still, I love the idea of it perhaps finally becoming rather accessible to try!
- Although how well it’d work, compared to just burning up the planet chewing through tokens and context, probably depends a lot on how well you’ve got your coding assistant set up to lean on non-AI deterministic tooling.
- Which is also true of Simon’s post about red/green TDD. The LLM can only work in small chunks and test after each one if it’s actually set up to run non-AI deterministic automated test suites early and often.
- Still, I love the idea of it perhaps finally becoming rather accessible to try!
