Should CI testing always be automated?
28 Nov 2022
Table of Contents
- Q: Should testing always be automated in continuous integration?
- A: Yes. Sorta. Huh?
Yes
I’m of the opinion that automated testing is at the heart of continuous integration.
According to Wikipedia, “CI is the practice of merging all developers’ working copies to a shared mainline several times a day.”
Humans are fallible – bad nights of sleep, etc. “Several times a day” is definitely often enough to result in a human forgetting to test their code before polluting a shared codebase with it.
Forcing contributions to a “mainline” to fail if any tests within an automated test suite fail is, theoretically, the only way to keep “merging all developers’ working copies several times a day” from becoming utter chaos.
Sorta
Some “OK, the codebase has been turned into a running system – go test it” QA processes take 10 minutes for a human to do while reading a Word document and would take months to develop into Selenium scripts.
And if the UI changes (possibly not even within your control for vendor-developed products), do you really want to have all CI/CD builds/deploys fail while your best Selenium developer is on vacation?
Force yourself to answer the question “Why not?” rather than “Why?” when deciding whether to build automated tests for CI processes. Start from an answer of “Yes, we should” and talk yourselves out of it.
But acknowledge that you might, realistically, talk yourselves out of it if you have limited developer resources and ample quality assurance resources.
Pro tip: Maintain Knowledge Base articles, Word documents, etc. full of instructions for humans as thoroughly as you would maintain automatically-executed code-based test suites. You can even practice test-driven development using a Word document full of instructions for a human as the “unit test.”
You don’t want to lose out on the ability to iterate rapidly that principles like test-driven development and “run all tests every time you release a new feature” bring to software development and operation just because you’re using a human instead of a machine to “run tests.”
Humans – particularly everyday end-users whose intuition informs excellent requirements analysis and test development – often have a hard time keeping track of rapid changes to what “it works correctly” means, so don’t make them think. Don’t set them up to fail or forget what you really wanted tested in a rapidly changing environment. Give them explicit QA instructions (even if you’re just writing their own words down) and keep the instructions meticulously up-to-date.
Huh?
If the consequences of chaos are insignificant enough that multiple developers have been collaboratively editing a centralized collection of source code for decades without stepping on each others’ toes just by “being careful” (despite human fallibility) and they can’t even begin to imagine how automated unit tests even could work against the code they’re contributing to the collection (that’s the “Huh?”)…
…You might be able to improve developer productivity with modern CI/CD tooling (e.g. using a main Git repository branch and Jenkins pipelines to put code onto a server rather than everyone SFTPing updates onto the server by hand, which helps them better comment their updates and roll back / fix incidents of having stepped on each others’ toes) even if no one takes the time to add any automated test suites to the Jenkins pipeline.
Additional resources
- Already manually running regression tests in Postman’s point-and-click software? Maybe Postman’s CLI can help you automatically run them as part of your CI/CD pipelines.
