Around five years ago, I was attending my very first conference when I first heard the term Continuous Testing. I had been living in a waterfall world: developers weren’t testing their own work, and life as a tester was less than amazing. Testing was then thought of as an activity only done by testers, who really just ensured the application worked met requirements (and we had a LOT of work coming into my team).
As I heard the speaker talk about this new (to me) phrase, I thought, “Great, now I have to test everything, all the time?” If we think about how many of us experienced ‘over the fence’ testing, this could be long, long cycles of manual checking. At one point for me, this took a team of almost 45 people, 3 weeks at the end. And that was just ONE CYCLE (we had as many as 6 cycles before something was released to production, if we were lucky, once a year). Even when we began automating, at best it was a nightly run for thousands of tests, and days of debugging if one failed. Since I didn’t know much about how Continuous Testing fit into a DevOps world, I took it to mean we had to figure out how to do that much “testing” all the time. I could think of nothing worse!
This session turned out to be a (very disappointing) vendor pitch, and I wish I had heard more about Continuous Delivery, DevOps, and how testing fits into that world.
Flow and Feedback
To understand Continuous Testing and Continuous Delivery (CD), it’s important to look at the world of DevOps – the culture driving how we build, test, release, and operate software. That culture thrives by focusing on collaboration and communication, flow, feedback, and continuous learning and improvement. So how does testing and CD fit in?
Let’s take a look at the concept of flow and feedback. How does an idea go from a concept to something of value for our customers? As we look at the entire flow or process, think about how we develop our ideas. How do we explore those thoughts and discover the unknowns? What risks can we think of? How ethical is what we are trying to build? Asking ourselves these questions (and more!) is a form of exploration to discover the unknowns and give feedback – which, even if it is not me sitting at a computer, is still testing our ideas and assumptions.
Once we are ready to build, do we know how our code flows from that first commit into production? This is where Continuous Integration (CI), Continuous Delivery (CD), and Continuous Deployment (hey! Also CD! Confusing, right?) come into play.
I started in a world where we didn’t have CI. Devs could go days, even weeks, without merging their code. This made it SUPER hard to know which change broke any test we may have had running at the time. Enter Continuous Integration. This brought us the practice of introducing smaller changes, several times a day, with tests kicked off on each individual change (not a bunch of changes together). As we evolved and further matured our pipeline, we started looking at Continuous Delivery. We used to have once a year releases (nightmarish to test), then quarterly (still bad, but better than the annual releases). When CD came into the picture, coupled with the smaller changes that CI brought us, we were able to release much more frequently. These smaller releases were WAY easier to test, but we were still waiting until we felt we had enough changes of value. Enter Continuous Deployment. We started using feature flags. We could deploy on every small change to production.
As we began to understand our feedback loops throughout our CI/CD pipelines, we could prevent failures from going any further down stream if our feedback told us that change broke something along the way. We no longer had to think of feedback as one huge giant testing suite checking requirements. We could have smaller suites along the way, giving much more frequent feedback.
If I think of Continuous Deployment as driving a car down the road at night, and automation and testing as my headlights, I could _technically_ drive without those headlights turned on – but I might end up in a ditch. By using automation to improve my flow, and testing to give me feedback, I have a much better chance of getting where I’m going! (I’ll ignore the fact i have gotten quite lost along the way sometimes).
What About Testing in Continuous Delivery?
Guess what?! We haven’t even talked about what happens once we’ve given our clients something of value. Just as we can test ideas at the beginning of our process and as we build and deploy, how do we discover more unknowns and see what’s happening in the wild? We need to test in production! We log. We monitor. We introduce chaos. We OBSERVE. If we see something, we understand what we missed and we start the cycle from idea to production over again. As we seek to get feedback, we are testing. We are constantly learning, and improving, and testing.
Dan Ashby has one of my favorite models that helps me ensure I’m thinking about the feedback I need at every step in our process, and as we repeat the cycle. We test our ideas, our artifacts, our design, our infrastructure and code, our operations, our environments, our pipelines, and more!
Image Credit: Dan Ashby, Building a Quality Culture through Continuous Testing
As you can see, Continuous Testing is not just about the testing that occurs in your CI/CD pipeline. It’s everywhere. Testing is the feedback we seek, in all aspects of design, developing, deploying, and maintaining our applications. It requires everyone to embrace testing, which has been one of the most positive mindsets I have seen evolve in my career.
For more resources and community, I highly recommend:
- Accelerate, Dr. Nicole Forsgren, Jez Humble, Gene Kim
- Continuous Delivery, Jez Humble, David Farley
- A Practical Guide to Testing in DevOps, Katrina Clokie
- https://testingindevops.org/ (community resource)