Anyone out there ever hear of delivery QA? Neither have we.
When delivery was a manual process, there was a release plan (or checklist) that was executed once in a blue moon for the release. Months were required to set up the release plan, and the release itself was a rather traumatic event for all involved. In a world like this you didn’t need QA, you needed some sort of audit mechanism (like a checklist).
In today’s world of DevOps and delivery automation, the release plan is actually code. The problem is that, as we all know, code needs QA – but code associated release doesn’t really go through any formal QA process (i.e. test plan, test harness, formal acceptance criteria).
In most cases the application sanity tests are assumed to check the deployment as well. In some organizations they actually differentiate between types of sanity tests – the ones you call Ops when they break vs. the ones you call Dev. The problem with this approach is that as deployment architectures become more complex, and deployment code grows it becomes more bug prone, and the lack of QA will become a bigger and bigger issue.
If staging environments were completely equivalent to production, you could claim that the application testing done during staging should be sufficient. If staging successfully completes the application test plan – then it has been deployed correctly. The problem is that because of economic considerations – production deployment is an environment similar to staging, but not a clone. Whenever things are different – you can be sure code will break especially when there is no way to test.
Since there really isn’t a way to assure Delivery in production, the answer has to be Production Assurance, a way to compare and understand the differences between the staging and production to ensure correct delivery without QA.