We like to think about application delivery as a promotion pipeline from development to production. Each step in the pipeline is usually a separate environment – e.g. a different topology, different configuration and different tools+methods for promotion to the next step.
One school of thought is that DevOps exists to solve the problem of wasted developer time for creating environments. Developers are such a scarce and expensive resource, that on the face of it makes sense to subordinate operations to development. Wrong!
Taking that approach you recreate the problem of silos inherent in the machine metaphor – each role with its own specific function. By allowing development to ignore the needs of production you are setting yourself up to create products that that are like a “spherical cow” – creating an architecture that is beautiful in theory – but doesn’t work in practice.
A better alternative is the “organic metaphor” where responsibility for the final deliverable is shared across the organization through “design for delivery”. For any new software project, there are many (sometimes conflicting) ways to architect the product. In fact, one of the most important jobs of the architect is to take into consideration conflicting design patterns and possibilities and choose the optimal architecture. Historically, delivery wasn’t a key consideration in software product architecture design. Actually that is true for many products, not just software.
Given today’s focus on delivery, “design for delivery” connects architecture to DevOps, and is the difference between product success and failure. Delivery must be taken into consideration at every stage of product creation.
Design for delivery is a critical part of the DevOps shift left paradigm. We like to add “think right” to the equation because shift left isn’t enough – the whole organisation needs to include the ability to easily deliver the product as part of product design and implementation.
A great illustration of problem with the machine metaphor is in Jez Humble’s post on “Elisabeth Hendrickson Discusses Agile Testing” where she discusses working at a product company which was suffering a series of quality problems. As a result, they hired a VP of QA who set up a QA division. The net result of this, counter intuitively, was to increase the number of bugs. One of the major causes of this was that developers felt that they were no longer responsible for quality, and instead focused on getting their features into “test” as quickly as they could. Thus they paid less attention to making sure the system was of high quality in the first place, which in turn put more stress on the testers. This created a death spiral of increasingly poor quality, which led to increasing stress on the testers, and so on.
Achieving Design for Delivery
- Identify delivery pains; organise and develop feedback loop for operations and development: delivery and operational experts should be pigs in every sprint 0 and should be chickens at scrum meetings – in the real world agile teams have developers, customers, product but lack a key component – delivery. It doesn’t matter why, but it ensures that what ever the team develops won’t be optimised for delivery – or even worse it won’t deliverable.
- Delivery must be a part of every agile team – the problem is that there just aren’t enough delivery people to get them involved in every sprint for every team. Of course, you can always hire a lot more delivery folks – but that just won’t happen.
- The solution is to make delivery a pig in every sprint 0 (planning sprint). They will plans and validate the tools and scripts to be built in support of team. They will also provide important input for time boxing and the order of feature development. They should also be part of every sprint review demo – just like customers. In a way delivery is a customer – they are the customers for the team’s artifacts and their job is to deliver them to the real customers.
- Design by doing: new functionality is set by business and combined with delivery requirements defined by operations – this is the “theme”. The theme is then broken into smaller viable components (epics and stories in agile). e.g. faster deployment time, automatic deployment of components, simpler restart and recovery, built-in an event console for errors and logs, built-in automatic DB upgrade.
- The first step for the architect is to introduce one of these epics even if it requires using a kludge, sacrificing architectural integrity and entering technical debt. Over time the architect is able to assess various implications of the technical debt, e.g. excessive response time, excessive memory footprint, dramatically higher maintenance requirements.
- In the next phase, the architect fixes the most costly debt of the previous epic, i.e. refactors and aligns the architecture, while kludging on another epic. This “Firefighter” mode contains the most damaging aspects of the technical debt while still delivering rapid response. By only fighting the most damaging “fires” we can achieve optimize for delivery without rearchitecting the entire product. The rule is to start with a quick-win, and later fix only the crucial problems.
- Shift left – think right: Recognise and identify deployment challenges earlier, i.e. during the development phase. Fix as soon as possible taking into consideration deployment and operational processes.
- Development labs include real deployment use-cases and works with release automation tools that includes automation of software, DB and configuration.
- Build-in health & monitoring tools based on KPI that are valuable for deployment and operations. Build-in an event console.
- Daily build, deploy & stabilise lead by development
- Review and manage, adopt and simplify deployment checklists
- Deployment checklists as a process
This is the last in our “What DevOp Days Doesn’t Want You to Know Series” – hope you enjoyed them.