On Tech

Category: Digital Transformation

Multi-Demand Operations

How can Multi-Demand Operations eliminate handoffs and adhere to ITIL? Why are Service Transition, Change Management, and Production Support activities inimical to Continuous Delivery? How can such Policy Rules can be turned into ITIL-compliant Policy Guidelines that increase flow?

This is part 5 of the Strategising for Continuous Delivery series

Know Operations activities

When an organisation has IT As A Cost Centre, its IT department will consist of siloed Delivery and Operations groups. This is based on the outdated COBIT notion of sequential Plan-Build-Run activities, with Delivery teams building applications and Operations teams running them. If the Operations group has adopted ITIL Service Management, its Run activities will include:

  • Service Transition – perform operational readiness checks for an application prior to live traffic
  • Change Management – approve releases for an application with live traffic 
  • Tiered Production Support – monitor for and respond to production incidents for applications with live traffic  

Well-intentioned, hard working Operations teams in IT As A Cost Centre will be incentivised to work in separate silos to implement these activities as context-free, centralised Policy Rules.

See rules as constraints

Policy Rules from Operations will inevitably inject delays and rework into a technology value stream, due to the handoffs and coordination costs involved.  One of those Policy Rules will likely constrain throughput for all applications in a high demand group, even if it has existed without complaint in lower demand groups for years.

Service Transition can delay an initial live launch by weeks or months. Handing over an application from Delivery to Operations means operational readiness is only checked at the last minute. This can result in substantial rework on operational features when a launch deadline looms, and little time is available. Furthermore, there is little incentive for Delivery teams to assess and improve operability when Operations will do it for them.

Change Management can delay a release by days or weeks. Requesting an approval means a Change Advisory Board (CAB) of Operations stakeholders must find the time to meet and assess the change, and agree a release date. An approval might require rework in the paperwork, or in the application changeset. Delays and rework are exacerbated during a Change Freeze, when most if not all approvals are suspended for days at a time. In Accelerate, Dr. Nicole Forsgren et al prove a negative correlation between external approvals and throughput, and conclude “it is worse than having no change approval process at all”.

Tiered Production Support can delay failure resolution by hours or days. Raising a ticket incurs a progression from a Level 1 service desk to Level 2 support agents, and onto Level 3 Delivery teams until the failure is resolved. Non-trivial tickets will go through one or more triage queues until the best-placed responder is found. A ticket might involve rework if repeated, unilateral reassignments occur between support levels, teams, and/or individuals. This is why Jon Hall argues in ITSM and why three-tier support should be replaced with Swarming “the current organizational structure of the vast majority of IT support organisations is fundamentally flawed”.

These Policy Rules will act as Risk Management Theatre to varying degrees in different demand groups. They are based on the misguided assumption that preventative controls on everyone will prevent anyone from making a mistake. They impede knowledge sharing, restrict situational awareness, increase opportunity costs, and actively contribute to Discontinuous Delivery.

Example – MediaTech

At MediaTech, an investment in re-architecting videogames-ui and videogames-data has increased videogames-ui deployment frequency to every 10 days. Yet the Website Services demand group has a target of 7 days, and using the Five Focussing Steps reveals Change Management is the constraint for all applications in the Website Services technology value stream.

A Multi-Demand lens shows a Change Management policy inherited from the lower demand Supplier Integrations and Heritage Apps demand groups. All Website Services releases must have an approved Normal Change, as has been the case with Supplier Integrations and Heritage Apps for years. Normal Changes have a lead time of 0-4 days. This is the most time-consuming activity in Operations, due to the handoffs between approver groups. It is the constraint on Website Services like videogames-ui.

Create ITIL guidelines

Siloed Operations activities are predicated on high compute costs, and the high transaction cost of a release. That may be true for lower demand applications in an on-premise estate. However, Cloud Computing and Continuous Delivery have invalidated that argument for high demand applications. Compute and transaction costs can be reduced to near-zero, and opportunity costs are far more significant.

The intent behind Service Transition, Change Management, and Production Support is laudable. It is possible to re-design such Policy Rules into Policy Guidelines, and implement ITIL principles according to the throughput target of a demand group as well as its service management needs. Those Policy Rules can be replaced with Policy Guidelines, so high demand applications have equivalent lightweight activities while lower demand applications retain the same as before.

Converting Operations Policy Rules into Policy Guidelines will be more palatable to Operations stakeholders if a Multi-Demand Architecture is in place, and hard dependencies have previously been re-designed to shrink failure blast radius. A deployment pipeline for high demand applications that offers extensive test automation and stable deployments is also important.

Multi-Demand Service Transition

Service Transition can be replaced by Delivery teams automating a continual assessment of operational readiness, based on ITIL standards and Operations recommendations. Operational readiness checks should include availability, request throughput, request latency, logging dashboards, monitoring dashboards, and alert rules.

There should be a mindset of continual service transition, with small batch sizes and tight production feedback loops used to identify leading signals of inoperability before a live launch. For example, an application might have automated checks for the presence of a Four Golden Signals dashboard, and Service Level Objective alerts based on Request Success Rate.

Multi-Demand Change Management

Change Management can be streamlined by Delivery teams automating change approval creation and auditing. ITIL has Normal and Emergency Changes for irregular changes. It also has Standard Changes for repeatable, low risk changes which can be pre-approved electronically. Standard Changes are entirely compatible with Continuous Delivery.

Regular, low risk changes for a high demand application should move to a Standard Change template. Low risk, repeatable changes would be pre-approved for live traffic as often as necessary. The criteria for Standard Changes should be pre-agreed with Change Management. Entry criteria could be 3 successful Normal Changes, while exit criteria could be 1 failure.

Irregular, variable risk changes for high demand applications should move to team-approved Normal Changes. The approver group for low and medium risk changes would be the Delivery team, and high risk changes would have Delivery team leadership as well. Entry criteria could be 3 successful Normal Changes and 100% on operational readiness checks.

A Change Freeze should be minimised for high demand applications. For 1-2 weeks before a peak business event, there could be a period of heightened awareness that allows Standard Changes and low-risk Normal Changes only. There could be a 24 hour Change Freeze for the peak business event itself, that allows Emergency Changes only.

The deployment pipeline should have traceability built in. A change approval should be linked to a versioned deployment, and the underlying code, configuration, infrastructure, and/or schema changes. This should be accompanied by a comprehensive engineering effort from Delivery teams for ever-smaller changesets, so changes can remain low risk as throughput increases. This should include Expand-Contract, Decouple Release From Launch, and Canary Deployments for zero downtime deployments.

Multi-Demand Production Support

Tiered Production Support can be replaced by Delivery teams adopting You Build It, You Run It. A Level 1 service desk should remain for any applications with direct customer contact. Level 2 and Level 3 support should be performed by Delivery team engineers on-call 24/7/365 for the applications they build. 

Logging dashboards, monitoring dashboards, and alert rules should be maintained by engineers, and alert notifications should be directed to the team. In working hours, a failure should be prioritised over feature development, and be investigated by multiple team members. Outside working hours, a failure should be handled by the on-call engineer. Teams should do their own incident management and post-incident reviews.

You Build It, You Run It maximises incentives for Delivery teams to build operability into their applications from the outset of development. Operational accountability should reside with the product owner. They should have to prioritise operational features against user features, from a single product backlog. There should be an emphasis on reliable live traffic over feature development, cross-functional collaboration within and between teams, and a cross-pollination of skills. 

Example – MediaTech

At MediaTech, a prolonged investment is made in Operations activities for Website Services. The Service Transition and  Tiered Production Support teams are repurposed to concentrate solely on lower demand, on-premise applications. Website Services teams take on continual service transition and You Build It, You Run It themselves. This provokes a paradigm shift in how operability is handled at MediaTech, as Website Services teams start to implement their own telemetry and share their learnings when failures occur.

Change Management agree with the Website Services teams that any application with a deployment pipeline and automated rollback can move to Standard Change after 3 successful Normal Changes. In addition, agreement is reached on experimental, team-approved Normal Changes. Applications with the Standard Change entry criteria and have passed all operational checks no longer require CAB approval for irregular changes.

The elimination of handoffs and rework between Website Services and Operations teams means videogames-ui and videogames-ui deployment frequency can be increased to every 5 days. The applications are finally in a state of Continuous Delivery, and the next round of improvements can begin elsewhere in the MediaTech estate.

This is part 5 of the Strategising for Continuous Delivery series

  1. Strategising for Continuous Delivery
  2. The Bimodal Delusion
  3. Multi-Demand IT
  4. Multi-Demand Architecture
  5. Multi-Demand Operations

Acknowledgements

Thanks to Thierry de Pauw for reviewing this series.

Multi-Demand Architecture

How can Multi-Demand Architecture accelerate reliability and delivery flow? Why should Policy Rules be based on Continuous Delivery predictors? What is the importance of a loosely-coupled architecture? How can architectural Policy Rules benefit Continuous Delivery and reliability

This is Part 4 of the Strategising for Continuous Delivery series

Increase flow with policies

Policy Rules are not inherently bad. Some policies should be established across all demand groups, to drive Continuous Delivery adoption:

  • Software management should be based on Work In Progress (WIP) limits to reduce batch sizes, visual displays, and production feedback
  • Development should involve comprehensive version control, a loosely-coupled architecture, Trunk Based Development, and Continuous Integration
  • Testing should include developer-driven automated tests, tester-driven exploratory testing, and self-service test data

These practices have been validated in Accelerate as statistically significant predictors of Continuous Delivery. A loosely-coupled architecture is the most important, with Dr. Forsgren et al stating “high performance is possible with all kinds of systems, provided that systems – and the teams that build and maintain them – are loosely coupled”.

Design rules for loose coupling

Team and application  architectures aligned with Conway’s Law enable applications to be deployed and tested independently, even as the number of teams and applications in an organisation increases. An application should represent a Bounded Context, and be an independently deployable unit.

The reliability level of an application cannot exceed the lowest reliability level of its hard dependencies. In particular, the reliability of an application in a lower demand group may be limited by an on-premise runtime environment. Therefore, a Policy Rule should be introduced to reduce coupling between applications, particularly those in different demand groups.

Data should be stored in the same demand group as its consumers, with an asynchronous push if it continues to be mastered in a lower demand group. Interactions between applications should be protected with stability patterns such as Circuit Breakers and Bulkheads. This will allow teams to shift from Optimising For Robustness to Optimising For Resilience, and achieve new levels of reliability.

Example – MediaTech

At MediaTech, there is a commitment to re-architecting video game dataflows. An asynchronous data push is built from videogames-data to a new videogames-details service, which transforms the data format and stores it in a cloud-based database. When this is used by videogames-ui, a reliability level of 99.9% is achieved. Reducing requests into the MediaTech data centre also improves videogames-ui latency and videogames-data responsiveness.

Unlock testing guidelines

Reducing coupling between applications in different demand groups also allows for context-free Policy Rules to be replaced with context-rich Policy Guidelines. Re-designing a policy previously inherited from a lower demand group can eliminate constraints in a high demand group, and result in dramatic improvements in delivery flow. A Policy Rule that all applications must do End-To-End Testing can be replaced with a Policy Guideline that high demand applications do Contract Testing, while lower demand applications continue to do End-To-End Testing. Such a Policy Guideline could be revisited later on for lower demand applications unable to meet their own throughput target.

At MediaTech, the End-To-End Testing between videogames-ui and videogames-data is stopped. Website Services teams take on more testing responsibilities, with Contract Testing used for the videogames-data asychronous data push. Eliminating testing handoffs increases videogames-ui deployment frequency to every 10 days, but every 7 days remains unattainable due to operational handoffs.

This is part 4 of the Strategising for Continuous Delivery series

  1. Strategising for Continuous Delivery
  2. The Bimodal Delusion
  3. Multi-Demand IT
  4. Multi-Demand Architecture
  5. Multi-Demand Operations

Acknowledgements

Thanks to Thierry de Pauw for reviewing this series.

Multi-Demand IT

What is Multi-Demand IT? How does it provide the means to drive a Continuous Delivery programme with incremental investments, according to product demand?

This is Part 3 of the Strategising for Continuous Delivery series

Introduction

Multi-Demand IT is a transformation strategy that recommends investing in groups of technology value streams, according to their product demand. While Bimodal IT recommends upfront, capital investments based on an architectural division of applications, Multi-Demand favours gradual investments in Continuous Delivery across an IT estate based on product Cost of Delay.

A technology value stream is a sequence of activities that converts product ideas into value-adding changes. A demand group is a set of applications in one or more technology value streams, with a shared throughput target that must be met for Continuous Delivery to be achieved. There may also be individual reliability targets for applications within a group, based on their criticality levels.

Uncover demand groups

An IT department should have at least three demand groups representing high, medium, and low throughput targets. This links to Dr. Nicole Forsgren’s research in The Role of Continuous Delivery in IT and Organizational Performance, and Simon Wardley’s Pioneers, Settlers, and Town Planners model in The Only Structure You’ll Ever Need. Additional demand groups representing very high and very low throughput targets may emerge over time. Talented, motivated people are needed to implement Continuous Delivery within the unique context of each demand group.

Multi-Demand creates a Continuous Delivery investment language. Demand groups make it easier to prioritise which applications are in a state of Discontinuous Delivery, and need urgent improvement. The aim is to incrementally invest until Continuous Delivery is achieved for all applications in a demand group.

Applications will rarely move between demand groups. If market disruption or upstream dependents cause a surge in product demand, a rip and replace migration will likely be required as a higher demand group will have its own practices, processes, and tools. When product demand has been filled for an application, its deployment target is adjusted for a long tail of low investment. The new deployment target will retain the same lead time as before, with a lower interval. This ensures the application remains launchable on demand.

A high or medium demand group should contain a single technology value stream. This means all applications with similar demand undergo the same activities and tasks. This reduces cognitive load for teams, and ensures all applications will benefit from a single flow efficiency gain. A low demand group is more likely to have multiple technology value streams, especially if some of its applications are part of a legacy estate.

Example – MediaTech

Assume MediaTech adopts Multi-Demand for its IT transformation. There is a concerted effort to assess technology value streams, and forecast product demand. As a result, the following demand groups are created:

videogames-ui is in the sole Website Services technology value stream, while videogames-data is in one of the Heritage Applications technology value streams.

Create Multi-Demand policies

A demand group will have a policy set which determines its practices, processes, and tools. Inspired by Cynefin, a policy can be a:

  • Policy Fix: single group, such as heightened permissions for teams in a specific group
  • Policy Rule: multi-group single implementation, such as mandatory use of a central incident management system for all groups
  • Policy Guideline: multi-group multi-implementation, such as mandatory test automation with different techniques in each group

A policy will shape one or more activities and tasks within a technology value stream. Each demand group should have a minimal set of policies, as Little’s Law dictates the higher the throughput target, the fewer activities and tasks must exist. Furthermore, applying the Theory Of Constraints to Continuous Delivery shows throughput in a technology value stream will likely be constrained by the impact of a single policy on a single activity.

At MediaTech, the Multi-Demand lens shows videogames-data is in a state of Continuous Delivery while videogames-ui is in Discontinuous Delivery. This is due to the inheritance of End-To-End Testing, CAB meetings, and central production support policies from Heritage Apps, which has lower product demand and a very different context.

Policy Rules should be treated with caution, as they ignore the context and throughput target of a particular demand group. A Policy Rule can easily incur handoffs and rework that constrain throughput in a high demand group, even if it has existed for lower demand groups for years. This can be resolved by turning a Policy Rule into a Policy Guideline, and re-designing an activity per-demand. For example, End-To-End Testing might be in widespread use for all medium and low demand applications. It will likely need to be replaced with Contract Testing or similar with high demand applications.

This is Part 3 of the Strategising for Continuous Delivery series

  1. Strategising for Continuous Delivery
  2. The Bimodal Delusion
  3. Multi-Demand IT
  4. Multi-Demand Architecture
  5. Multi-Demand Operations

Acknowledgements

Thanks to Thierry de Pauw for reviewing this series.

The Bimodal delusion

Why is Bimodal IT so fundamentally flawed? Why is it just a rehash of brownfield versus greenfield IT? What is the delusion that underpins it?

This is Part 2 of the Strategising for Continuous Delivery series

Introduction

Bimodal IT is a notoriously bad method of IT transformation. In 2014, Simon Mingay and Mary Mesaglio of Gartner recommended in How to Be Digitally Agile Without Making a Mess that organisations split their IT departments in two. The authors proposed a Mode 1 for predictability and stability of traditional backend applications, and a Mode 2 for exploration and speed of digital frontend services. They argued this would allow an IT department to protect high risk, low change systems of record, while experimenting with low risk, high change systems of engagement.

Example – MediaTech

For example, a MediaTech organisation has an on-premise application estate with separate development, testing, and operations teams. Product stakeholders demand an improvement from monthly to weekly deployments and from 99.0% to 99.9% reliability, so a commitment is made to Bimodal. Existing teams continue to work in the Mode 1 on-premise estate, while new teams of developers and testers start on Mode 2 cloud-based microservices.

This includes a Mode 2 videogames-ui team, who work on a new frontend that synchronously pulls data from a Mode 1 videogames-data backend application.

Money for old rope

Bimodal is a transformation strategy framed around technology-centric choices, that recommends capital investment in systems of engagement only. It is understandable why these choices might appeal to IT executives responsible for large, mixed estates of applications. Saying Continuous Delivery is only for digital frontend services can be a rich source of confirmation bias for people accustomed to modernisation failures.

However, the truth is Bimodal is just money for old rope. The Bimodal division between Mode 1 and Mode 2 is the same brownfield versus greenfield dichotomy that has existed since the Dawn Of Computer Time. Bimodal has the exact same problems:

  • Mode 1 teams will find it hard to recruit and retain talented people
  • Mode 1 teams will trap the domain knowledge needed by Mode 2 teams
  • Mode 2 teams will depend on Mode 1 teams
  • Mode 2 services will depend on Mode 1 applications

The dependency problems are critical. Bimodal architecture is predicated on frontend services distinct from backend applications, yet the former will inevitably be coupled to the latter. A Mode 2 service will have a faster development speed than its Mode 1 dependencies, but its deployment throughput will be constrained by inherited Mode 1 practices such as End-To-End Testing and heavyweight change management. Furthermore, the reliability of a Mode 2 service can only equal its least reliable Mode 1 dependency.

At MediaTech, the videogames-ui team are beset by problems:

  • Any business logic change in videogames-ui requires End-To-End Testing with videogames-data
  • Any failure in videogames-data prevents customer purchases in videogames-ui
  • Mode 1 change management practices still apply, including CABs and change freezes
  • Mode 1 operational practices still apply, such as a separate operations team and detailed handover plans pre-release

As a result, the videogames-ui team are only able to achieve fortnightly deployments and 99.0% reliability, much to the dissatisfaction of their product manager.

The delusion

This is the Bimodal delusion – that stability and speed are a zero-sum game. As Jez Humble explains in The Flaw at the Heart of Bimodal IT, “Gartner’s model rests on a false assumption that is still pervasive in our industry: that we must trade off responsiveness against reliability”. Peer-reviewed academic research by Dr. Nicole Forsgren et al such as The Role of Continuous Delivery in IT and Organizational Performance has proven this to be categorically false. Increasing deployment frequency does not need to have a negative impact on costs, quality, or reliability.

This is Part 2 of the Strategising for Continuous Delivery series

  1. Strategising for Continuous Delivery
  2. The Bimodal Delusion
  3. Multi-Demand IT
  4. Multi-Demand Architecture
  5. Multi-Demand Operations

Acknowledgements

Thanks to Thierry de Pauw for reviewing this series.

Strategising for Continuous Delivery

What strategy should an IT department adopt to incrementally and iteratively transform itself? How should different methods of software development, testing, and operations be managed at the same time?

Introduction

Organisations that wish to remain competitive in the years to come must explore new offerings, expand successful differentiators, and exploit established products. A 21st century, digital first organisational model of IT As A Business Differentiator focussed on cloud computing, smart mobile devices, and big data analytics is required.

This is tremendously difficult when an organisation has the 20th century, pre-Internet IT As A Cost Centre organisational model. This refers to disparate Product and IT departments, in which IT is a cost centre with fixed scope, fixed resource, and fixed deadline projects. Segregated development, testing, and operations teams mired in long-term Discontinuous Delivery are ill-equipped to rapidly build and run products.

Strategy is choice

An organisation-wide transformation from IT As A Cost Centre to IT As A Business Differentiator is an arduous, multi-year journey. It is hard to know where to invest in Continuous Delivery when an organisation has a large, mixed estate of well-established on-premise applications and emergent cloud applications. Such applications can act as significant revenue generators, regardless of deployment frequency and runtime environment. This means visionary leadership and a sense of urgency are required from IT executives, and a Continuous Delivery strategy.

A strategy is not a vision, a plan, a list of best practices, or an affirmation of the status quo. A strategy is a unified set of choices including a desire to succeed, a declaration on where to succeed, and a statement of how to succeed. It is a commitment to hard choices, amongst options with asymmetric value.

An effective Continuous Delivery strategy can be used to establish a culture of Continuous Improvement, powered by the Improvement Kata. It allows a programme of radical, far-reaching changes to be built around the choices made. This helps people to understand, and be motivated by changes to how teams work, how they interact, and the processes and tools they use.

On that basis, how can an IT department devise a Continuous Delivery strategy? How can it incrementally and iteratively transform itself?

This is part 1 of the Strategising for Continuous Delivery series

  1. Strategising for Continuous Delivery
  2. The Bimodal Delusion
  3. Multi-Demand IT
  4. Multi-Demand Architecture
  5. Multi-Demand Operations

Acknowledgements

Thanks to Thierry de Pauw for reviewing this series.

Build The Right Thing and Build The Thing Right

Should an organisation in peril start its journey towards IT enabled growth by investing in IT delivery first, or product development? Should it Build The Thing Right with Continuous Delivery first, or Build The Right Thing with Lean Product Development?

Introduction

The software revolution has caused a profound economic and technological shift in society. To remain competitive, organisations must rapidly explore new product offerings as well as exploiting established products. New ideas must be continuously validated and refined with customers, if product/market fit and repeatable sales are to be found.

That is extremely difficult when an organisation has the 20th century, pre-Internet IT As A Cost Centre organisational model. There will be a functionally segregated IT department, that is accounted for as a cost centre that can only incur costs. There will be an annual plan of projects, each with its scope, resources, and deadlines fixed in advance. IT delivery teams will be stuck in long-term Discontinuous Delivery, and IT executives will only be incentivised to meet deadlines and reduce costs.

If product experimentation is not possible and product/market fit for established products declines, overall profitability will suffer. What should be the first move of an organisation in such perilous conditions? Should it invest first in Lean Product Development to Build The Right Thing, and uncover new product offerings as quickly as possible? Or should it invest in Continuous Delivery to Build The Thing Right first, and create a powerful growth engine for locating future product/market fit?

Build The Thing Right first

In 2007, David Shpilberg et al published a survey of ~500 executives in Avoiding the Alignment Trap in IT. 74% of respondents reported an under-performing, undervalued IT department, unaligned with business objectives. 15% of respondents had an effective IT department, with variable business alignment, below average IT spending, and above average sales growth. Finally, 11% were in a so-called Alignment Trap, with negative sales growth despite above average IT spending and tight business alignment.

Avoiding the alignment trap in IT

Avoiding the Alignment Trap in IT (Shpilberg et al) – source praqma.com

Shpilberg et al report “general ineffectiveness at bringing projects in on time and on the dollar, and ineffectiveness with the added complication of alignment to an important business objective”. The authors argue organisations that prematurely align IT with business objectives will fall into an Alignment Trap. They conclude organisations should build a highly effective IT department first, and then align IT projects to business objectives. In other words, Build The Thing Right before trying to Build The Right Thing.

The conclusions of Avoiding the Alignment Trap in IT are naive, because they ignore the implications of IT As A Cost Centre. An organisation with ineffective IT will undoubtedly suffer if it increases business alignment and investment in IT as is. However, that is a consequence of functionally siloed product and IT departments, and the antiquated project delivery model tied to IT As A Cost Centre. Projects will run as a Large Batch Death Spiral for months without customer feedback, and invariably result in a dreadful waste of time and money. When Shpilberg et al define success as “delivered projects with promised functionality, timing, and cost”, they are measuring manipulable project outputs, rather than customer outcomes linked to profitability.

Build The Right Thing first

It is hard to Build The Right Thing without first learning to Build The Thing Right, but it is possible. If flow is improved through co-dependent product and IT teams, the negative consequences of IT As A Cost Centre can be reduced. New revenue streams can be unlocked and profitability increased, before a full Continuous Delivery programme can be completed.

An organisation will have a number of value streams to convert ideas into product offerings. Each value stream will have a Fuzzy Front End of product and development activities, and a technology value stream of build, testing, and operational activities. The time from ideation to customer is known as cycle time.

Flow in a value stream can be improved by:

  • understanding the flow of ideas
  • reducing batch sizes
  • quantifying value and urgency

The flow of ideas can be understood by conducting a Value Stream Mapping with senior product and IT stakeholders. Visualising the activities and teams required for ideas to reach customers will identify an approximate cycle time, and the sources of waste that delay feedback. A Value Stream Mapping will usually uncover a shocking amount of rework and queue times, with the majority in the Fuzzy Front End.

For example, a Value Stream Mapping might reveal a 10 month cycle time, with 8 months spent on ideas bundled up as projects in the Fuzzy Front End, and 2 months spent on IT in the technology value stream. Starting out with Build The Thing Right would only tackle 20% of cycle time.

Reducing batch sizes means unbundling projects into separate features, reducing the size of features, and using Work In Process (WIP) Limits across product and IT teams. Little’s Law guarantees distilling projects into small, per-feature deliverables and restricting in-flight work will result in shorter cycle times. In Lean Enterprise, Jez Humble, Joanne Molesky, and Barry O’Reilly describe reducing batch sizes as “the most important factor in systemically increasing flow and reducing variability”.

Quantifying value and urgency means working with product stakeholders to estimate the Cost Of Delay of each feature. Cost Of Delay is the economic benefit a feature could generate over time, if it was available immediately. Considering how time will affect how a feature might increase revenue, protect revenue, reduce costs, and/or avoid costs is extremely powerful. It encourages product teams to reduce cycle time by shrinking batch sizes and eliminating Fuzzy Front End activities. It uncovers shared assumptions and enables better trade-off decisions. It creates a shared sense of urgency for product and IT teams to quickly deliver high value features. As Don Reinertsen says in the The Principles of Product Development Flow, “if you only measure one thing, measure the Cost Of Delay”.

For example, a manual customer registration task generates £100Kpa of revenue, and is performed by one £50Kpa employee. The economic benefit of automating that task could be calculated as £100Kpa of revenue protection and £50Kpa of cost reduction, so £5.4Kpw is lost while the feature does not exist. If there is a feature with a Cost Of Delay greater than £5.4Kpw, the manual task should remain.

Build The Right Thing case study – Maersk Line

Black Swan Farming – Maersk Line by Joshua Arnold and Özlem Yüce demonstrates how an organisation can successfully Build The Right Thing first, by understanding the flow of ideas, reducing batch sizes, and quantifying value and urgency. In 2010, Maersk Line IT was a £60M cost centre with 20 outsourced development teams. Between 2008 and 2010 the median cycle time in all value streams was 150 days. 62% of ~3000 requirements in progress were stuck in Fuzzy Front End analysis.

Arnold and Yüce were asked to deliver more value, flow, and quality for a global booking system with a median cycle time of 208 days, and quarterly production releases in IT. They mapped the value stream, shrank features down to the smallest unit of deliverable value, and introduced Cost Of Delay into product teams alongside other Lean practices.

After 9 months, improvements in Fuzzy Front End processes resulted in a 48% reduction in median cycle time to 108 days, an 88% reduction in defect count, and increased customer satisfaction. Furthermore, using Cost Of Delay uncovered 25% of requirements were a thousand times more valuable than the alternatives, which led to a per-feature return on investment six times higher than the Maersk Line IT average. By applying the same Lean principles behind Continuous Delivery to product development prior to additional IT investment, Arnold and Yüce achieved spectacular results.

Build The Right Thing and Build The Thing Right

If an organisation in peril tries to Build The Right Thing first, it risks searching for product/market fit without the benefits of fast customer feedback. If it tries to Build The Thing Right first, it risks spending time and money on Continuous Delivery without any tangible business benefits.

An organisation should instead aim to Build The Right Thing and Build The Thing Right from the outset. A co-evolution of product development and IT delivery capabilities is necessary, if an organisation is to achieve the necessary profitability to thrive in a competitive market.

This approach is validated by Dr. Nicole Forsgren et al in Accelerate. Whereas Avoiding The Alignment Trap In IT was a one-off assessment of business alignment in IT As A Cost Centre, Accelerate is a multi-year, scientifically rigorous study of thousands of organisations worldwide. Interestingly, Lean product development is modelled as understanding the flow of work, reducing batch sizes, incorporating customer feedback, and team empowerment. The data shows:

  • Continuous Delivery and Lean product development both predict superior software delivery performance and organisational culture
  • Software delivery performance and organisational culture both predict superior organisational performance in terms of profitability, productivity, and market share
  • Software delivery performance predicts Lean product development

On the reciprocal relationship between software delivery performance and Lean product development, Dr. Nicole Forsgren et al conclude “the virtuous cycle of increased delivery performance and Lean product management practices drives better outcomes for your organisation”.

Exapting product development and technology

An organisational ambition to Build The Right Thing and Build The Thing Right needs to start with the executive team. Executives need to recognise an inability to create new offerings or protect established products equates to mortal peril. They need to share a vision of success with the organisation that articulates the current crisis, describes a state of future prosperity, and injects urgency into day-to-day work.

The executive team should introduce the Improvement Kata into all levels of the organisation. The Improvement Kata encourages problem solving via experimentation, to proceed towards a goal in iterative, incremental steps amid ambiguities, uncertainties, and difficulties. It is the most effective way to manage a gradual co-emergence of Lean Product Development and Continuous Delivery.

Experimentation with organisational change should include a transition from IT As A Cost Centre to IT As A Business Differentiator. This means technology staff moving from the IT department to work in long-lived, outcome-oriented teams in one or more product departments, which are accounted for as profit centres and responsible for their own investment decisions. One way to do this is to create a Digital department of co-located product and technology staff, with shared incentives to create new product offerings. Handoffs and activities in value streams will be dramatically reduced, resulting in much faster cycle times and tighter customer feedback loops.

Instead of an annual budget and a set of fixed scope projects, there needs to be a rolling budget that funds a rolling plan linked to desired outcomes and strategic business objectives. The scope, resources, and deadlines of the rolling plan should be constantly refined by validated learnings from customers, as delivery teams run experiments to find problem/solution fit and problem/market for a particular business capability.

Those delivery teams should be cross-functional, with all the necessary personnel and skills to apply Design Thinking and Lean principles to problem solving. This should include understanding the flow of ideas, reducing batch sizes, and the quantifying value and urgency. As Lean Product Development and Continuous Delivery capabilities gradually emerge, it will become much easier to innovate with new product offerings and enhance established products.

It might take months or years of investment, experimentation, and disruption before an organisation has adopted Lean Product Development and Continuous Delivery across all its value streams. It is important to protect delivery expectations and staff welfare by making changes one value stream at a time, by looking for existing products or new product offerings stuck in Discontinuous Delivery.

Acknowledgements

Thanks to Emily Bache, Ozlem Yuce, and Thierry de Pauw for reviewing this article.

Further Reading

  1. Lean Software Development by Mary and Tom Poppendieck
  2. Designing Delivery by Jeff Sussna
  3. The Essential Drucker by Peter Drucker
  4. Measuring Continuous Delivery by Steve Smith
  5. The Cost Centre Trap by Mary Poppendieck
  6. Making Work Visible by Dominica DeGrandis

IT as a Business Differentiator

How can Continuous Delivery power innovation in an organisation?

When an organisation is in a state of Continuous Delivery, its technology strategy can be described as IT as a Business Differentiator. IT staff will work in one or more product departments, which are accounted for as profit centres in which profits are generated from incoming revenues and outgoing costs. A profit centre provides services to customers, and is responsible for its own investment decisions and profitability.

IT as a Business Differentiator promotes IT to be a front office function. There will be a rolling budget, and a rolling plan consisting of dynamic product areas with scope, resources, and deadlines constantly refined by feedback. Long-lived, outcome-oriented delivery teams will implement experiments to find product/market fit for a particular business capability.

This is in direct contrast to Nicholas Carr’s 2003 proclamation that IT Doesn’t Matter, to which history has not been kind. Carr failed to predict the rise of Agile Development, Lean Product Development, and in particular Cloud Computing, which has commoditised many lower-order technology functions. These advancements have contributed to the ongoing software revolution termed “Software Is Eating The World” by Marc Andreessen in 2011, which has caused a profound economic and technological shift in society.

Continuous Delivery as the norm

IT as a Business Differentiator is an Internet-inspired, 21st century technology strategy in which IT contributes to uncovering new revenue streams that increase overall profitability for an organisation. This means executives and managers are incentivised to maximise revenue generating activities, as well as controlling cost generating activities.

Continuous Delivery is table stakes for IT as a Business Differentiator, as IT executives and managers are accountable for delays between ideation and customer launch. There will be an ongoing investment in technology and organisational change, to ensure deployment throughput meets market demand. There will be a focus on optimising flow by eliminating handoffs, reducing work in progress, and removing wasteful activities. The reliability strategy will be to Optimise For Resilience, in order to minimise failure response time and blast radius.

IT as a Business Differentiator and Continuous Delivery were validated by Dr. Nicole Forsgren et al, in the 2018 book Accelerate. Surveys of 23,000 people working at 2,000 organisations worldwide revealed:

  • Continuous Delivery results in high performance IT
  • High performance IT leads to simultaneous improvements in the stability and throughput of IT delivery, without trade-offs
  • High performance IT means an organisation is twice as likely to exceed profitability, market share, and productivity goals
  • Continuous Delivery also results in less rework, an improved organisational culture, reduced team burnout, and increased job satisfaction

Leaving IT As A Cost Centre

If an organisation has institutionalised IT as a Cost Centre as its technology strategy, moving to IT as a Business Differentiator would be difficult. It would require an executive-level decision, in one of the following scenarios:

  • Competition – rival organisations are increasing their market share
  • Cognition – IT is recognised as the engine of future business growth
  • Catastrophe – a serious IT failure has an enormously negative financial impact

If the executive leadership of the organisation agree there is an existential crisis, they should publicly commit to IT as a Business Differentiator. That should include an ambitious vision of success that explains the current crisis, describes a state of future economic prosperity, and injects a sense of urgency into the day-to-day work of personnel.

There is no recipe for moving from IT as a Cost Centre to IT as a Business Differentiator. As a complex, adaptive system, an organisation will have a dispositional state of time-dependent possibilities, rather than linear cause and effect. A continuous improvement method such as the Improvement Kata should be used to experiment with different changes. Experiments could include:

  • co-locating IT delivery teams with their product stakeholders
  • removing cost accounting metrics from IT executive incentives
  • creating a Digital department of product and IT staff, as a profit centre

This leaves the open question of whether an IT department should adopt Continuous Delivery before, during, or after a move from IT as a Cost Centre to IT as a Business Differentiator.

Further Reading

  1. The Principles Of Product Development Flow by Don Reinertsen
  2. Measuring Continuous Delivery by the author
  3. Lean Enterprise by Jez Humble, Joanne Molesky, and Barry O’Reilly
  4. Utility vs. Strategic Dichotomy by Martin Fowler
  5. Products Not Projects by Sriram Narayan

Acknowledgements

Thanks to Thierry de Pauw for his feedback.

IT as a Cost Centre

Why does Continuous Delivery encounter resistance from IT executives and managers in so many organisations, and why is it so difficult to implement? Why does IT as a Cost Centre results in long-term Discontinuous Delivery?

Introduction

When an organisation is in a state of Discontinuous Delivery, its organisational model can usually be described as IT as a Cost Centre. There will be a functionally segregated IT department, that is accounted for as a cost centre in which costs can be incurred but profits cannot be generated. The IT department will provide services to product departments, which are accounted for as profit centres responsible for investment decisions and profitability.

IT as a Cost Centre relegates IT to a back office function. Each year, the IT department will be allocated a fixed budget to deliver a set of projects required by the product departments. Each project will represent a large batch of pre-agreed requirements. Short-lived project teams will try to deliver those requirements with scope, resources, and deadlines all fixed in advance.

This dovetails with the traditional view of IT as a universal commodity, popularised by Nicholas Carr in IT Doesn’t Matter in 2003. Carr argued IT was merely an infrastructural technology, and concluded it was merely a cost of doing business that could not provide a sustained competitive advantage.

Cost accounting suzerainty

IT as a Cost Centre is a pre-Internet organisational model from the mid-20th century, that still persists today. The world is now reliant on, and connected by technology, yet in a 2014 survey of 700+ organisations by CIO 48% of organisations still had IT as a cost centre.

In The Cost Centre Trap, Mary Poppendieck traces the popularity of IT cost centres back to the ubiquity of cost accounting. Poppendieck describes how the performance of an IT cost centre is measured solely in terms of cost management. This means accounting metrics percolate into the performance metrics of IT executives and managers, creating a culture of cost control with little regard for product development or organisational performance. These incentives will be markedly different to those of product executives and managers, and usually contribute to an adversarial relationship between product departments and IT.

In cost accounting, inventory is tracked as an asset, maximum resource utilisation is encouraged, and development work is capitalised until production launch. These factors create a hidden bias towards the institutionalisation of large projects, due to their perceived economies of scale. In reality, the project delivery model is an ineffective, inefficient vehicle that impedes value, quality, and flow.

Discontinuous Delivery as the norm

A Continuous Delivery programme is an unending journey of continuous improvement, that requires a substantial investment in order to achieve a time to market that can improve product/market fit and increase revenues.

This is likely to be a hard sell in an organisation with IT as a Cost Centre. IT executives and managers will be incentivised to reduce costs wherever possible, while delivering projects that are supposedly on time, on scope, and on budget. As a result, there will be resistance to the idea of spending money on an internal programme with an explicit goal of improving organisational performance and no fixed end date.

Delivery teams will be short-lived, which encourages people to prioritise short-term feature work over long-term architectural work, which restricts the deployability and testability of different services. The reliability strategy will be to Optimise For Robustness, which increases lead times and failure blast radius. Furthermore, the lack of a mandate beyond development work will make it difficult to work with operations teams to establish consistent toolchains for deployments, logging, monitoring, and alerting.

In short, Dr. Eli Goldratt was right when he said in The Goal “if it comes from cost accounting, it must be wrong”.

Further Reading

  1. The Principles Of Product Development Flow by Don Reinertsen
  2. Measuring Continuous Delivery by the author
  3. Lean Enterprise by Jez Humble, Joanne Molesky, and Barry O’Reilly
  4. Utility vs. Strategic Dichotomy by Martin Fowler
  5. No Projects by the author

Acknowledgements

Thanks to Thierry de Pauw for his feedback

© 2024 Steve Smith

Theme by Anders NorénUp ↑