Industrial devops, p.25
Industrial DevOps, page 25
Figure 13.2: Schedule Grid
Decouple Scope from Time
A typical schedule tightly couples scope and time together. Epics and features are the scope needed to fulfill requirements that have been relatively sized. Time is years, half years, and quarters. We recommend identifying epics and features with key acceptance criteria. Then place the scope, such as epics and features, into time such as quarters and half years. Once we have placed scope in time, then we link predecessor or successor dependencies. The benefit of decoupling scope from time allows us to easily move scope around based on priorities.
Level of Detail in the Schedule
When reviewing the schedule, collapse small tasks into bigger chunks of work We will place the smaller, detailed work in the backlog. If you have a number of these small tasks, we recommend that you migrate those smaller tasks to the product backlog and link them to higher-level tasks within the schedule using a unique identifier. The goal of this is to create flexibility with the smaller tasks while providing traceability and support dependency management at the higher level.
Length of Time Remaining in Schedule
Review the schedule for the total length of time. For a five-year schedule, the goal would be to decompose into the varied time horizons. For year one, decompose the schedule into features planned by quarter; the second year, we recommend decomposing epics planned into half years. For the remaining years, place high-level epics in the out-years. Adjust your approach to the length of the schedule.
Resource Loading Approach
Next, we want to evaluate how the schedule has been resource loaded. In a typical schedule, individual resources are assigned to tasks. One of the goals for Industrial DevOps is team-based collaboration and delivery, which requires a move from individual resource loading to team-based loading. The mechanism of team-based loading is creating a fictional cross-functional team with capacity and cost and assigning teams to the feature-level tasks.
Business Rhythm to Update with Empirical Data
One of the tenets of Industrial DevOps is moving from predictive planning to empirical planning. Given that we need to use empirical data to adjust the plans, we need business rhythms with an established cadence that intentionally reviews data and updates the plans. Typical programs may do this weekly or every other week, depending on the size of the program and the level of volatility.
Maximize Team Alignment
We want to obtain agreement for a cadence that the teams can synchronize on. Our goal is that we can optimize the delivery of the system by using a common cadence. When cadence is implemented in conjunction with iterative and incremental time cycles, we obtain the maximum possible feedback at the system level. Given the potential lead times for hardware, there are many teams who select a longer sprint cycle to accommodate integration of hardware and software items. When we talk about cross-functional teams, we may have hardware and electronics on one team if they are collaborating on a specific set of features or software and hardware on a team if the component requires frequent modifications, but we do not necessarily need every skill type on every team—only the skills associated with a specific product we are working on.
P3: Implement Data-Driven Decisions
Industrial DevOps wants to use quantitative data to measure progress or make decisions. Moving away from phase-gate milestones requires different approaches to obtain and track program progress. Leverage product demonstrations and user feedback to build confidence in your progress on products. The benefit of the demonstrations is that we get a real understanding of where the validated product stands. If we can not demonstrate working capability, we likely do not have a validated capability.
Quantitative data regarding the value of a capability allows us to move away from a gut feeling when we prioritize work. Although many of us trust our gut instincts, there is significant research that shows we are almost always incorrect. Limiting work in progress and ensuring flow requires prioritization of the work so we can collaborate across the team on working products.
STEP 5: Execute
P4: Architect for Change and Speed
As we have discussed throughout the book, to deliver products at speed, we need to have modular architecture with standardized interfaces. If you have a legacy program, you may have a monolithic architecture that is tightly coupled. For Industrial DevOps, to provide optimal results, we recommend using a pattern known in software engineering as the strangler pattern, which is a pattern to gradually modernize legacy systems. We begin by identifying strangulation points, which are areas of the system that we need to replace. We next create integration points to enable communication between old components and new components. As illustrated in Figure 13.3, we incrementally replace components in the new system with new modular components with well-defined interfaces. We need to iterate the system architecture with the organizational structures changes, which may take years, depending on the size and complexity of the system. With each update, we run a complete verification and validation of the change to ensure the system works as expected. If we consider our CubeSat example, we may need to decouple the attitude control system from the guidance, navigation, and control system so that we could move the monolith to microservices. If we were to consider performing something similar with hardware portions, this could be something as simple as creating an adapter for the CubeSat deployer while we try out a new design for the deployer connector.
Figure 13.3: Strangler Pattern in Action
P7: Find Opportunities to Integrate Earlier
We discussed the need to integrate as frequently as feasible. Factors to consider when scheduling integration points are system size, complexity, level of automation, available environments, and associated risk. The larger and more complex the system, the more time it takes to integrate updates, which may make it unfeasible, especially if we are starting with a highly coupled system.
To reduce integration time, it will be important to refactor the solution into smaller modules. In addition, if we are beginning with all-manual test approaches, integration takes much longer. Defining an automation strategy early is to reduce the time between integrating. In traditional development, many companies choose not to invest in expensive testing and integration environments until later in the development cycle. As we transition to Industrial DevOps, we need to strategize how to obtain environments as early as possible. Leveraging software-in-the-loop (SIL), hardware-in-the-loop (HIL), and modeling environments allows us to integrate the system faster, at least in the virtual space.
P5: Review Product Flow
Begin by documenting the current system flow. Identify items such as key milestones, and review dates as well as lead time of new features delivered to customers or cycle time to make changes in the system. Also, measure the length of time between feedback loops. Feedback measures could include the frequency of feedback from users, business owners, manufacturing, suppliers, or any other members of the value stream whose feedback helps validate the system. From here, we break the work down into smaller items and put together a cycle from which to iterate.
P8: Begin with Tests and Shift Left
Moving tests earlier in the life cycle reduces risk and lowers rework exponentially. Leveraging BDD approaches enables us to begin with executable requirements. Evaluating how the requirements will be verified and validated before building the system allows us to identify untestable requirements early. A key enabler for testing early is ensuring you have the tools to do the job. If we delay getting modeling and hardware-in-the-loop environments, we will be unable to run tests early. If we delay in developing a multi-tiered test automation strategy, frequent testing is unfeasible. When we integrate early testing with frequent integration, we have the ability to learn and deliver much faster.
STEP 6: Improve
P9: Use the Improvement Kata Model as a Tool for Improvement
The Toyota improvement kata is a problem-solving model, based on scientific experimentation, that individuals and teams use to iterate and improve toward a new, desired future state. Lean expert Mike Rother defines the improvement kata as “a skill-building process to shift our mindset and habits from a natural tendency to jump to conclusions, to a tendency to think and act more scientifically.”7
How does this fit with the current and future-state activities? As the organization moves toward the future state that it has defined, it will apply an improvement kata approach to continuously and iteratively improve the adoption of the Industrial DevOps principles. Coupled with the improvement kata is the concept of the coaching kata.
The idea is that leaders and managers are the coaches in this process. They lead by example and coach their teams through improvements to develop new skills. We also recommend employing coaches experienced in Industrial DevOps to assist leaders and the teams as they adopt and improve the practices associated with each of the principles. The improvement kata and coaching kata are two well-documented techniques from the Lean community that you can employ as you iterate, learn, and improve toward the next future state.
Figure 13.4: The Toyota Improvement Kata
Source: Mike Rother, Toyota Kata.
STEP 7: Define a Path for Digital Capabilities
Industrial DevOps principles consider the integration of people, processes, and tools. Many of these principles leverage digital capabilities that require both budget and investment to be realized. Create your digital capabilities road map based on the priorities and greatest return on investment (ROI) of your unique cyber-physical system. Ideas for your road map might include things such as:
•Ensure tools leveraged across the value stream are integrated through enabling architecture and APIs. Build a continuous integration and continuous delivery (CI/CD) pipeline early in your project and incrementally enhance the fidelity at each iteration.
•Capture personas, models, and scenarios to define and improve understanding of the system.
•Invest early in labs that contain simulators, emulators, hardware-in-the-loop, and 3D modeling.
•Use your 3D models to build augmented and virtual reality to further visualize system capabilities.
•Build digital shadows, which evolve into digital twins as more of the system is built out.
•Generate digital threads that integrate temporal data to provide increased insight.
•Leverage 3D printing/additive manufacturing to buy down risk earlier or use to build the physical product itself if cost-effective.
Relationships between the Principles of Industrial DevOps
This section outlines the principles of Industrial DevOps and demonstrates how each principle is connected with the other principles. It’s important to remember that the principles work together and should be looked at holistically to achieve the greatest benefits.
As you define your Industrial DevOps journey, consider each of the principles and how they are interconnected. For example, if you choose to focus first on shift-left integration and testing but are still working in functional silos versus organizing teams around value delivery, you will not get the results you are anticipating.
We have organized the principles into three primary categories:
1.Organization and Structure
2.Execution
3.Continuous Improvement
Table 13.2: Principles Focused on Organization and Structure
P1: Organize for the Flow of Value
Connection to Other Principles
P2: Apply Multiple Horizons of Planning
P4: Architect for Change and Speed
P7: Integrate Early and Often
How the Principles Work Together
(P2 and P7) Once the value stream is understood, teams are organized around the value stream. The product is defined from vision and high-level yearly plans, with detailed backlog definition happening closer to execution. At the implementation level, teams are cross-functional and develop the product through a series of short iterations with frequent feedback loops to regularly validate and verify the features while continuously integrating and testing throughout the iterative development cycle.
(P4) Consider how the system is architected and how the organization aligns the teams. The teams must be organized around the flow of value delivery. Define your value stream and the products that the value stream produces. Organize the people around the flow of value. Revisit the architecture of the system and ensure modularity and reduce dependencies. While the backlog defines new functionality, it must also include the work that needs to be done to continuously evolve the architecture.
P2: Apply Multiple Horizons of Planning
Connection to Other Principles
P6: Establish Cadence and Synchronization
How the Principles Work Together
Each of the multiple horizons of planning (P2) occurs on a regular cadence. Each horizon of planning yields empirical data demonstrating the success of the plan as it is executed and identifying necessary adjustments.
The observe-orient-decide-act (OODA) model is utilized in the military to quickly respond in changing and unpredictable environments. Just as in the OODA model, with each cycle, take what you observe and learn and feed that learning into the next cycle for planning and implementation.
P3: Implement Data-Driven Decisions
Connection to Other Principles
P2: Apply Multiple Horizons of Planning
How the Principles Work Together
The backlog is defined at multiple levels of decomposition (epic, feature, user story, task) and is planned within multiple horizons of planning. As backlog items get closer to implementation, they are further refined. Each item contains a description, who needs the capability, business benefit, and acceptance criteria. As backlog items are demonstrated, the objective evidence oft what is and is not working is fed into the next planning cycle.
P6: Establish Cadence and Synchronization
Connection to Other Principles
P7: Integrate Early and Often
How the Principles Work Together
We bring the understanding of cadence and synchronization with us into the cyber-physical world as product teams plan, develop, and deliver system capabilities. Together they define the standard of repeatable planning sessions, large-system integration, and demonstrations of integrated working capabilities. With large cyber-physical solutions, these demonstrations are implemented in a hybrid manner with a mixture of digital and physical artifacts.
As your organization defines the cadence and synchronization points, communicate the need for CI at the system level as early as your environment can enable. Teams that integrate and demonstrate functionality together will want to be on the same cadence so their synchronization points align.
Table 13.3: Principles Focused on Execution
P4: Architect for Change and Speed
Connection to Other Principles
P1: Organize for the Flow of Value
How the Principles Work Together
Intentional modular architecture with standardized interfaces reduces the number of dependencies, which enables the flow of value to stakeholders. The product backlog defines new business-facing functionality as well as enablers to evolve the architecture. Organize teams around the flow of value in concert with your architecture. One approach when considering our CubeSat example is having stream-aligned or complicated subsystem teams implementing capabilities such as attitude control, attitude determination, and attitude estimation and filtering while having a platform team build the infrastructure needed to run the software.
P5: Iterate, Manage Queues, Create Flow
Connection to Other Principles
P1: Organize for the Flow of Value
How the Principles Work Together
Teams are organized around value streams to improve the flow and delivery of value. This yields the benefits of iterative development and incremental delivery. While iterative development is often thought to be for software development only, this is not the case, as we have conveyed throughout this book. Embedded systems and hardware teams are now engaged in iterative development cycles.
Players in the space industry have demonstrated this advantage. For example, Rocket Lab has “demonstrated the ability to support rapid integration and short notice customer-driven changes in launch schedule, inclination, and launch site.”8
As you are getting started, organize your teams around the value stream, create product backlogs that align with your cross-functional Agile team structure, and set up the tool environment that enables iterative development across software and hardware with feedback loops in manufacturing and with the customer.
P7: Integrate Early and Often
Connection to Other Principles
P3: Implement Data-Driven Decisions
P5: Iterate, Manage Queues, Create Flow
How the Principles Work Together
The goal of integration is to ensure that all of the systems being developed can communicate and share data effectively. Integrating early and often is not only from a software perspective; it means integrating at the system level. Integrating early and often not only buys down risks while ensuring fit for purpose, but when coupled with iterative demonstrations, it provides real-time information and data about what is working or not working. Metrics are reviewed to understand the current system state. As we manage and improve flow, integrated tool sets and dashboards generate these metrics, which can be used to see where the bottlenecks are in the system. Build the system iteratively, integrate early and often, improve flow, and use data to understand the current state and to make decisions on the next steps to take in developing the solution or improving the flow.
