Evolving a Change Vision and Strategy for Platform Engineering
This case study provides an anonymized example of prior work for a Nerd/Noir client in the e-commerce sector. The scope of our engagement centered on a specific portfolio within the company, responsible for software services used by most of the other teams in the organization — a central dependency. Executive leadership decided to restructure their development process and practices to address the complexity of coordination between teams and emerging competitive forces in the market.
ESTABLISHING A CHANGE VISION: WHAT & WHY
A change vision establishes a baseline understanding of the opportunity, obstacles, and proposed countermeasures. It serves as a hypothesis for subsequent experiments that we will validate or invalidate as we learn through doing. It begins by diagnosing the most significant challenge the group faces in terms of impact:
Our product portfolio relies on a collection of services developed over many years. Building new features on top of them currently requires excessive coordination and effort. These services have become a bottleneck for the entire organization.
Common core system capabilities are at the heart of our ability to grow revenue and compete in the marketplace. Improving their usability is a business-critical effort essential to our long-term viability and competitiveness. Much of the system consists of legacy code using tools and languages that our development teams no longer have the skill sets to maintain or expand upon.
Then, the vision describes what the future looks like and what will need to change to get there:
Going forward, we will unify these services into a cohesive platform. The platform will enable reusable functionality stream-aligned teams can incorporate into their own services and applications to reduce dependency overhead within the organization.
This change is a joint effort between the product and engineering organizations. It will require a significant amount of collaboration, alignment, and compromise from leadership on both the product and engineering sides of the house, involving many tradeoffs and negotiations over an extended period of time.
STRATEGIES: HOW IN PRINCIPLE
Strategies provide a framework of principles for resolving conflicting priorities or approaches. These enabling constraints form a basis for decision-making across product and engineering organizations. They drive team and contributor enrollment in the desired change when supported by coaching from leaders and managers. We can ask, “Does what I’m doing align with our strategy?” When the answer is no, either the action is misaligned to a valid strategy or our strategies need additional tuning.
Modernize the platform to accelerate stream-aligned teams. We reduce the cognitive load of our stream-aligned teams. We will develop our system as a platform of discoverable APIs, front ends, and other components that stream-aligned teams can assemble to solve novel, domain-specific use cases. We will deprecate capabilities that no longer deliver value to the business.
Align the evolution of the platform with delivering new product opportunities. Our platform enables new product capabilities that unlock business value. Platform teams will develop these just in time for high-impact product initiatives. This represents a shift in how product and engineering prioritize and collaborate – decisions must balance product ambitions with platform investment.
Redistribute responsibility to local creators. Stream-aligned teams fully own the orchestrations and use cases specific to them. Platform teams provide reusable components to their customers, the stream-aligned teams. Communicating needs through system-level contracts such as APIs will replace expensive human-driven coordination.
Evolve collaboration patterns as the platform matures. Our journey begins with joint development – new platform teams working to create necessary system functionality with principal engineers and stream-aligned teams. We move towards a self-service platform that dramatically reduces coordination overhead and dependency management.
EXECUTION, LEARNING, EVOLUTION: HOW IN ACTION
A change vision specifies a desired set of goals but does not lay out the exact details of how to achieve them. Leaders and teams use the guiding principles to make coherent plans for implementing their strategies.
For this organization, changes can be grouped into three main stages:
Initial rollout includes the first changes undertaken by executive leadership, staff+ engineers, product managers, and team managers. These set up a framework for change management and contributor enrollment and include foundational changes that are difficult to reverse later on.
Early changes occurred at all levels of the organization, aligning team workflows to new processes and introducing new techniques and tools.
Ongoing steps reflect behaviors and processes that require longer times to adopt, feedback, and organizational learning, leading to improved processes and increased flow of value.
1. Initial Rollout
Conway’s Law observes that product and organizational topologies tends to mimic team communication structures. If the change vision differs significantly from the current state, the existing organizational structure might be an impediment. However, significant personnel moves and sudden shifts in roles, responsibilities, or reporting lines create a fog of uncertainty that can make change very difficult and put sustained progress at risk.
Team Organization: Leadership examined how much to restructure the teams based on the new core systems concept. There was an initial push to shuffle personnel amongst the teams based on various leadership coalition-related factors. However, the principle of team coherence was considered more important than specific domain expertise and ownership boundaries. This approach resulted in a smaller, more tactical first wave of employee moves, aimed at balancing skill sets and team sizes. As a result, the teams were stable enough to take on the more challenging demands of the transformation with minimal disruption and attrition due to change.
Understanding how products are envisioned, explored, created, and validated provides insight into bottlenecks, communication barriers, and learning or feedback loops. The organization’s current outcomes result from these processes, so changing outcomes necessitates altering the processes:
Product Development Life Cycle: Teams mapped their existing discovery and delivery workflow, finding buildups of work in progress led to long latencies, stale work, and delivery of non-value-added features. Using the map and historical data from previous product deliveries, they made initial changes to their discovery practices and delivery pipelines to more closely align the product and engineering teams.
Building organizational momentum requires clear communication of the strategy across the organization. Teams creating their action plans can generate enrollment, agency, and empowerment within the staff at every level—the governing constraints of the strategy act to keep those plans aligned.
Socializing the change: Leadership presented the plans for change as they were developed, keeping the organization informed with broad strokes at all-hands meetings and more details through smaller forums with presentations and discussions. Feedback was actively solicited from teams to promote team enrollment and discover obstacles to implementing the first process changes.
2. Early Changes
Aligning teams requires a shared perspective. Collaboration and negotiation are only viable when all parties have sufficient information to make informed trade-offs. Information radiators, standard metrics, and working agreements around value decisions foster the ability to shape the organization's actions to meet its current needs. Strategic constraints can empower teams to act locally in support of a global strategy.
Value Prioritization: Previously, product managers worked on their own initiatives with a single engineering team but often required skills or domain knowledge from other teams. There was no portfolio-level capability to prioritize the most valuable features, so the teams followed their PM’s lead, inevitably leading to scheduling conflicts.
The organization introduced a single common backlog spanning all teams. Product managers collaborated to establish a global priority for the shared feature-level backlog, as senior engineers provided design input for the platform architecture. In doing so, they could map risk and effort to inform priority decisions. The single point of truth for work enabled them to identify long-lead items, conduct early product discovery, detect issues in scheduling, and negotiate conflicting efforts. This led to a shorter cycle time for delivering features and less switching between multiple initiatives.
Reducing Delivery Size: Features had grown to the point where some had become “projects” that could last for months or quarters, so teams couldn’t safely or quickly pivot when new priorities arose.
Teams added constraints for story size and limited work in progress. They adopted techniques to refine work, splitting it into small and manageable chunks. Product management began evaluating promised versus delivered value in financial terms. Overall lead time for stories decreased significantly, as did the rollover of work between sprints. It became easier to change priorities, and the group attained literal agility.
Continuous Experimentation: Driven by continuous feature requests, the teams couldn’t make deliberate investments in their practices and cohesion. There was no investment at an organizational level in maintaining a high-level architecture. With smaller increments of delivery, it became possible to deliver more work. Senior engineers adopted experiments for platform evolution that targeted specific gaps in the current platform capabilities and ways for teams to add architectural rigor while developing existing features on the backlog. Most notably, they created new APIs to allow dependent teams to self-serve needs that had once been fulfilled through toilsome, manual requests. A platform takes shape.
3. Ongoing Steps
Implementing a strategy requires altering work processes, but only the broad strokes of process change can be visible at the outset. As organizational behavior shifts, secondary effects may manifest either due to old habits acting in a new process or because more immediate problems hide root causes. A crucial part of iterating on and refining strategies includes probing these new issues and adopting actions to address them. The organization uses empirical data to adjust its processes using established change management to perform double-loop learning:
Just-in-Time Focus: Initially, the common backlog helped us understand all the work in progress. However, existing commitments led product managers to start refining work with teams even though they had no capacity to begin working on the resulting stories. This issue was identified through cumulative flow metrics in the backlog. PMs continued to reduce the total number of features they focused on to reduce inventories of planned work. The development teams have retrospective items to improve delivery.
Process Evolution: The core management team held retrospective meetings at sprint boundaries to evaluate the effectiveness of the current process and create small countermeasure experiments to evolve their PDLC. The early meetings yielded many changes, leading to confusion about what experiments were in progress versus the established plan of record. Specific documentation of metrics to drive the adoption of changes, along with a WIP limit on changes, was used to make changes both impactful and more visible. The process stabilized considerably as teams became accustomed to new ways of working.
As the organization leans into new modes of work, their proficiency unlocks new opportunities. Things that were deemed unwieldy or impossible become tractable, and structured experimentation can test the boundaries of the feasible:
Team Coordination: As the organization worked on stories that required cross-domain expertise, the teams experimented with various methods to coordinate and share knowledge, thereby avoiding impediments caused by siloing. These have included whole-team pairings, “guest star” experts from other teams, and story splitting across well-defined boundaries. Coordination with teams in organizations outside the core platform remains challenging, as they need to be brought up to speed on the relevant processes and techniques.
The outcomes of the change vision typically manifest only as the overall emergent system behavior passes a threshold level of effort. “Slowly, then all at once.”
Legacy Architecture: Senior engineers continue to work towards retiring older, impossible-to-maintain codebases. Stories to stand up new services or place facades in front of older services have been added to the backlog and are actively refined by teams.
THE RESULTS
The actions aligned with the strategies described above yielded many measurable results.
500+ new products were created in 60 days via new APIs, replacing a laborious, ticket-driven process. In doing so, they also validated key architectural patterns used by their nascent platform.
A 48% increase in value delivered sprint over sprint. This was done by breaking months-long project deliveries into weeks-long value deployments, leading to greater agility in prioritizing work supporting innovation experiments.
>100% reduction in work rollover. Teams stopped starting and started finishing work in their two-week iterations. This capability lets major initiatives be dynamically reprioritized.
EVOLUTION: HOW STRATEGY MATURES WITH LEARNING
Realizing a change vision is hard work. Groups wishing to change must push past the desire to set a plan and execute that plan's steps. Making informed bets and implementing small yet rigorous experiments aligned with strategic principles that must be tested and refined requires constant involvement and a willingness to learn from failures.
Our approach to strategy is heavily inspired by the Observe-Orient-Decide-Act loop created by United States Air Force Colonel John Boyd. Specifically, we employ the strategy-focused adaptation of this loop adapted by Simon Wardley.
Some actions will achieve the goals of the change vision. Others might reveal new obstacles or challenges. Measuring results validates whether the strategically aligned actions are meaningfully advancing toward the vision or whether iterating the approach is warranted.
Continuous improvement, both gradual and step-wise, is a capability of a learning organization.