Software Lifecycle Integration Architecture

by Mik Kersten, March 25th, 2013

v0.5.0, March 25, 2013


Download the whitepaper

Overview

Software delivery is plagued by disjointed siloes of information spanning software stakeholders and suppliers. The tool diversity that is the norm of today’s Application Lifecycle Management (ALM) stack brought with it productivity benefits for stakeholders such as developers, who have influenced the selection of the tools that make them most productive. However, the stakeholder-specific selection of best-of-breed tools has also undermined the decade-old promise of ALM. The lifecycle stacks of today are more fragmented than they ever were.
Imagine an integrated fabric that allows information to flow freely and in real-time across the stakeholders, tool siloes and vendor boundaries that define software delivery. We call that vision Software Lifecycle Integration (SLI). When the application development stack became overly complex, the role of the Enterprise Architect became clear, and a new discipline and middleware tool category formed. Just as connecting the back-end services that power our organizations required the creation of an Enterprise Service Bus (ESB), connecting software delivery requires an Application Lifecycle Bus (ALB). In addition, for the SLI category to form, we need:

A role to be responsible for it, the Lifecycle Architect
A taxonomy of integration concepts, types and approaches
A common data model for SLI
A lifecycle bus architecture
Integration patterns

The goal of this whitepaper is to provide an initial version of the above for the emerging Lifecycle Architect role, to define a set of architectural concepts than the Lifecycle Architect can use to plan SLI deployments, and to bootstrap the conversation evolving around the SLI discipline.


Lifecycle tool diversity

Software integration is not new. In the 1990s, when application development stacks became more complex by combining application servers, legacy systems and databases, often from different vendors, Enterprise Architecture and Middleware came of age. A decade later the Enterprise Service Bus (ESB) became a core part of the enterprise stack. These technologies focused on the integration of application runtimes and of application data. With today’s rise of the API Economy, we are now seeing a resurgence of both the need to connect application and behavior in the cloud.
On the application and data integration layer, we’re in good shape. The trouble lies on the Application Lifecycle Management (ALM) part of software delivery, which is not about getting the application to run, but about developing and evolving the application over time. In ALM, we have seen a decade of promises of a connected lifecycle, with many a great vision outlined but never a realization of ALM that has lined up to customers’ needs. Where we are today is that organizations who have more than 500 developers have ALM stacks that are connected by brittle, manual, and combinatorially exploding point-to-point connections between vendors’ tools. This approach has not scaled and has now become the bottleneck on ALM adoption. Worse yet, the end-to-end software delivery cycle is only as productive as it’s bottleneck, and while Agile, mobile, cloud and open source are transforming the way developers build and deploy software, little has been done to address what organizations now see as the main bottleneck on software delivery, the disconnection and communication impediments in the software lifecycle.

Accidental vs. fundamental diversity

The root cause of the disconnected lifecycle is tool diversity. If an organization could procure the ultimate ALM suite from a single vendor, and meet the needs of every stakeholder, there would be no need for SLI. However, tool diversity is now the norm, not the exception. Diversity is also the sign of a healthy ecosystem. We have witnessed this in the coming of age of Agile as well as the new breed of developer tools, where competition, low-cost and open source entrants, and innovative vendors abound. The result is that two types of lifecycle tool diversity exist:

  • Accidental diversity: This includes all of the heterogeneity in the tool stack that does not support organizational needs. Tools inherited as a result of M&A activities, or independent selection of similarly functioned tools due to a lack of centralized governance, fall into this category. For example, an organization could have three bug trackers, one a 20 year old legacy tool created in house, another the new developer-favored issue tracker, and the third an open source issue tracker that resulted from an acquisition.
  • Fundamental diversity: All heterogeneity that adds value to the lifecycle by adding to the productivity of software delivery. For example, an organization developing a Ruby on Rails app may be using Redmine, while Java developers within that organization are using JIRA, and .NET developers TFS.

Benefits of tool diversity

Lifecycle tool diversity allows organizations to assemble best-of-breed stacks that meet the varying needs of stakeholders in the lifecycle, and that align to platform demands. Key benefits of fundamental diversity include.

Specialized tools: The various stakeholders of software delivery require different tools to be effective at their particular discipline. For example, developers need customizable tools integrated with their development environment. In contrast, business analysts may want browser-based tools with the latest user interface mockup capabilities.
Right-sized tools: Some tools have become specialized to organizational size. For example, a new Agile planning tool may be great for a dozen teams, but a traditional PPM tool is often needed for planning across very large numbers of teams. Similarly, a cutting edge lifecycle tool may be suitable for new development projects done by a small set of teams while enterprise-scale lifecycle tools remain the corporate standard for core products.
Platform affinity: Lifecycle tool vendors who provide a development platform often provide an ALM on-ramp onto that platform. For example, an PaaS platform can provide deployment automation functionality, while a mobile platform can provide app store integration. As such, an organization creating a hosted back end for a mobile client may end up with two sets of ALM tools, each optimized for the development platform experience.
Vendor diversity: Different vendors specialize at different stages of the lifecycle, and target different development styles. For example, some vendors specialize in supporting cutting-edge Agile trends, while others focus on tried and proven features. An organization may choose to have both deployed if the goal is to prove out a new flavor of Agile in a part of the organization, especially across different horizons of R&D.
Supplier diversity: As outsourcing and consumption of open source software increases, it becomes either impractical to expect software suppliers to use the same ALM tools as the sourcing organization. For example, open source projects tend to use open source ALM tools, while small suppliers tend to use lightweight issue trackers in place of the enterprise ALM tools needed for large scale software delivery.
Support for legacy: The cost and disruption moving away from a legacy system, such as an older tool or in-house defect tracker, can be overly high to take on during a lifecycle modernization. Allowing the legacy system to interoperate with the new systems enables a more incremental migration path.


Costs of tool diversity

A core challenge of software delivery has been that, without a suitable integration layer, the business benefits of diversity are thwarted by the costs of a disconnected lifecycle. These costs boil down to undermining the promise of ALM itself, as the benefits of ALM tools can only be realized with a sufficiently connected tool chain. Key problems caused by a heterogeneous stack that lacks integration include:

Lack of traceability: Due to information being siloed into disparate systems, it is impossible to get cross-system traceability. For example, business requirements are disconnected from user stories which a disconnected from source code. Once a support ticket is filed with a major defect affecting a particular line of source code, the affected requirement cannot be determined. The less traceability there is in software, the more expensive the software is to maintain.
Lack of visibility: Reports, dashboards and planning tools only have visibility into the repositories that they were built on. For example, reports in an ALM tool fail to see defects tracked in the Quality Management tool and dashboards connected to a data warehouse are missing data from the lifecycle tools that do not provide a direct integration. Lack of visibility makes it impossible to effectively manage a software portfolio.
Limited governance: Governance, workflows, or compliance processes cannot be implemented across tools. For example, a compliance workflow implemented in a centralized ALM tool is not enforced in a lightweight issue tracker used by a large portion of the organization’s developers. Without governance, software delivery cannot be adequately connected to other business processes.
Manual processes: Without an integration layer, end users often need to enter data into two tools, reducing end user productivity and affecting data accuracy, and rendering automation impossible. For example, creating requirements links between one tool and another, or entering time sheets into a PPM tool while the Agile tool is intended to be the definitive record of time tracked at a finer level of granularity. The lack of automation has individual stakeholders wasting time on manual entry, and impedes the deployment of lean methodologies.


Signs of lifecycle failure

Without an adequate integration layer, the following anti-patterns occur in the software lifecycle. Each of these contributes to the costs listed above by preventing the end-to-end flow of information across the lifecycle stack.

Manual processes: Duplicate data entry and any other manual entry becomes the bottleneck of the lifecycle. These include excessive status reporting meetings, or status updates via email and spreadsheets.
Batch jobs: Attempts and connecting the lifecycle that periodically reconcile data models across tools, but result in numerous conflicts and make cross-silo collaboration impossible.
Custom solutions: Integration provide by services vendors or internal teams that result in brittle, costly to maintain and upgrade with frequent releases of ALM tools, creating lock-in to old versions
Point-to-point integrations: Integrating two systems directly, with no abstraction in between, results in a brittle system that does not scale, resulting in batch jobs, conflicts, and other workarounds.

In the following section, we will examine the architecture goals and principles of a connected application lifecycle.

Software integration architecture

This section provides an overview of the SLI architectural concepts, including goals, the role of the Lifecycle Architect, and the integration types and data model.

Goals of integration

The goal of SLI is to help organizations create a lifecycle that seamlessly connects all of the disciplines, stakeholders and suppliers involved in software delivery. This enables the benefits of diversity while mitigating the problems. The benefits of an integrated software lifecycle include:

Insight: The ability to deploy metrics, reports, and lean methods across the lifecycle. For example, implementing a lean build-measure-learn loop, from idea to customer validation.
Traceability: End-to-end and cross-system linking of lifecycle artifacts. For example, all code, tests, branches are linked to business requirement, regardless of system heterogeneity.
Flexibility: The ability to deploy new lifecycle management systems, and to retain legacy systems while modernizing the stack. For example, if a new mobile platform is being adopted for a new application, the lifecycle tools specific to that platform can be used.
Flow: Real-time flow of social and status information ensures stakeholders work within their system of choice while being connected to collaborators in adjacent parts of the lifecycle. For example, as soon as a tester comments on a defect, requesting more details on steps to reproduce, the support engineering sees the comment within their system.
Governance: While end-users get to work in the systems that make them most productive, the organization needs cross-lifecycle governance and workflows to function across systems. For example, policies and workflows can be implemented to define the cross-repository status and record updates that a creation of a defect results in.


The role of the Lifecycle Architect

When the complexity of software applications and the enterprise systems that they depended on grew, middleware formed as that glue that could connect end users to systems and data. In this transition, the role of the Enterprise Architect (EA) emerged and became a key technical stakeholder in the delivery process. It’s now time to define the Software Lifecycle Architect role to own this role, and to coordinate the activity of mapping the business of process delivery onto the Software Lifecycle Architecture. This architect’s goal is to define the stakeholders, workflow model, feedback loop that spans the systems and stakeholders that make up the lifecycle.

Lifecycle integration types

SLI breaks down into the following four types of integration, delineating the different silo boundaries and layers within a typical software delivery organization. The applicability of integration types varies with the size and amount of software delivery being done by the organization. For example, a small organization should have little or no “within discipline” integration needs as developers should be using a single tool. But it may have a need for “cross lifecycle stage” integration if testers are using another tool, or “cross-organization” integration if testing is being outsourced.

Integration Type

Description

Example

Within discipline

Similar purposed tools, likely due to accidental diversity

Two Agile planning tools were selected independently by two lines of business, need to be connected.

Within lifecycle stage

Multiple tools provide different level of planning and management granularity

Developers are using a lightweight issue trackers, developer managers an Agile tool, and program managers a PPM tool. All report on the same data, originating in the issue tracker.

Cross lifecycle stage

Different tools are used by different stakeholders within the organization

Requirements in BA tool need to be connected to Epics in Agile development tool.

Cross-organization

Different tools are used by different organizations connected in a software supply chain

An organization outsources parts of its development to a small company using a lightweight issue tracker, but uses it’s own large scale Agile ALM tool for tracking delivery

Integration architectures

Since the software lifecycle is about connecting stakeholders, a distinction needs to be made between the tools used by practitioners and where the data is stored. This distinction originates from Geoffrey Moore’s discussion of Systems of Record vs. Systems of Engagement.

  • System of Record (SoR), aka Repository: Encapsulates all back-end services and data storage, typically exposed via API. For example, the database and REST APIs of an Agile planning tool.
  • System of Engagement (SoE), aka UI: End-user facing system that supports interaction and provides a user interface. For example the mobile client, browser-based UI, or desktop-based developer IDE.

Part of the challenge of SLI is that most of today’s lifecycle systems have a coupling between the SoR and SoE. In contrast, Microsoft Outlook is an SoE that’s decoupled from its SoRs via email protocols, such as Microsoft Exchange and IMAP.
All of the integration types listed above can be achieved by each of the following integration architecture types. However, each has a set of tradeoffs, and potentially tool-specific limitations. For example, an Integrated Development Environment (IDE) based integration connect developers to testers to provide “cross lifecycle stage” integration. However, it is limited to developers if testers do not use a federated SoE such as an IDE, but instead use an SoE specific to the testing SoR, such as the web UI or rich client of the testing tool.

Architecture Type

Description

Limitations

Database-to-database

Databases of SoRs are connected, mappings between database schemas. Eg, ETL solutions.

Business logic of lifecycle tools is present in the application layer, such as workflows, and as such the data layer is insufficient. N^2 complexity.

API-to-API

Data transformation from one across two systems. Eg, connection between two REST APIs.

No cross-system abstraction or data model. N^2 complexity.

Open standards

SoRs connect to each other over open protocols, as with email servers. Eg, POP, IMAP.

With some exceptions [RTC ref], today’s lifecycle SoEs are not built on open standards. As such, cross lifecycle data has to be integrated into those tools’ SoRs.

User Interface (UI)

An SoE federates data from multiple SoRs. Eg, Eclipse Mylyn.

Supports cross-tool collaboration, but requires users to adopt a new tool.

Common repository

Tools connect to a common SoR that contains all or key parts of the data for the entire lifecycle.

Requires deployment of a new “master” SoR. Fails to support diversity of SoRs.

Bus/hub

A common data model with event-based processing of changes, and propagation across SoRs.

Overhead of mapping between schemas of various SoRs.

These integration architectures are not mutually exclusive and can be applied in conjunction. For example, a federated User Interface integration can augment an integration bus to show cross-system artifacts. An integration bus can, and should, build on open standards where possible.

SLI data model

Independent of integration type, any non point-to-point integration must support a common data model for the purpose of querying and interchanging lifecycle artifacts. We decompose this model into two main types of artifacts. The focus of SLI is to provide a data model for connecting SoRs via “tasks”, their social communication streams, and contexts that link the relevant assets.

Artifact

Description

Storage

Key attributes

Asset

Unit of value

Typically file-based

  • Knowledge-based content
  • Versions

Task

Unit of work

Typically SoR/REST

  • Description of work to create content
  • Social activity stream
  • Owner, collaborators
  • Timeframe
  • Process model & schema
  • Context, ie, links to related assets

 

 

Two main types of relationships exist between these artifacts:

  • Containment: Artifacts of one type can contain artifacts of another. Eg, tasks can contain subtasks, requirement definitions can be broken down hierarchically.
  • Linking: Tasks can be linked to assets, and vice-versa. Eg, a source code commit links to a user story, a requirement links to all tests that cover it and code changes that implement it.


Software lifecycle bus

Integration is not a new problem. However, solving the problem for the software lifecycle imposes a new set of constraints. In addition to connecting data sources and schemas, the flow of communication must be integrated. The software lifecycle is a conversation, and as such, the primary concern for SLI is to connect conversations across siloes and to model this flow. To do this, we propose that in addition to other integration architectures being deployed a software lifecycle integration bus is a required part of the architecture needed to meet the integration goals of large-scale software delivery. This new kind of bus infrastructure for the software lifecycle is something we refer to as the Software Lifecycle Bus (SLB), a new product category that enables the real-time and event-driven flow of information across the software delivery chain. In this section we discuss the requirements of the bus, as well as deployment and implementation concerns such as storage topology and failure tolerance.

Bus requirements

In order to meet the goals of SLI, the lifecycle bus must meet the following requirements.

Vendor neutrality: Capable of connecting to a tool from any vendor whose API supports the SLI data model.
Real-time communication: Support activity streams (eg, comment threads), identity and real-time activity propagation across tools.
Interoperability: Must support common query and data interchange formats (eg, OSLC, Mylyn).
Invisibility: End-users do not need to change their work practice or technical tools for the bus to function.
Manageability: Must run unattended with automated conflict resolution and error notification system.
Extensibility: Must support addition of connection points, business rules, and transformation semantics.
Openness: Must be query-able and expose its data model using open standards, and support transformations to other data formats.


Bus capabilities

The bus needs to support connecting the following capabilities between the tools that it connects.

Capability

Description

Example

Repository data

Fields of tasks are synchronized

HTML to wiki markup transformation of a task’s description.

Repository schema

Types and values of the repository data

Adding a new version to the QM system also adds it to the Dev system.

Project identity

Project, project area, and product scopes of artifacts

Creating a new project in one system creates it in the other.

User identity

User identity and access control are preserved

When a user creates a task in one SoR, a task in a mapped SoR is created with the same user identity.

Activity stream

Social/collaboration streams are synchronized

Comment on task Dev SoR immediately shows in mapped QA SoR.

Process model

State transitions are synchronized

Cross-repository defect close workflow execution.

Storage elements

Since the bus is not a new lifecycle SoR, it cannot act as the storage for lifecycle artifacts. Instead, it should only store state needed for data consistency, workflow transition transactions, and performance needed to query and flow data across the tools that it connects.

Storage element

Description

Link index

Index of all links between lifecycle artifacts. Must be stored in SLI system for link repair. Must additionally be stored in SoRs for SoR-specific reporting to work. Eg, based on W3C Linked Data spec.

Data index

Index of all SoR data, needed for query performance.

Data cache

Cache of the current data in each SoR. Coupled to schema model and process model. Supports query and other operations. Needed for performance of query and synchronization operations.

Schema cache

Current state of each Repository schema, capable of mapping between schemas. Needed for performance.

Bus operations

The lifecycle bus enables a broad range of operations to be implemented on the information flowing through it. Some of the common operations are listed below.

Operation

Description

Example

Query

Access to data across all SoRs in the lifecycle.

Query for all security defects across all tools in the organization.

Create/update

Create new tasks and update existing tasks.

Application monitoring tool can create a defect via the bus API, and have the defect automatically in the SoR corresponding to the project the defect is associated with.

Map

1 to 1 mapping tasks of the same type between SoRs, including data transformations.

A defect created in the QA system is created and bi-directionally synchronized with the Dev system.

Aggregate

1 to 1 mapping of tasks of the same type, or of subtypes, between repositories.

Time spent working on tasks in a Dev system is aggregated to a plan task in a PPM system.

Failure tolerance

While not a new SoR, the lifecycle bus connects systems critical to software delivery, and as such becomes mission critical to software delivery. If the bus supports write operations, it must also ensure data consistency for those operatinos. As such, the following failure scenarios must be supported.

Failure

Description

Resolution

End-point down

One of the integration targets is down, eg, due to system upgrade

Queued integrations must proceed once available

Temporary outage

Eg, due to integration system update, network outage

Changes during outage must be synchronized on restore

Major outage

Integration system is corrupted

Failover system instantiated with same configuration

Link data deletion

Artifact link is removed, eg, accidental removal by end-user

Links are automatically re-created

Artifact data deletion

Task artifact removed, eg, accidental deletion by end user

Synced artifact is restored, or marked for deletion

Extensibility

Since the lifecycle bus is intended to integrate unforeseen end-points and mapping scenarios, it needs multiple layers of extensibility.

Type

Description

Example

New mapping behavior

Programming model for adding synchronization behavior or semantics

Scripting language support for transformations, with programmatic access to the SLI data model

New custom end-point

A new integration with a custom API can be added

Addition of a REST API to lifecycle bus connector for a newly deployed system

New standard end-point

A new integration with a compliant web service API can be added

Addition of a web service standard endpoint (eg, OSLC)


Conclusion

In today’s software lifecycle, information is locked into stakeholder-specific repositories. Our inability to make information flow both within the organization and across organizational boundaries is preventing the realization of Lean Manufacturing methods in software delivery, and in establishing the infrastructure for a software supply chain. To achieve this, we need an integrated fabric that allows information to flow freely and in real-time across between stakeholders, across the tool silos and vendor boundaries that define the software lifecycle. Creating the Software Lifecycle Integration (SLI) discipline, architectural frameworks and technical tools will provide organizations with the lean methods that will scale software delivery, and pave the way for an integrated software ecosystem.

Appendix

Data model validation

We have used the following sources of information and architectures in creating the models described below.

Open source Eclipse Mylyn implementation and ecosystem of extensions: significant portions of the architecture have been implemented in Mylyn and vetted by it’s user and vendor community, which now includes over 100 extensions and 2M monthly downloads.
Open OSLC standards: These standards are compatible with the underpinnings SLI data model.
Commercial implementations: the Tasktop Sync tool implements the lifecycle bus architecture described here, and the data model has been validated by large-scale customer deployments of Tasktop Sync, as well as a customer-driven data model working group.


Open standards & OSLC

We are still in the early days of ALM interoperability standard adoption. However, standardization efforts have been improving, with the leading efforts supporting key aspects of the SLI data model:

W3C Linked Data: preferred format for artifact linking
OSLC: REST format for queries, cross-repository linking with rich tooltips

    • OSLC-Core: Implements core query and…
    • OSLC-CM: Implements the Task model
    • OSLC-RM: Implements the Requirement and Requirement Definition artifacts in the model
    • OSLC-QM: Implements the Test and Test Plan artifacts in the model
File-based interchange (eg, ReqIF): limited in the task data that can be exchanged, typically simple fields only.


Eclipse Mylyn

The original goal of the Eclipse Mylyn project was to redefine the developer experience around the collaborative and social workflow of task-focused collaboration, and insulate them from the overloaded of non-development related features in ALM tools. The goal of the newly proposed Mylyn m4 project is to leverage the proven interoperability of the existing Mylyn model and to create a new de facto SLI model and reference implementation.


Download the whitepaper

Leave a Reply