A shattered corporate reputation, a CEO forced to resign, a 35 percent drop in stock prices, multiple class action lawsuits, a Congressional hearing, and more than 145 million victims. That’s what happened to Equifax in September 2017 – and “value stream thinking” could’ve prevented it.
How did the Equifax breach happen?
According to the testimony given to Congress by former-Equifax CEO, Richard Smith, the Equifax IT department had neglected to apply a security patch. This patch would have fixed a security vulnerability in Apache Struts, an open source software application that Equifax was using for its online disputes portal.
Smith claims that on March 8th, the US-CERT – which stands for ‘The United States Computer Emergency Readiness Team’, a department of the DHS – notified Equifax and many other companies of the Apache Struts vulnerability.
Equifax sent an internal email to the company’s IT personnel to patch it. However, it was not patched. “An individual did not ensure communication got to the right person to manually patch the application,” said Smith. Moreover, the company’s automated vulnerability scans also failed to detect the vulnerability.
Across May-July 2017, hackers exploited the vulnerability to gain illegal access to Equifax’s systems. They obtained names, birthdates and social security numbers of more than 145 million Americans, exposing them as prime targets for identity theft.
The CEO was first notified of the breach on July 31st, whereby Equifax began compiling the names of the affected people. They only announced the breach to the public, however, on September 7th. This is one example of the many outrageous things in Equifax’s conduct following the breach, and it’s definitely too much to unpack here. But if you’re curious, read CSR’s complete timeline.
But here’s what we want to ask:
Just how can a credit bureau of Equifax’s size, responsible for safeguarding billions of sensitive records on Americans’ financial lives, be managing its security vulnerability patching by email? And how could Equifax not have an automated and traceable workflow for security patching?
The answer, we think, lies in the company’s lack of “value stream thinking”.
When IT is the CEO’s Biggest Blind Spot
If there’s one thing that has become painfully clear to Equifax following the breach, it’s this: a company is only as strong as its IT organization. And that’s true for every company in any industry. It doesn’t matter if you are in financial services, healthcare, government, retail or hospitality.
To acknowledge this fact is to recognize that IT exists to deliver critical business value. And as such, IT requires the same meticulous attention as other core organizational functions such as sales, finance and manufacturing.
At Equifax, that wasn’t the case. Richard Smith had allowed IT to become a place where requests go in and you never know what happened to them. A place managed by email with no follow through. A place with poor traceability and little accountability. IT, in that sense, was the CEO’s biggest blind spot.
Value stream thinking could have changed all that. Because the value stream mindset understands that all your processes are designed to deliver value to the business as efficiently as possible.
So, in the case of Equifax, where your business is keeping millions of Americans’ personal identifiable information safe so you can monetize it, the value stream includes the steps you take to keep that data secure.
To achieve an efficient value stream you must automate and trace the end-to-end flow of work from owner to owner, as it moves through the various phases till completion, applying value stream management principles:
If Equifax was thinking value stream, they wouldn’t have sent someone an email to apply the patch. Rather there would have been a process to create a task for the relevant owner to apply the patch. There would be automated alerts when 24 hours go by without the patch being applied. The alerts would clear only when the owner indicated the task as done.
There would also be reports that measure the cycle time for security patch application – so Equifax could see how long it takes them to apply security patches once notified, check if they’re on target, and adjust the process if there’s room for improvement.
Equifax would have complete traceability of the end-to-end process – from the moment the CERT notification was received until the patch was running in production.
Value Stream Strategies for Complex Software Delivery Organizations
The software delivery value stream is a tough one to manage, not least because the work is intangible. Unlike an assembly line, you simply cannot see the parts. And it surely was complex at Equifax as well, a company of more than 10,000 employees with 58 solutions and 154 products, according to their website.
But the good news for all of us is that the end-to-end value stream can be made visible through the digital artifacts that represent the work, such as features, code, tests, defects, builds, vulnerabilities, patches and support tickets.
By automating the flow of artifacts from tool to tool and from owner to owner, you can trace the work. You can see your value stream, manage it and optimize it. Equifax could have done this, and eliminated both the “human error and technology failures” that they blamed for the breach.
Discovering your software delivery value stream usually starts with a pen and paper, or more likely – a whiteboard and marker.
Here’s what we recommend:
- Start by drawing the end-to-end flow of work. Describe the phases work goes through from inception to completion – tracing the owners and the handoffs. At this point, we’re talking high level.
- Next, go deeper. Identify the artifacts that get created at each stage. Artifacts are things like features, epics, stories, tests, defects, builds, releases, vulnerabilities, and support tickets.
- Now identify the tools used to store and manage the artifacts. Identify the statuses an artifact goes through and think about how currently those handoffs are occurring. Is someone sending an email “I’m done”? Are you updating each other in status meetings? Do you need to send an instant message to your colleague or pop by her office? Is there a checklist you need to update?
- Finally, think about how you can automate the flow of information between the tools to eliminate manual entry, status meetings, and informal handoffs. Perhaps you’ll discover you need some new tools, as well.
Now, we’re the first to admit that this is not an easy undertaking. It’s actually quite challenging and it can’t be done overnight. You can read about our own experiences doing it right here at Tasktop.
But it is vital, and it’s your only choice if you don’t want to be the next Equifax.
