System testing in transformation programs usually includes unit acceptance, configuration, integration, performance, and end-to-end testing, among other methods. But there is a huge, important, often overlooked blind spot that is a missing requirement for transformation programs. Do your testing phases include system failure or unavailability, natural disasters, and cyber-attacks?
Today’s Headlines are a Lens to Your Transformation Program
Does your program training strategy include business continuity plans under these extreme conditions? As the Program Manager, you plan for testing and training, but:
- Have you included testing for business continuity?
- Have you only considered a system or platform failure?
- Have you contemplated the impact to company operations and customers in the event of a natural disaster?
- Did you think about cyber-attacks as part of your testing and training scope?
Recent news called to mind the impact of natural disasters and causal disasters including cyber-intrusion on transformation programs. Often the goal is just surviving the project itself and trying to get the competitive advantage of the program. There is a hidden, but growing, necessity for adequate planning, testing, and strengthening for business continuity in transformation programs.
Case Studies: Boeing and SpaceX
Boeing and SpaceX are providing contrasting news stories on system testing in the press lately. Your board of directors, shareholders, and customers are also learning about system testing when they read about Boeing’s continued challenges and delays related to system testing of Boeing’s 737 MAX.
Based upon my experience, enterprise leaders and communication departments are taking notes on how to deal with a visible public relations and product quality issue. The missed “aha” opportunity is that many senior executives may not have considered that the same alleged root cause of MAX issues is also buried in their current developing transformation, or even their newly released, programs.
On a positive note, recently Elon Musk’s SpaceX successfully destroyed their rocket boosters in midflight to test the escape systems of the Dragon space capsule that will someday carry astronauts. SpaceX said the successful test generated large amounts of data that will take months to analyze.
SpaceX conducted this test and other tests to meet NASA requirements, but you have to appreciate efforts demonstrated in light of the Challenger and Columbia disasters of the former NASA Space Shuttle program that resulted in the visible loss of life and damage of NASA’s credibility.
Lessons Learned from Recent History
If deadly crashes of Boeing 737 Max jets are not raising your risk flags, here are two more real stories that may get your attention – the PG&E wildfires in California and the City of Flint water contamination in Michigan. If you are familiar with these stories you will recall:
- An intentional bypass of controls
- An elapsed time between initial causal events to first steps in problem resolution
- The resolution required loss of service, both temporary and even to this day
- The loss of life, long term impact to health, and loss of property.
Although there are no known cyber incidents to these aforementioned stories, I am looking at PG&E and the City of Flint Water as if they were “hacked” systems. Known controls that were in place to protect the systems and customers were overridden by nature and human intervention – control systems were breached. For purposes of this discussion, I’ll consider these systems were “hacked”.
Like the ongoing 737 MAX story, the impact of these system overrides (“hacks”) and catastrophic failures was magnified because alleged limited training provided little-to-no plan to respond quickly to a dangerous situation – and may have been a compounding contributor to the loss of life.
Transformation and Cyber
Although there is always an elevated focus on transforming your business and delivering working systems, you and your company are taxiing down the runway to leave your comfort zone. Consider these four transformation “known knowns”:
- It is highly likely that some, if not all, of your transformation program will reside in a new cloud environment
- Transformation programs have moved beyond the ERP swap out and are now incorporating Internet accessed and controlled devices in manufacturing plants, autonomous vehicles, power grids, pipelines, etc.
- Your vigilant cyber team did a good job protecting your legacy systems. It was not too long ago that you or your predecessor probably argued with them as they presented various risk scenarios that are now common
- Your vendor has included language in the Master Services Agreement and accompanying Statement of Work that the customer (you) is responsible for cyber security.
Your Role in the Supply Chain
Protect your data and systems, but do not forget the public and your role in the supply chain. You may not realize the elevated risk environment your new systems are operating in unless your Chief Information Security Officer is part of your steering committee or PMO. Compounding the risk level is that the cost of tools to launch cyber-attacks has become cheaper on the dark web. You do not have to be a “foreign dark state” to finance and launch an attack or even someone intent on stealing data. You could just be anyone trying to satisfy their curiosity or intent on creating damage or making an anarchy statement.
Scared? Don’t be scared, just respect the current heightened level of cyber-risk. All of us are operating in a different atmosphere of risk than you experienced when implementing your current legacy systems. Think “worst-case scenario” of the impact your company could experience if your newly deployed transformation program experienced a cyber-attack, including:
- How would you know if your various gauges, meters, and pumps were hacked resulting in misleading systems control data?
- Other than a ransomware note, would you know if your control systems were taken over or purposely reset for destructive and deadly intent?
- Would you know the difference, or how long it would take to detect the intrusion?
- Do your operational teams and cyber teams have a response plan in place for the new environment?
Every company and system will have a different risk profile.
50 Million Reasons for a TESTED Business Continuity Plan
Assume you have invested $50M over a year in your transformation program. You would expect if your new system did not work, your failover would. As we all know, that was not the case in the Iowa Caucuses.
National Public Radio reported candidates spent $50M over the last year campaigning in Iowa, a key hurdle in the 2020 U.S. Presidential primary process. No candidate was able to quickly claim victory due to a failure in the new Iowa Caucus smartphone app and challenges with the backup telephone reporting process.
The root cause of the app failure may never be fully known, but we know the resulting carnage inflicted by this failure is substantial and includes diminished opportunity for candidates to gain momentum as a result of a clear win. Results will be always in question. Iowa’s 2020 Caucus legacy will be a controversy, and perhaps question its legitimacy. Lastly, the technology providers associated with the app will probably never get elected for anything.
While it is not uncommon to spend $50M on a transformation program, you would not expect your investment to sink your business, but it does happen.
Consider System Availability Failure, Disaster and Cyber in Your Transformation Program
The point is to raise awareness and advocate inclusion of testing and training for restricted system availability, disaster, cyber-testing, and continuity planning in your transformation program. As you consider transformation program requirements, it is important to understand how vendor’s methodologies address your concerns on the potential for extreme failures and its associated business continuity.
You still have time to put a plan in place if your program is underway or launched. Collaborate with internal team members, vendors, and third-party cybersecurity suppliers to develop testing plans and train for situations and recovery.
- Fear of Premature Escalation (FPE): Diagnose and Treat this Silent ERP Program Killer
- High-Performing Multi-Vendor Transformation Teams – How to Make Them Work
- Feeling Handcuffed by Your Current IT Vendor? Here is How to Increase Your Renegotiation Leverage.
- Don’t Become an ERP Horror Story – Implement Solid Risk Management