Researchers are warning firms should make sure they are prepared for partial failures in IT infrastructure after a “very rare” data center glitch caused a payment meltdown on Visa systems.
Earlier this month, a partial network switch failure in one of Visa's two data centers caused nearly 10 percent of 51.2 million attempted transaction in Europe to fail, resulting in millions of credit cards being declined over the course of 10 hours.
Visa has since launched several reviews and is in the process of migrating its European systems to a more resilient global processing system, while Visa was able to resolve the issues relatively fast, researchers are warning other firms to better plan for partial network failures as well.
Databarracks Managing Director Peter Groucutt said companies are usually better prepared for a complete outage than they are for partial failures.
“When a system fails completely the process to fail-over is more clearly defined to whether it is a manual action, or automatic process,” Groucutt said. “Partial failures however, make that change-over difficult.”
Groucutt added, the importance of making the decision to either fully switch to the secondary system or fix the problem on the primary once the issue has been identified. In Visa's case, the company had a very mature incident management process in place and based on the reports able to quickly identify the problem and initiate the proper response and protocols, he said.