On the ABC medical drama "The Good Doctor," the surgeons at San Jose St. Bonaventure Hospital constantly find themselves making life-or-death ethical and medical decisions.
But last Monday night, when the facility was infected with a ransomware program, the hospital staffers were forced to make another critical choice: Pay or don’t pay?
The premise was, as they often say in television, ripped from the headlines.
Of course, boiling down the intricacies of a hospital ransomware incident into an hour-long drama is no easy task. But the season-four episode “Decrypt” did capture many of the key elements that play into an attack scenario, including incident response, the role of cyber insurance agencies, and the ethics of paying.
SC Media asked two cyber experts with health care experience for their own take on which parts of the episode – written by Thomas L. Moran and Adam Scott Weissman – authentically represented lessons from real-life hospital ransomware attacks, and where dramatic license stretched the limits of believability. (And for an analysis of a similarly themed Grey’s Anatomy episode, see this two-part article from SC Media.)
The attack’s consequences
In the episode’s cold open, the doctors are working on a triathlete with respiratory problems and need to grab a chest tube from the operating room’s automated dispensing system. But for some reason, it won’t open, forcing surgical resident Dr. Claire Browne to smash it open with an ax. A nurse soon explains that the computer systems are down throughout the hospital.
Later, the doctors contend with additional consequences resulting from the attack. For instance, the incident slows down the doctors’ ability to track down living kidney donors. Also, several patients require chemo infusions, but with the electronic health records system down, the doctors must try to acquire their medical histories from external clinics that also treated them. Dr. Browne couldn’t even use a smart microwave to heat up her lunch.
This was only a small taste of what can go wrong during an actual attack. Still, “I think they did a good job of portraying the kinds of problems that can occur. The inability to feel confident in the care provided because of the EHR outage is the primary challenge,” said Drex DeFord, healthcare executive strategist at CI Security and president of Drexio Innovation Network.
“They could have complicated the story further by having the bad guys leapfrog into other organizations’ networks – those connected to the hospital,” DeFord continued. And there was also no mention of the hospital’s business systems being affected, but “it’s a TV show and time is short.”
Tony Cook, head of threat intelligence at GuidePoint Security, thought the graveness of an attack could have been even more starkly represented. “After having the unfortunate experience of working a few incidents involving ransomed hospitals I’m not sure this episode quite hit the mark on how serious a ransomware attack can be,” he said.
To that end, DeFord would have liked to have seen a greater sense of urgency and stress from the cast.
“There seems to be a lot of ‘business as usual,’” he said. From experience, when the network is down for any reason, staff is always very stressed, especially clinical staff caring for patients. They’ve lost data – lab results, med lists, notes, flowsheets – and feel they’ve lost significant control of patient care.”
The initial compromise
In a development that rang very true to the experts, the attackers had actually breached the hospital’s network months before launching their attack, which gave them ample time to compromise and ultimately encrypt not only the active servers, but also on-site and cloud-based back-ups.
“It’s realistic that the bad guys had infected all the backups. Long dwell time is something we see way more often than we should,” said DeFord. “That’s why managed detection and response and end-point detection and response are critical. You may not be able to keep cyber criminals out – but if you catch them, you can kick them out and limit the damage.”
This was definitely a realistic portion of the scenario,” Cook agreed. “However, what isn’t quite accurately depicted here is that offsite backups were also encrypted. Offsite backups may have been affected by the attack by having some means of persistence or resident malware on them from when the backups occurred, but it's highly doubtful that true ‘offline’ backups were affected by the ransomware itself. In some cases, it’s possible to clean offline backups to restore from even if they were affected.”
The episode also refers the critical need for hospitals to patch the initial attack vector that led to the breach. Lea Dilallo (played by Paige Spara), the IT director at the hospital, states that even if she were able to decrypt the impacted data, “I'll also have to retrace the attack chain to find the exact door they came in. Otherwise, we might as well toss every computer we have in the recycle... bin,” because the attackers could simply re-infect the network. (The actual attack vector is never identified.)
One saving grace for the doctors at San Jose St. Bonaventure was that the operating rooms – where much of the show’s drama unfolds – were isolated from the main network, allowing procedures to continue. But was this a mere plot contrivance so that episode could still include several medical plotlines?
“Networks are often segmented, with one part of a subnetwork protected from another, so having the OR on a separate network segment isn’t completely unrealistic,” said DeFord. “If they caught the breach in time, they could have limited the damage to one set of devices, or one network segment.”
“While there are certain portions of most hospitals that are hopefully segmented off of the user network, there are quite a few things that are not, which may or may not affect the various operations,” said Cook. “In some cases when the actors gain access to these networks, they are simply trying to gain access to as much of the network as they possibly can – and if the right security controls have not been implemented then they will gain access to even the most sensitive systems, perhaps without even understanding the harm that could be done.”
In the moments after the attack, the hospital launched a counter response, closing the clinic and ER, diverting incoming patients, and scrapping elective surgery procedures while allowing the most urgent surgeries to proceed. The staff members also contacted the law enforcement authorities and a cyber insurance company.
Many of the key decisions fell into the hands of one man – the president of the hospital, Dr. Aaron Glassman (Richard Schiff). This included the final call of whether or not to pay a ransom of $2 million.
“This seemed to be a one-person IR decision-making system,” said DeFord. “More realistically, we should have seen them bring together a team of folks, and break out the IR plan they’ve written and practiced. But it’s TV, and I understand the shortcut.”
As for the decision to allow some surgeries to continue, “the hospital first has to make sure they have complete visibility of what has occurred along with what systems have been impacted then make risk-based decisions on what operations should be allowed to continue,” said Cook.
Cook was especially critical of the condensed timeframe of the incident response, which he said lacked “anything that resembles a real-life scenario.”
“A hospital rarely goes from being hit with ransomware, to becoming aware they’ve been hit with ransomware, then having all of the offline processes enabled in less than 24 hours. Even in cases where hospitals can restore from backup it can take days to do so, as well as then validating those backups. In the meantime, hospital staffs having to complete arduous offline processes does occur and we commonly see them reverting to paper-based workflows.”
“Even attempting to get records from outside sources while all of your IT sources are down can take quite some time,” Cook added.
Cyberinsurance and the “pay or don’t pay?” debate
Playing a foil to the character Lea in this episode is a cyber insurance representative (Nick D'Agosto) who confidently assures Dr. Glassman that he can negotiate down the cyberattackers’ ransom price to a few hundred thousand.
“You don’t negotiate with terrorists,” she says.
“What I can't do is pay an exorbitant amount of money to an insurance company and then not use them,” responds Dr. Glassman, giving Lea 24 hours to find an alternative solution.
The involvement of the insurance company was a realistic touch, the experts said. However, insurance companies aren’t just there to negotiate the ransom. “They’ll often bring in technical assistance… to assess the situation and help… with the review and IT work – all of which informs the ultimate pay/no-pay decision,” said DeFord.
Also in real life, there would be even more hands on deck.
For instance, at no point did the show portray the hospital “engaging their legal counsel for any legal obligations they might have, as well as coordinating an effective strategy with IT and the rest of the staff,” said Cook.
In most cases, said Cook, the insurance companies “would advise the hospital to contract an external incident response firm, an external counsel/breach coach, as well as potentially IT staff to assist with the restoration from the incident.”
“Usually law-enforcement is involved too,” added DeFord. “Law enforcement usually is against making payment, but ultimately it’s the hospital’s decision. And because of often low investment in critical infrastructure – especially with small- and mid-sized hospitals and clinics – payment is usually the most expedient to get back on-line.
While consulting with Dr. Glassman, this is how the insurance rep accurately sums how ransomware gangs often operate:
“Despite what you see on TV, these attacks don't come from lone wolves. It's a global business. With websites and customer service call centers.”
In this case, however, he notes that the attackers are more amateurish and an “underfunded startup.” Still, he apparently misplays his hand, because the adversaries double their ransom price after being insulted by the negotiations. They also threatened to use a malware program or worm to erase all data and destroy the radiology equipment, which turned out to be a bluff.
“Unfortunately, you’re dealing with a criminal, so you’re trusting that they’ll give you the encryption keys,” said DeFord. “And you’re trusting that the keys will work – because the quality-control in some of these encryption programs isn’t top-notch. And you’re trusting that they won’t take your data anyway, and do something else with it – like go directly to the patient asking for individual ransom.”
Still, attackers can hurt themselves in the long run if they don’t keep their word or if their decryption keys are proven ineffectual. Future victims may refuse to pay up.
“In my experience dealing with ransomware actors, this is a business for them in which they’ve done their homework with how much to request to ensure they can receive a payment,” said Cook. “Typically the only reason for the ransomware to double is that the victim doesn’t pay the ransom in a particular timeframe. However in most cases this can be negotiated as well.”
Lea has 24 hourss to see if she can decrypt the malware or find a coding error on the part of the attackers. Such a scenario in real life would be a Hail Mary, though if the adversaries were truly amateurish, then perhaps a slip-up might be caught (but almost certainly not in one day).
To the writers’ credit, Lea did not defeat the encryption. Instead, she discovered that there was an unspoiled server that had been disconnected and set aside just days before the attack, after one of her workers had accidentally spilled coffee on it, corrupting some of the circuitry.
Still, that’s a major lucky break bordering on “deus ex machina” territory – ultimately allowing Lea to restore the network from this one server.
“I think this was some TV magic to give the team a way out,” said DeFord. “I didn’t really understand the idea that they found one disk from a server, and they could then restore the whole network from that disk. Ransomware is complicated. The amount of data usually encrypted is extensive. So the save might have been a bit simplistic – but hey, it’s TV.”
Cook was less forgiving.
“Even under the guise that all the cloud and offsite backups were somehow unavailable, a single IT person was able to find a single damaged hard drive from a single server and then somehow within a few hours be able to restore the entire network while then somehow booting all the computers up restored from a completely powered off state. Shenanigans,” he said.
If a real hospital had found itself in a similar situation as St. Bonaventure Hospital, “a payment is almost always going to occur. The alternative would be to rebuild the affected hospital infrastructure as fast as possible with no historical data.”
In real life, the cost of paying the ransom would end up being less than the expenses incurred from rebuilding the network, Cook added.
“The cost of having your services interrupted is $1 million a day, and getting access to your offline backups is going to take a week, versus 24 hours to receive decryption keys,” said Cook. “They might opt to pay the $2 million to reduce the time to recovery if successful decryption is a viable solution. That said, there's obviously a level of risk associated with transacting and trusting a criminal organization to help you successfully recover.”
And just as the incident response felt unrealistically hurried in the episode, so was the time to full restoration.
“Recovery typically takes a significant amount of time," Cook said. Victims "need to prioritize systems that need to be brought online and progressively recover. Sometimes this process can take weeks, if not months.”
So then, what's the final word?
Cook: "I understand it was [for drama], but it was grossly negligent to romanticize the idea that one person, with one damaged hard drive, can bring back up an entire hospital network – really any network – in less than 24 hours. For anyone that doesn’t understand the hard work and time that goes into the entire incident response, this could lead them to believe the fantasy that this is an easy process and that is completely false."
DeFord: "I’d give them a solid B-. It’s TV. [There’s] nothing technically perfect in these shows. It’s decent entertainment, and this was insightful for viewers who don’t know much or anything about ransomware. I wish they could have shown more urgency where patient care was concerned. Lea is a superhero in this show. Seemed like she did this all alone – not realistic. And we never found out how this started, and how they’re keeping it from happening again."