Some of the biggest players who worked behind the scenes during the run-up to the Jan. 3 disclosure of Meltdown and Spectre came together at Black Hat 2018 to discuss what their companies, and others, did after the vulnerabilities first became known.
Eric Doerr, GM, Microsoft Security Response Center; Christopher Robinson, manager, product security program management at Red Hat; and Matt Linton, chaos specialist at Google, sat down before a large audience at the Las Vegas security conference to detail the path taken, from when Google's Project Zero first co-discovered the vulnerabilities in June 2017, to when the news was released to the public.
In their panel session, entitled “The True Story of Fighting Meltdown and Spectre,” the trio also addressed how the industry came together in a rare, but desperately needed cross-company collaboration, and the challenges of dealing with intransigent processor vendors and answering tough questions from U.S. congressional committees investigating the problem.
Google's Linton told the crowd that his company's initial involvement necessitated an act of brilliance by simply deciphering the problems in the first place -- but then a communications error added to the industry's collective headache.
The communications faux pas happened because even though Project Zero is a Google entity, it does not treat its parent any differently than any other company. So when Project Zero disclosed Meltdown/Spectre to Intel and the other processor vendors, it was up to those companies to tell their customers to complete the circle and inform Google. However, the tech giant was somehow was left out of the loop.
Linton said Project Zero believed Intel would tell Google, and the chip makers figured Project Zero would tell its own company of the situation. The end result was that Google was not among the first to know about the situation. However, once they were informed it became Black Swan event for the company, said Linton, indicating the problem was not only rare, but critical in nature.
At first, all the affected companies worked on the problem individually, but after a few months it became obvious that a solution was not going to be easily forthcoming unless their efforts were combined. Linton said the next move was to call an industry summit, to bring all the stakeholders into one tent to work together.
Fellow panelist Doerr explained that pulling together such a team was an almost unheard of action, and many hoops had to be jumped through just to get everyone in the same room. He noted that legal teams had to hammer out an agreement so that no proprietary information would be disclosed and the participants themselves had to get over their basic distrust of each other.
“It might come as a surprise, but Google and Microsoft don't always get along,” Doerr joked.
The summit was held in November and Doerr said he was “blown away” by the level of collaboration that took place. Once ideas began being shared, progress was made.
From a technical standpoint the vulnerabilities were not easier to fix, but “I felt we were all pulling together,” Doerr said.
Red Hat's Robinson noted he was not initially included in the inner circle of those working to resolve the problem, but once his team was brought in they quickly became integrated due to Red Hat's preset group culture.
“We are an open-source company, so working with other companies is nothing new, but not at this level,” he said.
For a time, the U.S. government was also kept in the dark, a fact that would cause Robinson to receive a rather blunt and uncomfortable letter in the near future.
One of the first problems faced by the consortium of companies looking to mitigate the damage was that the processor makers themselves suffered similar problems. They also did not want to speak to each other in order to hash out the problem, Robinson said, but this was eventually rectified through negotiation.
The companies were working together with the intention of releasing the information publicly on Jan. 9, but The Register broke the news on Jan. 2, forcing everyone involved to bump up their timetable to Jan. 3.
Robinson said Red Hat had to scramble to push out patches to meet the new deadline. Interestingly, Robinson said his organization patches vulnerabilities that he considers much more dangerous than Meltdown or Spectre each year and, in fact, he had only rated these flaws as “important” not critical.
Once all the news was public, the U.S. government expressed its annoyance at having not been informed in the form of letters from two congressional committees investigating the vulnerabilities. The elected officials were particularly perturbed because they believed the Chinese government had been informed.
One recipient of the letter, Google's Linton, explained that the government was not told because there was nothing it could do to help.
“We only want people who can help solve the problem involved and there was nothing that they could have done with this knowledge,” he said.
The panelists also denied directly telling the Chinese government anything. Linton noted that China-based Lenovo was informed, but it is only speculation that its engineers told their government the news.
In a rather unusual twist at this point, an audience member stood up and identified himself as one of the Lenovo people contacted. He claimed only about a dozen company employees knew of the flaws and all were U.S.-based and did not pass on any information to the Chinese government.
In the end, the three panelists all agreed that without collaborating at the highest level it would have been much more difficult to roll out an industry-wide response to Spectre and Meltdown.
Meanwhile, the fallout from the two vulnerabilities continues. Just yesterday, Reuters reported that researchers from Austria's Graz Technical University have discovered that Meltdown can be exploited to attack Galaxy S7 smartphones.
