The incident has been dubbed Cloudbleed, by Tavis Ormandy, a researcher with Google's Project Zero, who was examining publicly available datasets when he detected an anomaly. He came across corrupted web pages being returned by some HTTP requests. The bizarre code – "chunks of uninitialized memory interspersed with valid data" – was exposing chat messages, encryption keys, cookies, passwords and other personal data leaked from the edge servers of the internet infrastructure company. In other words, personal information that usually would be encrpted or somehow obfuscated was available.
At that point Ormandy understood the seriousness of the situation and, as it was a Friday night, took to Twitter to reach a contact at Cloudflare he could alert. Luckily, he reached a rep who recognized the severity of the situation and mitigated the flaw within an hour.
However, while Cloudflare's system was sealed, it's unknown at this time how much leaked data – much of it unencrypted – ended up being cached by the major search engines. The company had to reach out to the search engines and request that they manually scrub the exposed data.
More than five million websites use Cloudflare's content delivery network and internet security services. A number of major services were affected, including Uber, 1Password, FitBit and OKCupid.
The coding error was traced to an HTML parser, an asset that helps obscure email addresses associated with websites from scraping bots.
| OUR EXPERTS: Cloudbleed George Avetisov, CEO, HYPR
|
After a few days of scrambling, Cloudflare reported that it scrubbed the cached data. However, while Ormandy said a copy he received of Cloudflare's postmertem was excellent, it "severely downplays the risk to customers."
John Graham-Cumming, CTO at Cloudflare, writing on a company blog, said no evidence of malicious exploits of the bug or other reports of its existence had been detected.Gunter Ollmann, CSO at Vectra Networks, told SC Media on Friday that he commended Cloudflare for its rapid reaction to the vulnerability once they had been alerted to its existence – quickly removing the vulnerable process and effectively fixing it over the course of a few days. "Their detailed step-through of the vulnerability clearly underlines the severity of the issue and points out, once again, the frailty of modern systems to latent bugs in old software that can be suddenly exposed and exploited through the smallest of code changes."
However, he added, while Cloudflare responded quickly and appropriately to the disclosed vulnerability, the vulnerability and the exposure it brought to the confidential and personal data of all internet users of the online businesses that Cloudflare provides a service to, is a critical issue that has existed for a substantial period of time – likely for a year – when they started making changes away from their Ragel-based parser.
"It is unclear whether the vulnerability had been exploited by malicious actors before Google's alert to Cloudflare, Ollmann told SC. "However there is much clean up to be done regardless, as search engine and data caching server providers around the world will need to purge erroneous and confidential data cause by this critical security flaw.”
Other experts fear this might have the potential to be the most serious security event and leak we have seen to date.
"Unfortunately, we won't know the full scope of the damage done for some time now," RJ Gazarek, product manager at Thycotic, told SC Media on Friday. "Sadly, this will come primarily from the selling of discovered data and secondary breaches due to this leak."
While Gazarek acknowledged that Cloudflare worked hard with Google to scrub the search engines before the announcement was made public, several online forums, he pointed out, will quickly tell you they didn't get it all. "There is still a ton of cached data and, as more surfaces, we're seeing it disappear (which is a good thing)."
He believes malicious attackers are sending out their web crawlers and their botnets to scour the internet for this data. However, he said, this is not the most concerning part. "What's troubling is the mere thought that Google or Bing are the only search engines out there. This leak has allegedly been happening for months now, and there are tons of crawlers, that do not belong to Google, that scrape the web for other search engines, or in some cases, for malicious intent. Despite the scrubbing, Gazarek said he feared that this data is still out there in other lesser known search engines and web scrapers.
Chris Roberts, chief security architect at Acalvio, agreed that this is a very serious leak. "It's a security company whose code migration introduced a flaw that wasn't caught and that's never good, especially when that flaw is introducing the ability for code to store/cache any number of elements (credentials, preferences, history etc.), he told SC on Friday. "However, with that being said, there's been little chatter about it on the darker side of the world, so the actual exploitation of the code is hopefully limited."
Roberts said he hoped Cloudflare is making every effort to scrub the platform to ensure nothing is retained. But, he said the company might need to retain it for evidentiary purposes. "If someone's going to lose stuff, they'll sue. Cloudflare will need to have evidence for/against it."
Nathan Wenzler, chief security strategist at AsTech, told SC Media on Friday that considering the sheer scope of sites involved alone, this would be seen as a very serious leak. But, he said, what makes it more troubling is the type of data reported to be found from the incident. "It's not just leaked passwords, but it's personal messages, in-app emails, pictures, videos and more. The personal nature of this kind of information makes this one of the more potentially devastating security incidents we've seen in recent years."
But, he believes there's no reason to believe Cloudflare has not scrubbed the data. The question lies more in where this data may be cached outside of its own networks and on another CDN or search engine which may have crawled the data before it was cleared from its own caches, he told SC. "One of the strengths of the internet also works against us in situations like this: nothing is ever truly gone when so many systems are interconnected like this."
Wenzler said since there was no shadowy hacker maliciously breaking in from the depths of the DarkNet and stealing this information, it's possible the average user may not take this as seriously as they should and thus take the necessary steps to protect themselves. "This is a cultural shift that has been happening for several years now, and it could make matters even worse in the long run if people do not take these sorts of things seriously and respond with the urgency that is warranted."
Yes, mistakes do happen, he said, adding that he expected that more companies will have issues like this where old code or other software dependencies come back to haunt them. "But it is imperative that users realize this is a dangerous security breach and take action."
His advice: change all your passwords immediately, use a password manager to make it easier to do so in the future and add two-factor authentication to accounts wherever possible.
Organizations which develop software should take this incident as a warning sign for their own development teams and processes, Wenzler said. "While speed is key in order to bring products to market, overlooking basic code review and security validation processes can result in something like Cloudbleed."
This is more than just a technical problem to be solved, he added, but it hits right at the reputation and credibility of the company and could end up costing an organization huge amounts of revenue from customers taking their business somewhere else. "I am a strong advocate that development houses take the time to review their applications as holistically as possible, and not solely look at the latest block of code, with the assumption that it will work perfectly with all the previously developed software."
John Bambenek, threat systems manager at Fidelis Cybersecurity, pointed out a lack of skilled workers to do thorough testing of custom security tools, but had praise for both Cloudflare and Google for their collaboration in fixing the bug. "Cloudflare should be commended for their transparency and quick response to the issue. This morning I received an email from them (I have a site behind CloudFlare) that detailed the issue and steps they've taken to remediate the issue. Both companies have demonstrated how to work together to address vulnerabilities quickly and protect the internet at large.”
George Avetisov, CEO at HYPR, agreed that this is a major hack as content has already been leaked and cached by search engines. As there are a few digital currency websites on the list of affected sites, we're are likely to see quantifiable financial damage as a result of this breach, he told SC.
When asked whether he believed Cloudflare was able to scrub data from the search engines, Avetisov told SC that the company cannot scrub search engines but only work with them to identify and remove incorrectly cached info. "Obviously, bad actors will not be known or comply if they are asked. Unfortunately, this problem is unlikely to be remedied entirely by Cloudflare and will require cooperation cross-industry."
Enterprises are going to see a lot of employee password resets today, he added. "But the true extent of the damage might not be revealed for several weeks. Employees are known to re-use passwords across personal and corporate accounts, so we are likely see further indirect breaches as a result of the sites impacted by CloudLeak."
That's because any data transferred over these services during the vulnerable time should be considered public, he said. "This unfortunately includes personal conversations and dating services.
Right now, it's probably better for enterprises and consumers to remain calm and update their passwords, Avetisov said. "Overreacting and playing the blame game, like the internet did when HeartBleed was revealed, is not going to remedy the situation."
Alex Heid, chief research officer at SecurityScorecard, agreed that the leak appears to be quite serious, as the impact appears to be a complete compromise of confidential information being passed over HTTPS.
He told SC that as of mid-day Friday, he was still able to find cached information on Google. "The leaked data appears to have impacted everything being sent over an HTTP post request, which can include authentication credentials such as passwords and API keys, as well as any text communications being passed," Heid told SC.
"It will be interesting to observe the fallout over the next several months, as information obtained during this leak is leveraged for future attacks, much in the same way compromised credentials were harvested and used from HeartBleed incident, or from publicly circulating databases such as LinkedIn and Dropbox breaches."
Erik Knight, CEO at SimpleWan, also believes the leak is very serious, "especially since Cloudflare has been touting its security." Knight believes it will take some time before the data is scrubbed from all the search engines.
"The data is already out there, it's not possible to undo what's already been captured," he told SC. And, he added, internal audits by Cloudflare should catch things like this.
This is a big event because of the wide reach of Cloudflare, Knight said. "But it's not surprising, we'll see more like this from other vendors in the space."
