Cloudbleed has the potential to resonate for some time.
Cloudbleed has the potential to resonate for some time.
One errant character in a coding string caused a buffer overrun which, in turn, led to a memory leak that dumped personal data in plaintext from a variety of Cloudflare's customers' sites.

The incident has been dubbed Cloudbleed, by Tavis Ormandy, a researcher with Google's Project Zero, who was examining publicly available datasets when he detected an anomaly. He came across corrupted web pages being returned by some HTTP requests. The bizarre code – "chunks of uninitialized memory interspersed with valid data" – was exposing chat messages, encryption keys, cookies, passwords and other personal data leaked from the edge servers of the internet infrastructure company. In other words, personal information that usually would be encrpted or somehow obfuscated was available.

At that point Ormandy understood the seriousness of the situation and, as it was a Friday night, took to Twitter to reach a contact at Cloudflare he could alert. Luckily, he reached a rep who recognized the severity of the situation and mitigated the flaw within an hour.

However, while Cloudflare's system was sealed, it's unknown at this time how much leaked data – much of it unencrypted – ended up being cached by the major search engines. The company had to reach out to the search engines and request that they manually scrub the exposed data.

More than five million websites use Cloudflare's content delivery network and internet security services. A number of major services were affected, including Uber, 1Password, FitBit and OKCupid.

The coding error was traced to an HTML parser, an asset that helps obscure email addresses associated with websites from scraping bots.

OUR EXPERTS: Cloudbleed

George Avetisov, CEO, HYPR
John Bambenek, threat systems manager, Fidelis Cybersecurity
RJ Gazarek, product manager, Thycotic
Alex Heid, chief research officer, SecurityScorecard
Erik Knight, CEO, SimpleWan
Gunter Ollmann, CSO, Vectra Networks
Chris Roberts, chief security architect, Acalvio
Nathan Wenzler, chief security strategist, AsTech

After a few days of scrambling, Cloudflare reported that it scrubbed the cached data. However, while Ormandy said a copy he received of Cloudflare's postmertem was excellent, it "severely downplays the risk to customers."

John Graham-Cumming, CTO at Cloudflare, writing on a company blog, said no evidence of malicious exploits of the bug or other reports of its existence had been detected.

Gunter Ollmann, CSO at Vectra Networks, told SC Media on Friday that he commended Cloudflare for its rapid reaction to the vulnerability once they had been alerted to its existence – quickly removing the vulnerable process and effectively fixing it over the course of a few days. "Their detailed step-through of the vulnerability clearly underlines the severity of the issue and points out, once again, the frailty of modern systems to latent bugs in old software that can be suddenly exposed and exploited through the smallest of code changes."

However, he added, while Cloudflare responded quickly and appropriately to the disclosed vulnerability, the vulnerability and the exposure it brought to the confidential and personal data of all internet users of the online businesses that Cloudflare provides a service to, is a critical issue that has existed for a substantial period of time – likely for a year – when they started making changes away from their Ragel-based parser.

"It is unclear whether the vulnerability had been exploited by malicious actors before Google's alert to Cloudflare, Ollmann told SC. "However there is much clean up to be done regardless, as search engine and data caching server providers around the world will need to purge erroneous and confidential data cause by this critical security flaw.”

Other experts fear this might have the potential to be the most serious security event and leak we have seen to date. 

"Unfortunately, we won't know the full scope of the damage done for some time now," RJ Gazarek, product manager at Thycotic, told SC Media on Friday. "Sadly, this will come primarily from the selling of discovered data and secondary breaches due to this leak."