Hundreds of millions of Internet of Things (IoT) products use a TCP/IP software library containing severe vulnerabilities that can be exploited for remote code execution and complete device takeover, say researchers who also warn that the bug has been extremely difficult to track across the IoT supply chain due to liberal adoption of the third-party code. Making matters worse, the bugs may never be fully patched in some products.

Researchers at JSOF discovered a total of 19 flaws in the low-level stack from Treck, Inc., known as a a leading provider of embedded Internet protocols. Collectively dubbed Ripple20, the vulnerabilities include four critical flaws with CVSS base scores over 9.0.

Affected products include electronic appliances, smart devices and home and office printers, as well as information and communications technologies, industrial controls systems and operational technologies, cellular networks and data centers. Virtually every major industrial sector uses impacted devices of some kind, including health care, energy, transportation, telecom, retail government and aviation.

Vendors confirmed to be affected include B. Braun, Baxter, Carestream, Caterpillar, Cisco (through Starent), Digi International, Green Hills, HCL Tech, Hewlett Packard Enterprise, HP, Intel, Maxlinear (through HLFN), Rockwell Automation and Schneider Electric/APC, according to an online report published by JSOF and vulnerability advisories posted by the CERT/CC at Carnegie Mellon University (here) and the U.S. ISC-CERT (here). However, the fate of dozens more product-makers remain unknown. (June 23 Update: Sandia National Labs, which was originally listed as affect, is actually not affected.

Leveraging certain bugs, adversaries could hide malicious code in devices for years before striking in a strategically timed attack. They could remotely hijack internet-facing network devices — targeting very specific devices if they so choose or taking over everything at once. And in some of the most serious cases, bad actors could attack devices from outside the network while bypassing any Network Address Translation (NAT) configurations by performing man-in-the-middle (MITM) attacks, by conducting a DNS cache poisoning attack or by replying to packets that leave network boundaries.

In their write-up, the researchers ticked off a series of potential attack scenarios made possible by the vulnerabilities, including stealing data off printers, sabotaging an infusion pump, or causing the malfunction of an ICS device, perhaps one used in a critical infrastructure facility.

“A vulnerability in the TCP/IP stack can be dangerous for the low level of access that it can give attackers. By now we know that most businesses are running a huge number of connected devices in the workplace. If an attacker exploits Ripple20 on one connected device, it could have a significant impact on an entire business network,” said Ben Seri, VP of research at Armis, who last year led the disclosure effort around Urgent/11, another set of TCP/IP vulnerabilities found in the real-time operating system VxWorks. “Ripple20 is an example of how IoT can be a roadmap for attackers looking to target businesses. Since many companies don’t know what kind of IoT devices sit within their environments or know how to protect them, attackers are starting to see IoT as an easy entry point for short term gains and long-term attack campaigns.”

Treck issued a release that fixed the vulnerability on March 30 (see vulnerability advisory here), but there will be scenarios where patching a device is impossible, especially if the vulnerable library exists on a separate physical component or a component manufactured by a company that’s no longer in business, JSOF reports.

Hunting down vulnerable instances across the IoT supply chain has proven difficult as well, the researchers explain, due to rampant use of the third-party library code in many different forms and scenarios. Indeed, Scott Caveza, research engineering manager at Tenable, said the report “highlights an often overlooked security concern: vendors reusing and repurposing common software libraries. This practice creates challenges when it comes to identifying and patching logic and security issues in code, as it becomes a vendor-specific issue. A fix for one vulnerability might have multiple solutions from various vendors, and it’s possible specific patch attempts could open up additional attack vectors if not properly implemented.”

“The library could be used as-is, configured for a wide range of uses, or incorporated into a larger library,” the JSOF report explains. “The user could buy the library in source code format and edit it extensively. It can be incorporated into the code and implanted into a wide range of device types. The original purchaser could decide to rebrand, or could be acquired by a different corporation, with the original library history lost in company archives. Over time, the original library component could become virtually unrecognizable. This is why, long after the original vulnerability was identified and patched, vulnerabilities may still remain in the field, since tracing the supply chain trail may be practically impossible.”

“Many of the affected vendors in this case and many of the affected devices were not even clients of Treck; they were clients of a company that was a client of Treck,” said Shlomi Oberman, CEO of JSOF in an interview with SC Media. But ultimately, it’s the vendor’s responsibility to ensure that even the tiniest, second-hand embedded component meets security expectations, he said.

“Every company in the supply chain, and especially companies selling to operators and end users should really be worried about the risk taken on with third party components because they’re the consumer-facing brands at the end of the day” and will have to answer to the purchaser, continued Oberman, one of five JSOF researchers credited with work on the project. Oberman said one viable course of action to reduce such risk in the future is for product manufacturers to require a software bill of materials listing all the third-party code and components used.

Jonathan Knudsen, senior security strategist at Synopsys, agreed that Ripple20 demonstrates why organizations that create software “must manage their third-party components.”

“The main reason for the far-reaching effects of the Ripple20 vulnerabilities is that they are vulnerabilities in a network component used by many organizations in many products,” Knudsen said. “Each software development organization must understand the third-party components they are using to minimize the risk that they represent.”

Knudsen also identified two other key takeaways from this research: “security must be integrated [into] every part of software development” and “systems and devices must be able to update themselves securely, and the manufacturer must make a commitment to maintaining the software for some clearly stated time period.”

Complicating matters further, Treck in the 1990s co-developed the buggy TCP/IP stack with Japanese company Elmic Systems — now known as Zuken Elmic — until they later split from each other and began independently managing their own version under different product names. This meant the researchers also needed to separately reach out to Zuken Elmic with the help of Japan’s CERT/CC. (JSOF also credited Israel’s national CERT for help with global outreach, along with the aforementioned CERT/CC and ICS-CERT.)

Seri from Armis understands why this must have been a daunting undertaking. “From my experience in finding URGENT/11, I saw just how difficult it was to reach out to the ecosystem of vendors using one TCP/IP stack,” he said. “The IoT system is interconnected to the point of confusion, and can expose many businesses.”

As Treck attempted to track down its own customers, JSOF also worked with security vendors CyberMDX and Forescout to check their respective client networks for any signatures corresponding to use of the TCP/IP stack.

For those unable to apply Treck’s patch, JSOF said other mitigative actions include minimizing embedded and critical devices’ exposure to the network and internet, limiting internet access, segregating OT networks and devices behind firewalls and isolating them, and enabling only secure remote access methods. JSOF also suggests blocking anomalous IP traffic, blocking network attacks via deep packet inspection, and employing preemptive traffic filtering.

“This is a classic case of finding critical vulnerabilities in embedded IoT devices that were designed years ago and may now be impossible or impractical to patch,” said Phil Neray, VP of IoT & industrial cybersecurity at CyberX. “The best strategy is to implement compensating controls such as network segmentation to make it harder for adversaries to connect to these devices, plus Network Traffic Analysis (NTA) with Security Orchestration, Automation, and Response (SOAR) to quickly spot anomalous behavior — and stop it — before they cause a safety incident, shut down production, or steal intellectual property.”

JSOF researchers say they started analyzing Treck in September 2019 and found troubling signs of vulnerabilities within months. So began the large-scale campaign to alert as many affected vendors as possible.

JSOF reports says it twice postponed public disclosure, ultimately giving companies a 120-day window to patch the bugs because multiple vendors requested more time, with some blaming COVID-19 for delays.

“The interesting thing about Ripple20 is the incredible extent of its impact, magnified by the supply chain factor,” the JSOF report states. “The widespread dissemination of the software library and its internal vulnerabilities was a natural consequence of the supply chain ripple effect. A single vulnerable component, through it may be relative small in and of itself, can ripple outward to impact a wide range of industries, applications, companies and people.”

“As a dissemination vector, the complex supply chain provides the perfect channel, making it possible for the original vulnerability to infiltrate and camouflage itself almost endlessly.”

Although it was not the highest-scoring vulnerability, JSOF believes the single most dangerous Ripple20 bug is CVE-2020-11901, which the ICS-CERT describes as an “improper input validation in [the] DNS resolver component when handling a packet sent by an unauthorized network attacker.”

JSOF says the flaw was successfully demonstrated to perform remote code execution on a Schneider Electric APC UPS, but it “affects any device running Treck with DNS support.”

“This vulnerability can be triggered by answering a single DNS request made from the device,” the report states. “In our opinion this is the most severe of the vulnerabilities… due to the fact that DNS requests may leave the network in which the device is located, and a sophisticated attacker may be able to use this vulnerability to take over a device from outside the network through DNS cache poisoning, or other methods. Thus an attacker can infiltrate the network and take over the device with one vulnerability bypassing any security measures.”

“The malformed packet is almost completely RFC [Requests for Comments] compliant, and so it will likely be difficult for security products such as firewalls to detect this vulnerability. On very old versions of the Treck stack, still running on some devices, the transaction ID is not randomized making the attack easier.”

Two flaws have the maximum CVSS base score of 10: CVE-2020-
11896 and CVE-2020-11897. The former is a RCE and denial of service flaw that was demonstrated on a Digital Internet device, but can affect any device running Treck with a specific configuration. It “can be triggered by sending multiple malformed IPv4 packets to a device supporting IPv4 tunneling,” the JSOF report states.

The latter of the two top-scoring bugs is an out-of-bounds write that JSOF says “can be triggered by sending multiple malformed IPv6 packets to a device.” It was previously patched by routine code changes taking place back in 2009, but devices running older versions of Treck with IPv6 support could be affected.

The other 16 flaws sport CVSS base scores ranging from 3.1 to 9.1 and can cause exposure of sensitive information, remote code execution or denial of service, depending on the bug. Fifteen of the 19 total bugs were discovered as true zero-days, while the remainder had been fixed by code changes made over time.