Hot: It's one of the primary methods that malicious hackers use to find new application and operating system vulnerabilities. And it's also a powerful tool that professionals use to analyze the security strength of their applications. We're talking about reverse code engineering.
If you've never considered rolling up your sleeves and sinking your hands into learning how certain system drivers and newly-found malware applications work, now is the time. Reverse engineering can be a powerful way for security professionals to ensure and to keep systems safe.Reverse code engineering is the process of examining exactly how a software application, or component, actually works. While malcontents reverse engineer software to find weaknesses in systems and design spyware, illegal adware, and trojans, the same techniques can be used by security practitioners. In this way, they can perform a forensic analysis on a virus or a spyware-infected system to learn just how much of a threat the malware really is. Was it simply semi-harmless adware? Or was the program capable of capturing every keystroke typed on the system? Proficient reverse engineers also can use this tactic to create on-the-fly signatures to be deployed in their intrusion detection/prevention systems, as well as insightful application penetration assessments.
Other uses for reverse engineering include the discovery of undocumented APIs or porting drivers, and for software patch analysis.
By familiarizing yourself with a few tools listed below, and studying (or taking a programming course in assembly language), you'll add a significant new capability to your security skill set. As a caveat, before you begin reverse-engineering any software application, make certain you have the necessary legal clearances. Many commercial applications have agreements that forbid it, and reversing may be illegal under certain laws.
DisassemblersOne of the primary tools used in reverse engineering is a disassembler, which reverses the process of assemblers. That is, they attempt to recreate assembly code from unreadable compiled binary machine code. Commercial Windows disassemblers include IDA Pro and PE Explorer , and popular freeware disassemblers include IDA 3.7 , IDA Pro Freeware and the BOR Disassembler . Dissassembling the code makes it possible to study exactly how the program works, and even identify potential vulnerabilities. For example, if you reverse engineer spyware on a system, you could determine exactly what type of information the application was trying to snoop, as well as its other capabilities.
Decompilers
Decompilers take the process a step further and actually try to reproduce the code in a high-level language. Frequently, this language is C, because C is simple and primitive enough to facilitate the decompilation process. However, decompilation has its drawbacks. Plenty of data and readability constructs are lost during the original compilation process, and they cannot be reproduced. While the science of decompilation is still young, the results are good but not what I'd call great. However, it's still a craft very much worth learning. Common decompilers include DCC Compiler, the Boomerang Decompiler Project, Reverse Engineering Compiler and ExeToC.
Debuggers enable reversers to step through the execution of a program and examine various values and actions throughout application flow. Reversers can set application "breakpoints" on instructions, function calls, and even memory locations, so you can study specific locations of program execution. Windows Debuggers include OllyDbg, WinDBG and IDA Pro.
As is the case with most every aspect of IT security, the practice of reverse engineering has created somewhat of an arms race. And the result is the practice of "anti-reversing" techniques known as code obfuscation, which is the process of encrypting or scrambling machine code. Virus writers have been known to scramble the code to hide the capabilities of malware, and to thwart signature creation. While the process still is relatively immature, more organizations and companies are obfuscating their proprietary code to protect their intellectual property, or make it much more difficult to find vulnerabilities that attackers can exploit.
Now, once you have the ability to reverse engineer, the next time you're engaged to penetrate test a server or website for which you don't have the source code, or you find an unknown program that you know or suspect to be malware, you'll be able to determine the extent of the threat. Bad guys certainly use reverse engineering to exploit systems, so there's no reason why security practitioners shouldn't use the same tools to protect them.
-Amol Sarwate is director of Qualys' vulnerability research lab
