Testing competitions can improve code, but crowdsourcing brings new concerns to security governance, reports Deb Radcliff.
Downsizing of corporate structures coupled with a stronger-than-ever need for agile development, has allowed an extreme form of outsourcing, known as crowdsourcing, to take foot in the third-party development economy, according to analysts.
With crowdsourcing, code is exposed to groups of developers and testers in the form of contests. The benefit of more contestants – that is, developers and testers – reviewing the code is more reliable applications, experts say. The more who review, the more bugs they catch, and the harder they make it for a bad apple to hide intentional back doors.
On the other hand, there are concerns around turning over code development to an anonymous pool of contributors. And while crowdsourcing may result in more reliable code, the model doesn’t support formal security review during and after development. That’s because most of the clients acquiring web, iPhone, and some client/server apps through crowdsourcing do not think their applications require security review.
TopCoder, an established crowdsourcer with more than 231,500 registered members, offers a good model of how the process works. TopCoder operates a community-based model to attract developers to contests that match their skills and interests.
“When you get more into the engineering, design and development contest calls, you’ll have somewhere in the neighborhood of 20 competitors,” says Tony Jefts (left), TopCoder’s director of software operations. “Our community will rigorously score and test those submissions, and some of the members will do line-by-line vetting. Then we select the top two or three.”
In addition to offering reliability bonuses as part of the development award, TopCoder also offers additional awards – $2 per bug or hundreds of dollars to the top contestant – in bug-finding contests for specified modules.
“When we put a contest out there, we’re structuring it to filter in the best talent for that particular piece of work,” explains Jefts. “Any project we do is a series of different contests to tap into this crowd.”
One of the concerns that comes to mind in this model is who is vetting the vetters, asks Stan Lepeak, managing director of global research for EquaTerra, a global outsourcing analysis and services firm.
“With crowdsourcing, you’re exposing your application to a broader group of programmers over whom you have less knolwedge and control,” he explains.
In TopCoder’s case, contestants are vetted by online registration, through signed documentation and through reputation over time. In some cases, and in some countries, a notary is required, and in others, customers may specifically request only coders who have passed background checks. Requests for things like background checks increases the cost of the competition and narrows the pool of candidates considerably, adds Brendan Wright, TopCoder’s VP of software development.
These developers enter contests for the love of it, Wright says, and are often students without formal training. The community is engaged in self-improvement and helping one another through a variety of forums offered by TopCoder, as well as through TopCoder-supported training, standards frameworks and development tools that are frequently updated.
“Crowdsourcers like TopCoder do a good job of establishing formal relationships with their contractors,” says Lepeak. “Where you run into problems is in testing. Even after the code leaves the development house, how do you test it? And if the application fails, do you have any recourse?”
In the rare cases when the purchaser specifies that security testing is required, crowdsourcers like TopCoder, and even a testing-specific crowdsourcing firm, uTest, don’t have this capacity in-house.
“We don’t test explicitly around security. We don’t do a code-level audit,” says Matt Johnston, VP and evangelist for crowdsourcing at uTest, a software testing crowdsourcer with 22,000 testers from 160 countries. “We do functional, technical and GUI tests in real-world conditions. In those contests, we do catch cross-site scripting, SQL injections and other vulnerabilities.”
In addition to limitations in testing the code during development, testing after the code is compiled into the application is also difficult, as Lepeak says. In the crowdsource model, source code is kept by the crowdsourcer in modules for re-use and not opened to the client.
“When you’re trying to manage application security risk, particularly in third-party code, the concept of layered testing is critical,” says Donna Durkin, information security and privacy officer for Computershare, a $1.5 billion global service provider of investor and corporate administration services. “You want testing and remediation throughout the software development lifecycle. You want to test the various components being developed, then retest them again as they plug into the production environment.”
For security testing, TopCoder and uTest send clients to Veracode and Fortify for static analysis review on already compiled binary code (Coverity, Ounce, Klockworks, Microsoft and others have static analysis tools that do this as well).
“Static analysis review can find coding errors, as well as hidden commands that are intentional backdoors,” explains Chris Wysopal, co-founder and CTO of Veracode. “Backdoors could include embedded passwords, IP and email addresses used to leak information out of the application, ” he says.
Static analysis of binary code may work for Java, .NET and other compiled code formats. However, the ubiquitous PHP scripting language and other web development frameworks cannot be compiled and must be examined as source code, says Jacob West, director of security research at Fortify. Static analysis tools can test this source code, but with the code being housed at the crowdsourcer, there is no access.
Not being able to test PHP code, then, leaves a huge gap in an application security analysis where PHP is involved. So in addition to its binary analysis, Fortify will also refer customers to its partner, Whitehat Security, for additional pen testing of their web applications.
Bugs that are found are reported back to the crowdsourcer, such as with TopCoder, where the source code resides, making the crowdsourcing communities the gatekeepers responsible for ongoing maintenance of the source code they’re developing to.
This model for outsourcing development to one provider and security review to a different provider is nothing new, says Lepeak (left). It’s just more complicated in a crowdsourced environment where projects are developed in multiple modules, and the acquirers of the applications can’t access the source code.
Fortify’s West argues that security review can be sold as a value-add for enterprises like ComputerShare, which doesn’t use crowdsourcing for its core applications. Johnston of uTest says he’s willing to look into value-add security review for his test group, so long as there’s value there.
Overall, Lepeak would like to see some simple, low-overhead form of security review embedded in all crowdsourcing best practices, so long as the reviews are balanced against the risk profile of the applications being developed and are not overly prescriptive.
“If you’re a security professional concerned about crowdsourcing, you’re not going to stop it,” Lepeak says. “These shops are booming in the era of scarce resources, so security professionals need to prepare and pick their battles.”
CROWDSOURCING: Tips to consider:
- Assess the criticality of the application and its security requirements.
- Determine if crowdsourcing is the best approach.
- If it’s a secure application being crowdsourced, define security requirements for the application up front without being too prescriptive so as not to overly-limit the third party.
- Confirm the level of testing review the crowdsourcer can apply and compare to your requirements. If you need code development and review certifications and documentation, crowdsourcing may not work.
- Use crowdsourcers with good reputations for quality, adherence to standards, etc.
- Don’t expect skills certifications – the community is skills and training based, but most are not certified.
~ Source: Stan Lepeak, managing director of global research for EquaTerra
Illustration by Doriano Solinas