NewGen: New Generation Crawling for Secure Web

The web is a key enabler for today’s ever more digital world. Our societyrelies on the web to support the economic, governmental, and militaryinfrastructure. Yet securing web applications remains a grand challenge. Thestate of the art in web crawling and security scanning falls short of deeplypenetrating web applications.

A major roadblock is the crawlers´ limitedability to pass input validation checks in web applications, when data ofcertain format, such as email, phone number, or zip code must be suppliedbefore proceeding.Our project NewGen: New Generation Crawling for Secure Web, develops aprincipled approach to deep web crawling and scanning.

The key idea isto leverage string-based constraint solving, based on satisfiabilitymodulo theories (SMT), to infer suitable input from input validationpatterns discovered in web applications and thereby automatically passinput validation checks.

This goal requires addressing the researchchallenge of reasoning about regular expressions and JavaScript by SMTsolvers.

NewGen will develop the theory for this challenge and put itinto practice, leveraging the Black Widow web crawler technology, developed bythe PI´s group, and the OSTRICH SMT solver technology, developed by the co-PI´sgroup.

We will demonstrate that NewGen will boost both code coverageand vulnerability detection, with respect to the state-of-the-artweb crawlers and scanners.