Today, NSS Labs published one of its Comparative Analysis Reports on Web Application Firewalls. The publishing of this report will likely prompt glowing press releases from other vendors, each one citing the “security effectiveness” and affordability of their solution.
The fact that Imperva is publishing a blog commenting on its NSS Labs test result should tip you off to the fact that we’re not thrilled with the results of the comparative test. Here’s why: The test results do not accurately consider the true value of having a low false-positive rate.
Why does that matter, you ask? It matters because how a WAF handles false-positives has everything to do with its accuracy. And a WAF’s accuracy has everything to do with its security effectiveness.
Imperva’s SecureSphere WAF has an extremely low false-positive rate, while virtually all of the other vendors included in the NSS Labs report have comparatively high false positive rates. With a focus on performance rather than false-positives, they tend to block any and all suspicious traffic.
When performance is weighted over a low false-positive rate, customers get blocked from being able to access web sites and applications that they may rely on for critical services. This is bad for business; a false-positive in many systems is a death blow which can cause a company and its customers to lose time and money.
When ‘outages’ occur and customers are blocked, WAFs get turned off or taken out in order to prevent further interruption of business. The end result is a powered-down product with a 0% ‘Security Effectiveness Rating’, and a security buyer out however much money they paid for the ineffective security solution in the first place.
The graph you’ll see from NSS Labs and from the other WAF vendors only reflects what NSS tested at an initial state, and does not accurately reflect the medium- and long-term operational impact of false-positive rates. Imperva’s customers have reinforced the importance of accuracy time and time again when they purchase our product to replace other false-positive prone products.
If NSS had run more advanced tests that better represent what customers see in the wild, the dots on the graph would have drastically different positions. Some of the products listed would have fallen into the 70% range on the Security Value Map, and SecureSphere would widen the gap between products. From our work with NSS Labs during the testing process, we know they evaluated the WAF products for the following protections:
- URL Parameter Manipulation
- Form/Hidden Field Manipulation
- Cookie/Session Poisoning
- Cross-Site Scripting (XSS)
- Directory Traversal
- SQL Injection
- Padding Oracle Attacks
Only the basics were tested, and these basics do not adequately represent the real-world scenarios we see every day. Here are some of the protections that we always test our product for and that were NOT part of the NSS Labs testing process:
In our experience, our customers regularly see attacks in the aforementioned categories. Looking at recent well known attacks, such as Slowpost, Heartbleed and Shellshock, customers know how critical it is to have a modern, updated WAF that protects against modern threats. In fact, the tests conducted by NSS barely touch on industry accepted security standards that our customers adhere to, such as the OWASP Top 10, DISA STIG, SOX, HIPAA, or PCI requirements.
The protections that are standard features in the SecureSphere WAF help security admins deploy, tune and protect systems, lower the false-positive rate, lower TCO, and make the product easier to manage. By not testing or including these critical features, NSS is not accounting for the positive impact they provide, resulting in a narrow view of what it takes to keep a WAF operating in production environments.
NSS Labs is in the difficult position of trying to test security products in a Lab environment. While we appreciate all the hard work that went into this new WAF report, we’re disappointed that their ultimate criterion was performance and the perception of a low price point over real security effectiveness and a compelling TCO.
It appears that Imperva brought a gun to a knife fight. By defining performance in its broader sense—not just speeds and feeds, but a real commitment to accuracy–we ensure our customers can keep their businesses up and running without sacrificing their security.
Try Imperva for Free
Protect your business for 30 days on Imperva.
