Why do pen-tests suck?

I was just listening to the Exotic Liability Podcast and once again, Chris and gang were lamenting the sorry state of pen-testing. While I've ranted before on the poor quality of the risk reporting in pen-tests, EL was lamenting the watered-down nature of most testing.

Specifically, they asked "Why are pentests so limited?" And that's true. In most external security testing (which includes both pen-testing and vulnerability scanning), there is often no intelligence gathering, no social engineeringn testing, and no physical security testing. Of course, no "cheating", like hitting DNS or business partners, either. Very often the scope of the attack is limited in both targets (only touch these assets and these IP addresses), and limited in time (you can only attack us during this timeframe and spend only 40 hours on the testing). Implied by these restrictions, include restrictions - no time for extensive manual testing, deep analysis, or reverse engineering.

A water-down test of your defenses means a myopic analysis of the strength of your perimeter. And remember, even in the best of the times, security testing only tells you two things: where some of the holes are and a measure of the skill of the attacker. Passing a security test never means you are secure. The more "real world" your testing, the closer you approach some kind of reasonable measure of useful information about possible holes. But why water them down?

Well, the obvious reason for the reason for these limitations is not wanting to spend a lot of money on consultants. Of course, I think this is a distractor. Having been a tester and now, one who hires testers, I can tell you a bigger reason is not wanting the liability. Consider, most testing that is going on right now is because of compliance. PCI requires vulnerability scanning. Most organizations acting as custodians for other organization's data are beholden to demonstrate "best practices" - and that includes pen-testing. And here's the real rub - many auditors and customers want to see the results of those security tests.

As a tester, I've also been told by very large e-tailers that they were limiting the scope of our engagement not because they knew we wouldn't find anything, but because they knew we would. They knew we would find too many security issues for them to feasibly fix without going out of business. And if they had a report of all those holes, well, now they're liable for fixing them.

So what's a poor organization to do? They need to hire someone to do security testing that has a strong reputation but at the same time, won't do too good a job. Credibility but not competence. Or barring lack of competence, someone who will sell them a testing service that is so cookie cutter that the scope will be automatically limited to the basic scan-and-patch kind of findings. Enter the big organizations, like Veri zon Cyb ertrust, I BM, Hac kerSafe, etc. Yes, there is some collusion there. But hey, it's all about staying in business and meeting unreal expectations. After all, most people don't actually want to pay to have their data protected. At least pay what it would really cost.

BTW, you can lather, rinse, repeat this post for entire financial audit industry. See Enron, WorldCom, Lehman Brothers, WaMu, etc.