Tuesday, October 27, 2009

Why do pen-tests suck?

I was just listening to the Exotic Liability Podcast and once again, Chris and gang were lamenting the sorry state of pen-testing. While I've ranted before on the poor quality of the risk reporting in pen-tests, EL was lamenting the watered-down nature of most testing.

Specifically, they asked "Why are pentests so limited?" And that's true. In most external security testing (which includes both pen-testing and vulnerability scanning), there is often no intelligence gathering, no social engineeringn testing, and no physical security testing. Of course, no "cheating", like hitting DNS or business partners, either. Very often the scope of the attack is limited in both targets (only touch these assets and these IP addresses), and limited in time (you can only attack us during this timeframe and spend only 40 hours on the testing). Implied by these restrictions, include restrictions - no time for extensive manual testing, deep analysis, or reverse engineering.

A water-down test of your defenses means a myopic analysis of the strength of your perimeter. And remember, even in the best of the times, security testing only tells you two things: where some of the holes are and a measure of the skill of the attacker. Passing a security test never means you are secure. The more "real world" your testing, the closer you approach some kind of reasonable measure of useful information about possible holes. But why water them down?

Well, the obvious reason for the reason for these limitations is not wanting to spend a lot of money on consultants. Of course, I think this is a distractor. Having been a tester and now, one who hires testers, I can tell you a bigger reason is not wanting the liability. Consider, most testing that is going on right now is because of compliance. PCI requires vulnerability scanning. Most organizations acting as custodians for other organization's data are beholden to demonstrate "best practices" - and that includes pen-testing. And here's the real rub - many auditors and customers want to see the results of those security tests.

As a tester, I've also been told by very large e-tailers that they were limiting the scope of our engagement not because they knew we wouldn't find anything, but because they knew we would. They knew we would find too many security issues for them to feasibly fix without going out of business. And if they had a report of all those holes, well, now they're liable for fixing them.

So what's a poor organization to do? They need to hire someone to do security testing that has a strong reputation but at the same time, won't do too good a job. Credibility but not competence. Or barring lack of competence, someone who will sell them a testing service that is so cookie cutter that the scope will be automatically limited to the basic scan-and-patch kind of findings. Enter the big organizations, like Veri zon Cyb ertrust, I BM, Hac kerSafe, etc. Yes, there is some collusion there. But hey, it's all about staying in business and meeting unreal expectations. After all, most people don't actually want to pay to have their data protected. At least pay what it would really cost.

BTW, you can lather, rinse, repeat this post for entire financial audit industry. See Enron, WorldCom, Lehman Brothers, WaMu, etc.


christopher said...

Gates and I are giving a speech on this at CSI tomorrow *err..today* 10/27/2009

I think a ton of what you have posted is right on the money. I don't think it is a reason to go soft though. I think that people need to call out those organizations. Kinda like the "Government Watch Dogs."

I know there is no real answer or fix, but i have to believe that education that compliance is not security would be a good place to start. However, every marketer,big5,product vendor and security industry cash leach and their mother... has beaten this into their heads... to sell them more shit!

I rant and rave... with hopes that we can make a change. Our engineers joke that on EVERY test we do, we pick up another hacker or red team tester. This is because we teach and inspire our clients to see through the bullshit and get down to biz in security. I can only pray that everyone puts in the love and starts to turn this shit show around.... even if it is only one client at a time.

Author, Planet Heidi said...

Agreed. Passion is a critical part of success in this job. It's something I talk about when I lecture to new students - it's a tough challenging field, but if you love it, it's fun and rewarding.

I also tell students to make your organization secure first. Then go back and do the compliance junk.