Visibility

How many security people have had near total visibility into their critical regions of their networks?

For a decent-sized enterprise trying to show a profit, this is a difficult challenge.

Well, with a very large deployment of Snare agents, syslog streams off firewalls and authentication servers, some scripting magic, and a ton of backend AWK processing... I'm getting near total visibility.

And lemme tell ya... it is a frightening thing.

What is good pen-testing?

Over my last couple of decades in InfoSec, I've been both a consumer and a provider of pen-tests. As a consumer, I've seen quite a few tests that I've simply thrown back in the face of the consultant requesting a do-over, mostly because the data was so inaccurate that it was unusable. I'm writing this to raise the bar on what is being offered. And rather than curse the lameness, I'm lighting a candle.

Wait, what do I mean by pen-test? Well, I'm lumping a wide spectrum into this post. So when I say pen-testing, assume I could also mean vulnerability testing/scanning, perimeter scanning, web app vulnerability analysis. For me, it all falls into the bucket of technical risk analysis. Some tests are done for certification purposes (PCI, Cybertrust), but I'm talking about actual value. And that's, providing a technical risk analysis.

Let me decompose this technical risk analysis business. A risk analysis requires several pieces, namely: 1) asset identification, 2) threat analysis, 3) vulnerability analysis, 4) impact analysis and, 5) control effectiveness analysis. Most pen-testers deliver a vulnerability analysis. Good testers do the threat analysis and ask for the asset identification before they start. The best spend a lot of time upfront to help figure out potential impacts and do a decent job on control effectiveness analysis. And yes, this means you should spend a bunch of time talking and analyzing before doing. Trust me, it pays off.

Of course, it's worthy assumption that an external pen-tester will get some of these things wrong. Either their guesses about impacts to an organization ("hey, we don't rely on that service so who cares if it's DOSed") or more likely the vulnerability assessment itself. Specifically, many pen-testers run scanning tools, jam the results into a report and call it done. Well, a lotta scanning tools do basic banner grabs on listening services and assume from there. A good pen-test report always shows its work. When I was a consultant, what I often wrote in my reports was something along the lines of "We initiated an TCP connect on port 5678 to IP address XYZ and sent along a packet stream MNOP and got back result ABC. This matches vulnerability #123. We verified by doing PQR and then reviewed JKL, which leads us to believe this is a high-risk root compromise vulnerability." And of course, by showing your work, the client can reproduce your results and test later to see if the hole is closed. Bonus points for including source code for scripts in the report to facilitate easy test duplication. If you can't be accurate, at least be transparent. And if you're extremely inaccurate, the act of writing up the assumptions should be a red flag for you.

All good pen-test reports will be shared everywhere by the client. Sometimes they will go to the client's clients and almost always they will go up the management chain. Keep this in mind when you are writing the analysis. Don't be glib, don't hide your assumptions, provide concrete proof, and don't ever assign blame. Contrary to what your Eighth Grade English teacher told you, passive voice is your friend. "This server was found to contain a database dump full of social security numbers. I have no idea how it got there and who fcked up, but this what we found on this day at this time using this tool. Oh, and here's a screenshot to prove that I'm not lying."

Also, you need to include an executive overview. Not only is this the shortest part of the report, usually 3-4 paragraphs, but it is also the hardest to write. I usually follow the BLUF method when writing these: Bottom Line Up Front. Open with a "we did a scan on this date/time against these assets. We found one serious vulnerability, which we alerted staff about during the test and it was immediately corrected." - Hint: make the client look good whenever the opportunity presents itself. These people sign your checks. Continuing - "We also found several other low risks..." Then the next paragraph summarizes those risks in as plain language as you can possibly can - be sure to speak of likelihood of exploit and potential business impacts. Executives will only read the summary, yet will be making a decision to spend money based on it (possible spending money to hire you to fix them or scan more later). Be concise and remember your audience. I usually spend 6-8 hours writing the summary, and spend the entire engagement thinking about what will go into it.

If there is a compliance requirement (and bingo, there almost always is), there should be a separate section highlighting the gaps found there as well. Usually a client considers this a high-priority (we need to pass PCI!) but I've found the compliance junk provides the least value. Heck, the reason pen-testing is in most compliance requirement lists is so that a good technical risk analysis is done regularly. But anyway, it is usually a requirement, so there you go. And if it's a key requirement, then the results here should also bubble up to the top of the executive overview - "Blomo corp. appears to be 95% compliant with RHINO rules with the exception of the weak password on the mail server."

Okay, I've ranted enough. I just wanted to pop off my experiences and wants regarding pen-testing. Who knows, maybe in a few years I'll go back into consulting and do these again. Until then, I'll just keep rewarding the competent professionals with more business and keep shunning the lamers.

Thought experiment

Economists say, incentives matter.

Here's a thought experiment -

What if we did away with all the security regulations and rules. No more GLBA security rules, no more HIPAA privacy, etc. And for contracts and b2b relationships, no more SAS-70's, no more PCI, no more ISO certifications.

Just one new rule - each person who's confidential information is breached gets a cash settlement. For example, your credit card ended up with some hackers. Here's $250. And if we didn't warn you or tried to cover it and you later found about it the hard way... well, now we gotta pay you $2500.

That's it. Let each organization figure out how to secure themselves and what the trade-offs are. Next step in my hypothetical world - organizations would need to post bonds or have insurance to make sure they can pay people off when breached. And then the insurance companies will come up with criteria for good controls. And with all the payoffs, they'll be able to build actuarial tables to see what works, what doesn't.

Hmmm.

Compromised integrity

Two words we often use in InfoSec... compromise and integrity. But their origins outside of technology are of interest to me today.

Specifically,

Compromise - To expose or make liable to danger, suspicion, or disrepute.

and

Integrity - Steadfast adherence to a strict moral or ethical code.

What do I mean by this? I mean asking yourself how what these things mean to you as a member of a community (substitute nation, organization, company, family).

In the tech world, when a machine gets compromised, there is often a battle between the security team and... well, everyone else. Security says, once it's compromised it can't be trusted again - reformat and rebuild from scratch. The techs often say "that'll take hours" or worse, "that'll take days" and of course, it's a critical system. Management agrees, after all, it's going to cost us money to take those services down. And no, there are no hot spares or quick reloads available. In the end, the risk is made clear and the machines are replaced.

But what happens when the human soul's integrity is compromised? Many security folks have been a part of internal investigations. Often evidence isn't entirely clear how much an insider may be compromised. Did they make a one-time mistake? Or are they actually malicious? Again, the security folks often recommend termination and replacement. Again, there is often some (but not nearly as much) push back - especially if that person is critical to the organization.

Those who read Geek Girl Detective know how devastating it can be to a company when someone in an important position becomes compromised. It could be the end of the organization itself. But again, is it better for an organization to implode, minimizing the damage of the compromise, or explode with a devastating shockwave of litigation?

I've been told recently that no person should be replaceable. Same is true when designing systems. However, in either case, is this always feasible? No. But it's on the table as something that should be thought about and planned for - People and computers.

What is the bare minimum we can do and still operate as a business?

In her column, The Agency Insider, Linda McGlasson writes in a post GLBA and Security Avoidance Questions - Why Are We Not Surprised? about GLBA compliance.

Thee post is about her dismay when hearing "What is the bare minimum we can do and still operate as a business?" from many large banks. She goes as far as saying that hearing that is "the number one sign that there is something wrong with the approach many financial services companies are taking on GLBA."

Okay, granted on the surface, this statement does appear to be like sloppiness, cheapness, and/or general dereliction of duty.

But wait, let's unpack this:

Is she saying that banks should spend MORE than necessary on GLBA compliance?

Does spending more on GLBA compliance entail better security?

Audit checklists and industry regs usually do not always entail improved risk management. But hey, for argument's sake, let's just assume GLBA Compliance = adequate security.

Now, let's restate and clarify:

"What is the bare minimum amount of risk management we can do and still operate as a business?"

But what's wrong with adequate (or minimum amount). It's that tipping point where it becomes too costly to protect an asset than to lose it. That's risk management and just plain dollars and sense.

Okay, but what is that bare minimum? How do you know what that is?

Wouldn't knowing what the minimum amount of risk management needed imply a thorough examination of risk and the value of the protected services and assets?

So to truly make that statement, banks will have be doing some pretty darned good risk assessment.

And to choose the bare minimum, means they are making an informed decision about the tradeoff between business value and risk mitigation.

The only problem I see with all of this is banks should not be asking their auditors "what is the bare minimum I need to do?"

They should be asking their security people.

And they should be answering in a manner that makes sense to someone who's job it is to choose how money is spent for the overall good of the organization.

Can you buy security? How can you make things better?

So I'd twittered a bit lately about the question of organizations "buying" security. What I really wanted to know was if an organization was taking unnecessary InfoSec risks ("insecure" in lay-man's parlance), then they could simply bypass the whole cultural change thing and just write big checks to get improve their risk management ("become secure").

I've been asking this question myself a lot lately. I've spent nearly a decade as a security consultant, trying to fix organizational security programs (both for profit and non-profits). I've also spent about that much time working inside companies as a either a network guy with security responsibilities or a pure-play security guy. In my current job, I am *the* security guy. The buck stops with me. Now, with that in mind:

When I was a security consultant, I noticed there were basically two types of customers:

• The organizations that were already practicing pretty good risk management but wanted to improve
• The organizations who were being forced to be "more secure"

Naturally, the first type of organization made the best kind of client. It was fun to come up with innovative solutions to push them from a B to an A. We just ate up those technically complex security challenges. And yes, we were very successful. Usually these were complex, highly-regulated organizations like hospitals, law-firms, banks and public utilities.

Now the second type of organization, the ones forced to be more secure. These are the folks who've failed an audit, experienced a breach, or have an important customer dissatisfied with their security. Many of these organizations produced online products or services, and they "simply didn't have time to worry about security."

Not surprisingly, these were often the least successful clients of my consulting career. Advice was ignored, reports were shelved, warnings were rationalized away, blame was shoveled about. And things rarely improved much beyond some cosmetic fixes. They wanted to "buy" a fix to their security problem. Sadly, a lot of money was often spent on either new hardware or expensive audit reports.

As Jerry Weinberg said "Things are the way they are, because they got that way." And that's very true in these organizations. Their risk management processes are seriously messed up. And hiring a bunch of consultants and buying a bunch of tools doesn't seem to make a dent in that.

Over time, I quickly sussed out some warning signs:

• Not invented here syndrome. When given a suggestion for a new process or tool, you are often told "That won't work, we're special." Hint: the only things that should be unique should be your cash cows. Usually wasn't true for IT operations or infosec.


• Ill-fitting and ignored policies. "The security policy says we can't do that. But we need to do that. So we just ignore the policy." And thus begins the precedent to ignore everything in the policy. Usually a soup-to-nuts rewrite of policy with copious re-education to regain the user's trust is needed here. Difficult to do, even more difficult to do as a consultant.


• Lack of defined process and/or roles for critical things. These are the folks who are surviving because they have a lot smart people thinking on their feet. No time for docs, no time for process. We can figure out as it comes at us. Unless you don't know anything about security.


• A culture of reactive fixes. Managing risk just becomes another reactive fix. After we're done patching that hole, we can get back to The Real Work. Right?

Of course, there's the obvious: evidence of breaches, failed audits, high turnover, etc

Compare this to the things I see in the organizations who are doing a decent job of managing risk:

• Change management, especially for critical services and components (hat tip, Gene Kim)
• A clear understanding of their environment, the risks present in that environment and an awareness of how well they're dealing with them.(hat tip, Alex Hutton)
• Actually looking at their logs and understanding them.(hat tip, Anton Chuvakin)
• Someone who's primary focus is security. And this person is internally credible and willing to learn more.
  
• Reliable infrastructure. May not be the best, the fastest, the latest or the most flexible; but they know how it behaves and they know where everything is

Conversely, many of the aforementioned "obvious" signs can mislead here: places with strong security can experience breaches, failed audits and high-turnover.

But what about those broken organizations? Now we circle back to the original question? Can they buy thier way out of the hole? Not in my experience. External consultants fail... "Like trying to throwing sod down on cement and hoping it'll grow" as a colleague used to say. Heck, even hiring in good security folks and having them try to turn the ship is mighty tough. Of course, everyone says you gotta have management buy-in if you want to effect cultural change. And that's usually the cognitive disconnect you have in these kinds of organizations.

I could go into another long list of the types of executive paralysis that I've seen. It usually starts with "We really care about security, do what you gotta do." And it ends with endless trickle of dying projects that go nowhere.

Not to be all doom and gloom, but I've had some success starting to fix these kinds of organizations. But not from the outside - meaning the answer to my original question is "no", you can't "buy" security. But I've made change from the inside with slow and steady grinding away at the old culture. Gaining trust, garnering political will and slowly building up the layers of paint.

The two things that have been helpful in selling security culturally have been:improving system reliability and increasing organizational agility. Reliability is easy... that's the A of the CIA triad of security. The second is more interesting. At some point, security's just another characteristic of the overall health of operations. Organizations that are doing a poor job of managing risk are not very agile. They can't to market changes without increasing their risk exposure (usually exponentially). They can't scale very well and they can't hire/replace people. Look back at the warning signs and the positive attributes. A lot of them can tie directly to agility. And I find it ironic selling security as an agility improver when security is often cited as something that "slows us down". It's also ironic that the organizations that need agility the most (the software product and service producers), have it the least.

That's the end of my ramble. I'm going to think on this more. Feel free to comment.

Still alive

Just been super busy with work, family, life, Heidi, etc.

I've got a whole bunch of stuff lined up to post... just gimmie some time.