Monday, December 12, 2011

Where the security rubber hits the operational road


Where have I been? Absent from this blog, that's for sure. Mostly silent on Twitter as well. What up with that? As you mighta expected, life's been busy around here. But I thought I'd give a little more detail on what that meant.

Let's start with some of the problems that had come to a head this summer.

Gunnar had a great point when he spoke of the Top 5 security influencers. For past year or so, I'd already been working extensively with the Dev and QA team - now to the point where QA is taking bug directly from vulnerability reports from WhiteHatSec and
IOActive and developing QA testing procedures from them which is now spotting more holes, faster, and resolving them more deeply.

I already work alongside the DBAs and the Ops team, but I was really running up against a wall regarding just how much I could get done.  Basically, a lot of security projects were getting sidelined in favor of large infrastructure projects and I was starting to lose visibility into the whole process.  Worst, overlapping concerns between ops and infosec, like uptime and integrity, were losing ground to not well-planned horizontal expansion and vertical infrastructure structure issues.

The other big bugga-boo in my life was audit. Enough that I'd ranted a bunch about it at Source Seattle.



Well, I knew in the coming year that we'd be shifting some of our operational and access control models to a more expansive system to accommodate some new business directives. This was going to a problem as our existing set of control objectives needed to be redone from the ground up. And this change needed to come both from infosec as well as the operations team, but so far, they had not come to the table with any big ideas.  As mentioned before, they were too swamped fighting fires and keeping the lights on. This is an even bigger problem as auditing impacts operations as much as security, especially when you're doing SSAE-16 Type 2.  Without needed changes and strong leadership, Ops and Infosec were going to sink together come the auditors next year.

Note on SSAE-16/SAS-70: They're not all worthless. It's naive to assume they're a perfect measure of security and operational efficiency. But's also naive to assume they're worthless. As with anything, you need to do the work and actually read the report - does the scope match what you need to test? Was the audit conducted by observation and testing? Or just documentation review and attestation? A telling factor is how many individual control failures are noted. If the number is zero, then likely the report is scoped too tight to be useful or the auditors didn't do anything beyond interview people and leave.

Lastly, my Corporate Masters, were having problems getting a handle on operational projects.  As I mentioned in the lead, important work was being left on the table, while other lesser, more tactical projects were getting funded.  Operations needed help articulating and justifying critical infrastructure upgrades and aligning them to key business processes.  From my infosec seat, the solution was clear - risk to business objectives was not being defined with respect to operational problems.  Simplified example, conversations like "We need a better storage system because until we do, we can't take on any new customers" were not happening in the right places. Again, the team was too busy treading water to document, assess, analyze, strategize, and communicate. 

Enter me.

Our Chief of Operations wanted to inject a big does of risk management and clear process into the technical group.  Since infosec had been already been doing that in spades for the past four years, he asked me to step in and manage a chunk of the operations team. So about four months ago, I got promoted. In addition to security, I am now in charge of the Infrastructure team.


Believe it or not, I passed on some more lucrative opportunities to take on essentially a doubling of my workload. I'm silly that way.

Actually, it's more than doubled since it's been over a decade since I've done anything deep with infrastructure. For past few months, I've been playing catch up learning about large scale virtualization systems, storage area networks, operational resiliency and IT automation.

It's also more of a chance for me to bake security directly into the existing operational processes. Get my hands dirty and see what is going wrong.  For example, instead of auditing and dictating firewall policy, the team that directly manages network security reports to me.  I can directly see the effects of new security processes and technologies, both good and ill.  I'm on the front lines not just for security incidents (which also land in ops before they are identified and escalated to security anyway) but any other interruption events (which helps me design towards better integrity and availability).  So yes, it's good.

But not all good. Because, as I stated in the beginning, I've been treading water myself. In addition to getting to know the team and the technologies, I've also had to immerse myself in learning a bunch of new things.  Specifically:
I'll blog more in the future as learn interesting new things.

So, where have I been? A: Where the security rubber hits the operational road.