Build vs Buy - the auditor's perspective

Sat through a comprehensive demo of IBM's Tivoli Compliance Insight Manager. Overall, the product is another SEIM, which means it aggregates logs from a wide variety of servers and lets you write queries against data. In short, if your servers are configured to see something and log it, then you can alert and report on it. That's all well and good.

Here's my problem - My requirements include pretty tight change control oversight. I need to be able to confidently tell auditors that I am aware all of unauthorized changes to my systems. Now here's where the rubber meets the road: our team developed a customized change control monitoring system that's part log scraper, part file watcher (ala Tripwire) with some dashes of config dumping-&-diffing. It's laser-focused to our environment, our apps, and the types of work (and mistakes) our Operations team does. It produces a daily report that's mostly readable that gives me a very accurate answer to the question "what changed yesterday". Even when the system has problems, the data is still captured and flags about the errors are usually thrown.

But, and this is a big BUT -when auditors see the report and see that we developed this system in-house the suddenly become very inquisitive. "Oh, it's home-grown. Well, we need to test it." It's not trustworthy. Every piece of the system is in question. Okay, that's understandable and we do our best to deal.

However, if I were to buy this IBM system (or any professional system), would the auditors feel the same way? One would hope they would have some doubts about how the system was implemented and how accurately it monitors. So far in my overview of vendor landscape of these types of products, I've found no particular product has the monitoring coverage we need. So if I were to buy a single system (and I really could only afford a single system of this magnitude), I know for a fact that I'll be missing about 20% of the changes being made on my network.

What I wonder is this: what is the real value of one of these professional change management tools? I suspect it's the trustworthiness of the brand name. I know I've been through this argument before with open-source homemade firewalls versus professional products, but at least the products go through some kind of testing (Common criteria, ICSA, etc). Moreover, that still doesn't address the concept of "best fit.” We all know that in-house works better (but can be more costly to maintain) than COTS products.

For the matter of change control, I felt that best-fit was more important since I needed (according to the auditors) to be able to confidently assert that I was aware of all changes. If I bought something off the shelf, I wouldn't be able to assert that (they're only catching 80%). I could buy something and then implement some homegrown stuff for the remaining 20%, but frankly, the effort on our part is about the same as just writing the whole thing ourselves. Plus we have the added bonus of being to adapt to infrastructure changes better than a canned product.

I wonder how many auditors out there will see the product with it's fancy dashboards and professional reports and go check the box "monitoring - compliant" and never question how well the system fits the environment? I bet a whole lot more than those who will needle me relentlessly on the effectiveness of our internally-developed system. So the real question becomes: is the cost of a canned product worth the cost of making the dimmer auditors leave me alone?