One type of value which really seems to be low-hanging fruit is efficiency.  Incredible amounts of efficiency can be uncovered with proper analysis.  Making things that were previously hard easier, making time-consuming tasks quicker, or making error-prone processes run much more quickly and efficiently.  All of this adds value which is the only way to create real money.  And this is all relatively easy to do when you can see the patterns in your business.

But finding those patterns is not easy.  Experience and knowledge can get you part of the way there, and in fact it's what typically sets apart great executives and leading companies.  But it's also what data mining technology is built to do. It finds patterns in the data, lets you make accurate projections, and do what-if analysis.  It's fascinating stuff and has really made my mind spin with possibilities.

It is, however, highlighting something that has become more and more obvious to me over time--individual tools and technology can only go so far; in order to do the really cool next-generation stuff you also need people who are intimately familiar with the data and, most critically, what it means, to drive it.  

You can have people that know the tools inside and out, but if they're not familiar with the data--how it's generated, what it means, how it applies to the business--and if they aren't involved in the design and implementation of the technology side, the whole process won't add much value.  The really cool stuff like the predictive technology and so on requires that you know not just the tools, but the domain where it's being applied as well.  

In order to truly leverage both the technology and the experience you need them both in one person--or two people at the most.  I strongly believe that more than two people trying to do this function would cripple progress more than help simply because of the attention to every single detail required to pull this off successfully.  There is so much inter-dependency between the two sides that injecting a layer of middle-management or large team would simply kill the process dead.  (Or at best turn it into a multi-year project with dissappointing results.)

The dual skillset of both the technology and the business experience is a very unique one for one person, or even two people working well together as a team, to have.  Here is an area where you simply can't automate the person away, he's an essential part of the system and in fact the most important cog.

The temptation here is to find people who understand the technology but are light on the business side of things.  After all, the experienced businessman who is willing to pick up a big fat book on predictive analytics and slog through it must be a very rare breed.  And the business that is willing to pluck a statistician and insert him into the business side of the company as on-the-job training must be extremely rare as well.

Even the banks that were relying on this technology to create the derivatives market--now worth hundreds of trillions of dollars--didn't get this part right.  Paul Wilmott, a financial modeler and quant genius,  wrote last year (well before everything went totally to hell, in a display of uncanny foresight):

Part of the problem is that many of the people who produce mathematical models and write books know nothing about finance. You can see this in the abstractness of their writing, you can hear it in their voices when they lecture... Sometimes they don’t want to understand the markets, somehow they believe that pure mathematics for its own sake is better than mathematics that can actually be used. Sometimes they don’t know they don’t understand.

Banks and hedge funds employ mathematicians with no financial-market experience to build models that no one is testing scientifically for use in situations where they were not intended by traders who don’t understand them. And people are surprised by the losses!

If the banks and funds who have (had) virtually unlimited pockets could not locate these specialists, what hope does the average enterprise have?

It seems like one of the most beneficial investments a company could make would be to start a program to grow individuals with these all-important diverse skill sets.  This would require a very unique career development path, one that a company would have to actively set up and nurture.

These people must surely exist, and in fact I suspect that Google has several of them when I see stories like this one, where Google applies data mining technology against their employee population.  

The Internet search giant recently began crunching data from employee reviews and promotion and pay histories in a mathematical formula Google says can identify which of its 20,000 employees are most likely to quit.

Google officials are reluctant to share details of the formula, which is still being tested. The inputs include information from surveys and peer reviews, and Google says the algorithm already has identified employees who felt underused, a key complaint among those who contemplate leaving.

I can tell you with a pretty high level of confidence that they've probably run a neural network or decision tree analysis against their HR database.  I imagine they're treating it like a state secret strictly for PR reasons.  But the fact that this is big enough news for the Wall Street Journal to do a story on it speaks to how rarely this technology is being used.

I would be curious to these individuals' resumes, what their titles are, and how much these people are paid.  They must be some of the most profitable people in the entire world, and it would be interesting to try to find a way to leverage them better.