What I'm talking about is predictive analytics.  Not business intelligence, reporting, or even OLAP--but techniques for finding patterns in data.  Sounds kind of trivial, except for the fact that countless hours are spent by just about every medium-sized companies and up looking for those patterns so that they can address them.  Not just that, but you can then apply those patterns against incomplete data to literally peer into the future to some extent.  Then tweak something and see what the result will be, and so on.  It's about moving from a gut-based approach to decision-making to a vastly more informed approach that almost makes things a no-brainer.

I've always had a vague interest in this area, but I honestly never knew enough about it to fully comprehend it.  The farthest I ever really got was OLAP.  What's funny is that a lot of times OLAP is just used to feed this stuff data--that's pretty cool when you think about it.

The bottom line is that you can see into the future within a given range of probability.  Ever see that movie with Nicholas Cage where he can see 15 minutes into the future?  It feels a little like that.

What sucks is that I can easily see how such a powerful thing can be misused.  Heh, that almost sounds super-villainish if it wasn't so true.  The real estate valuation algorithms used by banks, quants, and hedge funds were built on similar technology, and what's worse is that they actually worked very well--until they didn't.  The old patterns that they had discovered stopped working.

The data was feeding on itself, and it accurately predicted rising prices.  This begat derivatives.  Derivatives begat hundreds of trillions of dollars in money which is now quite literally disintegrating.

But then the consumer debt celining was reached.  They failed to account for the fact that there wouldn't be enough money in existence to keep the patterns running.

They missed an input in their model, and it just so happened to be a doozy--the consumer debt limit.  Unfortunately when their model stopped working they levered up instead of shutting it down.  They could have turned it off when it stopped working, but got greedy.  Losing 99% of your revenue stream instantly is not an appealing proposition.

Tweak a probability threshold here, a standard deviation tolerance there, and pretty soon you've got an honest-to-goodness economic apocalypse on your hands.

Anyway, I guess the question of whether this technology truly geeks you out or not lies in whether you believe it works or not.  If you believe that things generally run in patterns, then you have to pay attention to this.  If you don't, you can safely ignore it as quackery.  I think the existence of spam pretty much puts that issue to bed for me.  If you can get people to give random Nigerians their bank account numbers, to the point where this activity is profitable for you, then there is obviously a lot of brainless activity going on out there running on autopilot.

Makes me wonder if some vertical other than finance could be so optimized like this that there's explosive growth in the sector.  The only catch is you'd really have to look out for the turn where you've reached maximum potential and not be greedy.  This would require that the technical guys watching the system be allowed to turn it off at some point without the executives who can override the decision.  I don't know if that's within the realm of possibility--at some point you'd just have to say "ok, profits are going to stop growing now, sorry.".

I'd like to talk to the teams at the investment banks who use this stuff.  I wonder if they're still like "yeah, this is really cool stuff I can't believe it works" or more like "aw crap we just ran over the economy... I told you this was a bad idea..."

Now, if only it could clean my garage...