A few minutes after I have posted UTOPE, the HBR October 2012 magazine fluttered on my desk.
It is about Big Data - how vast new streams of data are changing the art of management.
(no question on apophenia aspects ..)
Data streams all around that will radically improve our company's performance ... and no surprise: Data Scientist will be the sexiest job of the 21st century - akin to quants of the 1980s and 1990s.
I am working with machine learning methods since 1990. And I am full of enthusiasm about the developments that have driven the field of data mining. And I have conducted industrial data mining projects that improved products and processes (from ski production to pulp and paper, energy production to metal forming). And I agree that with the availability of Big Data, the insight from complex systems research, machine learning and the availability of the new computing muscles (to make larger simulations), the behavior of interacting agents can be analyzed on a much larger scale - and better managed.
But still, to create knowledge and meaning from data needs special skills - especially when the extracted information shall be computational and understandable.
Rule bases are understandable, but hardly computational. Fuzzy rule bases are both. Often in inverse problems, small perturbations of data lead to horrendous errors - if you not know how to regularize.
What is your problem? Supervised or unsupervised analysis? How do you generalize? Are you capable of transfer learning techniques? Do you transform high-dimensional data into flat maps, like with SOMs (Self Organizing Maps) or create high dimensional parameter spaces from simple trajectories (like with Support Vector Machines)? Or apply neural networks to everything?
I am afraid, a whirlwind of activities is not enough ... we still need to rely on the intelligent combination of analytic and data driven methods and cross-model testing and multi-method approaches.
But no doubt ... a caravan of talented people will follow the call ... Big Data everywhere.
