Yes, we have got massive data and more are streaming each second from many sources around the world. And there is a lot of computational knowledge that has been accumulated in our civilization. And there are methods so extract knowledge from data.
How to combine them to get big answers and insight. In general, it is practically impossible to create a meta-method that looks into data and the knowledge base and combines them adequately.
The exception: special domains.
If we know nothing, we use data driven methods only, extracting models in a "learning" step and apply those models to get insight in a running step.
So, selecting the adequate computational knowledge base is already a big project.
This is what we did in UnRisk. We looked into the data, identified them as financial objects, contract features, schedules, conventions, market data and what have you and selected or created the computational knowledge (models, methods or algorithms) that is effective end efficient in transforming those data into prices, risk spectra calculated from portfolio across scenario simulations, VaRs, VaR deltas across risk factors and many other useful results.
It really needs essential math and advances numerical schemes.
In our UnRisk FACTORY all input data, results and useful internal data are made persistent in a data base over the whole history of any valuation of anybody who has a defined user role.
This is implemented in a hybrid environment of C++, Java, SQL and Mathematica.
The communication with the FACTORY: interactively manipulating from a web browser or automatically by scheduled tasks.
If you feed the tasks in the FACTORY with input streams it will produce tirelessly structured output streams. You can view them in the FACTORY web front-end or process them.
UnRisk/Mathematica is the sophisticated programming language that is provided in UnRisk-Q.
It enables developers to use all UnRisk and Mathematica core algorithms from instrument and portfolio valuation, VaR calculations over comprehensive math and statistics functions to data analysis and dynamic visualization. It includes constructs that link to the FACTORY data base to enabling the creation of other knowledge bases that transforms this massive data into individual insight and so on.
So, with the FACTORY and UnRisk-Q, having everything made computable, you can create insight for, all type of decisions you need to make.
Computation is a strong concept. Especially, if it is intelligently combined with data management.
