Insight Discovery

The purpose of any data analytic project is to improve the understanding of the past, optimize the present, or forecast the future. In each of these cases the result is based on the insight gained from analyzing the data. The popular belief is that to develop an insight the only thing you have to do is to fit a mathematical model to the existing data. As this presentation shows, for small data sets it is quite easy to fit a very large number of very different models. Even for very large data sets the number of models that can be fitted on that data is relatively large at least if we account for the measurement errors. There might be many aspects of an insight but the fundamental component is unique otherwise nature would break its own laws.

In epistemology Knowledge is defined as justified true belief which implies an absolute degree of verity conceivable only ex-post or in some very limited cases. However, for the modern corporation even partial Knowledge could have tremendous value if the degree of verity could be correctly estimated. Since Knowledge is built in a step by step process involving sampling, codification, and confirmation the degree of verity can be estimated before the process is finished.

Insight is the verified justification of Knowledge. It results thru a process of abstraction and ex-post verification over the full range of underlying Knowledge. It allows forecasting the Knowledge and building the frame of scientific discovery.

Sometimes due to limited data, measurement errors, and modeling limitations it might be difficult to choose the best mathematical model. Under these circumstances the best approach is to estimate the level of uncertainty and adapt the strategy to it. Take a look at this and this presentations to find more.

This application shows the result of one of our insight discovery algorithms implementation. By scanning numerous public sources it builds a map of the DNA origin of European populations deciding how to validate and combine sources of partial knowledge to answer a question.