As quantitative methods have migrated beyond the physical out into the social sciences over the past several decades, the developments have often been proffered and adopted with undo sanguinity. Much of this is certainly due to hope that the “unreasonable effectiveness of mathematics in the physical sciences” would transfer. But this itself is unreasonable. The problems faced in financial investing, social planning, and so-on are not subject to proof-testing via the scientific method. Even when they are elegant and compelling, theories regarding aspects of the dynamics of financial markets or social structuring cannot be isolated and subjected to controlled experimental verification. Also, because the driver of all social phenomena is human behavior, no theory can be based on any principle that can expected to hold universally.
As a result, statistical inference is heavily relied-upon. Now that’ big data’ records almost every day-to-day behavior we exhibit , the power this approach offers has become greatly magnified. But since it still cannot link behavioral responses to causality, it needs to be utilized with a healthy familiarity of the problem under study, and with a regard for the perfidy, such as overfitting, that can arise depending on how the statistical modeling was implemented. Statistical models are only indicative tools. To be illuminating, they must be applied and interpreted with a proper regard for such potential ambiguities.
There are many people picking up data science now. The good ones have a rationale for selecting and applying a particular type of model to a given problem, and make earnest efforts to interview stakeholders a-priori for important domain knowledge regarding its characteristics. Practitioners must learn as much as they can about each new setting before they bring data science to bear to resolve losses and inefficiencies.
We provide data science services, quantitative analysis, strategic analysis, and consulting. My areas of depth are investments & financial risk management, economics, statistics, and mechanics. There are many areas that need solutions where our analytical means can be brought to bear. If you are interested in hearing what we can do to help you find such solutions, please contact us.