Uncertainty has a very clear meaning and is based on probabilistic concepts going back over 300 years. Uncertainty is not accurately quantified by accident, or by tweaking hyperparameters of a deterministic regression, including ordinary kriging, or machine learning (ML) model. Systematically changing input parameters provides an interesting understanding of sensitivity but do not provide an accurate and precise prediction of uncertainty. Specialized ML techniques may assess uncertainty through their intentional design, including a method cited below. This blog will discuss what constitutes correct uncertainty, how to calculate uncertainty and why simply generating different predictions is not uncertainty.
The state of nature is uncertain. The future, another World Series win by the Jays, the resources of a deposit with few drill holes – they are all uncertain. The challenge we have as scientists is how to express uncertainty. This was investigated thoughtfully in the Scientific Revolution when thinking transitioned from belief and religious views to more empirical and mathematically based views. Probability was accepted as the means to express uncertainty.
The Frequentist and Bayesian views of probability inference are similar in their understanding of what probability means: the proportion of times something will happen under similar circumstances. When we say a volume of rock has a 40% probability of ore, we mean that 40 out of 100 identical volumes, in similar circumstances, will be ore. In the practice of resource estimation, probabilities are fundamentally proportions. This permits straightforward checking and validation.
A standard practice within RMSP workflows is to perform cross-validation, k-fold validation, or validation against a fully held back set of drill holes. Drill holes are left out and the uncertainty is predicted at those locations, then the uncertainty is checked. Do 80% of the held back true values fall within the 0.1 and 0.9 quantiles of the predicted distributions of uncertainty? Do 20% of the true values fall within the 0.4 and 0.6 quantiles? The uncertainty is deemed accurate when the correct fraction of true values falls within the predicted probability intervals. See below: a probability plot highlights a probabilistic estimate with the corresponding true value. Iterating over all cross-validation estimates, for the green region, we expect the true value to fall inside 40% of the time, and yellow region, we expect it fall in 80% of the time. The corresponding accuracy plot would ideally lie on a 45 degree line.
For an estimator that produces estimates of uncertainty that are too narrow (for example using too much continuity in the variogram model), or estimates of uncertainty that are much too wide (for example too much short scale variability in the variogram model), this will be visible in the accuracy plot.
Accuracy is not enough; we want precision! Precision is the narrowness of the uncertainty. For a continuous variable, the distributions should be as narrow as possible. For a categorical variable, the uncertainty should be as close to 0 and 1 as possible. This is measured by variance and entropy. A professional resource modeler will insist on accuracy, then strive for the greatest precision possible. Accurate and precise predictions of uncertainty are possible with the careful application of (geo)statistical tools developed and tested over the past 40 years. It is unfortunate that the very clear meaning of probability is lost by some. There are those who claim to predict uncertainty by simply varying the input parameters of a deterministic prediction model (regression or ML prediction). Such prediction models can be very powerful. They can make predictions with low MSE and high coefficient of determination (R2) values; however, they are not designed to quantify probability.
Many unsubstantiated claims were made prior to the Scientific Revolution. Empirical observation and careful validation studies emerged as the antidote to such claims. Let’s see! A proper validation study is a key component of any algorithm or application claiming to predict uncertainty. A trusted party should remove data from a comprehensive data set, predictions of uncertainty be made, then those predictions tested for their accuracy and precision.
There are other ML algorithms, such as quantile regression, Bayesian Neural Networks and Monte Carlo Dropout, that may generate valid probabilistic predictions if carefully calibrated to match the target distribution. These are the subject of research in resource modeling (the interested reader is referred to Kirkwood, C., Economou, T., Pugeault, N., and Odbert, H., 2022, Bayesian Deep Learning for Spatial Interpolation in the Presence of Auxiliary Information for an application in spatial modeling). The scope of their application will emerge over time. The accuracy and precision of their predictions will be checked and compared to current geostatistical algorithms.
Any probabilistic statement should be supported by sound science and a thoroughly checked model. Resource Modeling Solutions (RMS) has the latest techniques implemented for probabilistic resources.
