Competitions can be useful to determine best practices, stress test legacy techniques, investigate new alternatives and determine the circumstances where certain techniques work well. In numerical prediction, perhaps the best-known competitions are hosted by Kaggle where nearly half a million datasets are considered in competitions engaged in by nearly 20 million users. There are over one million notebooks implementing more than 7000 models (interesting how notebooks are the principal mode of implementing data science workflows?!?). Top data scientists from world-class research organizations and companies participate in these competitions. There is some element of chance, but the leaderboards of these competitions are not populated by the lucky; the skilled and knowledgeable dominate the top positions.
The AusIMM is hosting a Parker Challenge at the MRE2025 conference where they aim to find out “How much variability would there be if multiple geologists or geostatisticians all estimated and classified a mineral resource from the same data.” A meta-analysis of multiple submissions would answer this question and will be interesting, yet there are challenges. Are the rules for winning the challenge unambiguous? Are all participants presenting regulation compliant work products? Could the meta-analysis of variable-quality submissions be representative of resource estimation professionals?
If the best practitioners in resource modeling were to participate, then the leaderboard likely would be populated by the most skilled and knowledgeable of those practitioners. A novice team employing advanced technology and inspired by sound fundamentals would do well (and could even win!), but the focused attention of skilled practitioners with appropriate geological background would be hard to beat.
Considering those thousands of Kaggle competitions, there is an overview of the problem, then the metrics of evaluation. The starting point under Evaluation is “Submissions are evaluated based on their...” with a very precise metric that is spelled out. Often, it is classification accuracy, that is, the fraction of predictions that are correct. The leaderboard is often very tight, for example, a difference of 0.83516 compared to 0.83165 for first and second places, respectively. The winner is clear and near winners are acknowledged by a close score.
This brings up my second comment on the Parker Challenge. What exactly are the rules? Could they be audited and compared with three to five decimal place precision? The declaration of a winner should be unambiguous and clear. You get a score, and it is posted on the leaderboard. There is little room for subjectivity (although judged events are completely reasonable if the rules are clear and with impartial judges whose scores are made public in a transparent manner).
A reasonable score (S) may be reconciliation precision, that is, the squared error from predicted to actual monthly ore tonnage (Tore) and ore grade (Zore) over m=1,…,M monthly time periods:
Each term would be weighted by the variance of the actual quantities. There are many other factors including credibility of stakeholders, visual realism, fit-for-purpose models that provide reasonable/best mine plans and schedules, and (of note) the goodness of classification into Measured, Indicated and Inferred. Perhaps Harry’s rules in this regard could be directly assessed? Regarding measured, do nine out of ten actual quarterly tonnes, grade and metal estimates fall within 15 percent of predicted? Regarding indicated, do nine out of ten actual annual tonnes, grade and metal estimates fall within 15 percent of predicted? Regarding inferred, do nine out of ten actual annual tonnes, grade and metal estimates fall within 50 percent of predicted? The results of such a comparison would be fascinating.
Not all submissions to the Parker Challenge would be compliant with the spirit and letter of relevant reporting codes. Submissions that do not meet reporting guidelines would have a random influence on the meta-analysis of the submissions. Perhaps submissions should self-declare, or be assessed, as to whether they would be compliant? Noncompliant submissions may be the outliers. In any case, the results will be interesting.
There are challenges with the Parker challenge, but it should be great fun. Harry would have appreciated the nuanced discussion we are having now. He may have given a small Harry smile and made a Zen-like comment such as: Winning or losing the Parker challenge is not the point. The challenge is to appreciate the beauty of geological variability and realise your insignificance.
