Quantifying uncertainty with structured expert judgement
Quantifying uncertainty with structured expert judgement
Expert judgement may be required to inform a range of tasks under uncertainty, including, model development, estimates of probabilities and quantities, and to inform prioritisation tasks. In this workshop we concentrate on the elicitation and aggregation of expert judgements related to uncertain events and quantities. For quantitative estimates, a common approach is to elicit a point estimate. However, there are strong theoretical and practical arguments to say that the proper representation of experts' knowledge about uncertain quantities is through probability distributions.
Repeated evidence also indicates that these judgements should be elicited from multiple experts. However, challenges arise when the model requires a single probability distribution, which means that the various judgements must be aggregated. This aggregation can be done by the experts themselves, through a process of interaction that is designed to encourage consensus (behavioural aggregation). Alternatively, it may be done externally, by applying an aggregation formula (mathematical aggregation). We will present and motivate a third (combined) way of aggregation which combines the IDEA protocol for structured expert judgement with the mathematical aggregation scheme of the Classical Model (CM) (i.e. the weighted linear combination of judgements, where weights are calculated based on experts' prior performance on similar tasks).
At the end of this workshop participants will be familiar with both the IDEA and the CM protocols. They will benefit from a short hands on exercises, lecture style explanations, a list of relevant literature, and relevant contacts in the field. The workshop is aimed at professionals, academics, policymakers, regulators, and (MSc, PhD) students who are, or will soon be involved in decision problems or risk analysis modelling with scarce resources, and insufficient data.