Uncertainty quantification seeks to determine how trustable the predictions are. Predictive uncertainty can be divided into epistemic uncertainty and aleatoric uncertainty. Epistemic uncertainty measures what the model does not know about the input due to a lack of knowledge. Epistemic uncertainty is hence inversely proportional to the training data density. High epistemic uncertainty arises in regions, where there are few or no observations. Aleatoric uncertainty measures the natural stochasticity in the data. High aleatoric uncertainty is expected for inputs with high perturbations.
Bayesian deep learning (BDL) offers a principled approach to compute both aleatoric and epistemic uncertainties. Instead of using a point-estimated model to perform the prediction, it constructs the posterior distribution of the model parameters. BDL performs prediction by integrating predictions from different models sampled from the parameter posterior, hence avoiding overfitting and allowing systematically quantifying the predictive uncertainty.
Our research develops advanced BDL methods for accurately and efficiently quantifying the predictive uncertainties. We have demonstrated the effectiveness of our proposed methods on various machine learning applications such as out-of-distribution detection and image classification under distributional shifts.
In this research, we propose the diversity-enhanced probabilistic ensemble method, a Bayesian framework to estimate both epistemic and aleatoric uncertainty. Specifically, we construct the probabilistic ensemble model by building a Gaussian distribution of the model parameters for each ensemble component using Laplacian approximation in a post-processing manner. Then a mixture of Gaussian model is established with learnable and refinable parameters in an EM-like algorithm. During ensemble training, we leverage the uncertainty estimated from previous models as guidance when training the next one such that the new model will focus more on the regions where previous models do not have sufficient knowledge.
We both theoretically and empirically show improved diversity by the exploration of each ensemble subspace using LA and the adaptive ensemble training strategy. The enhanced diversity enables better modeling of the parameter posterior, hence leading to better uncertainty quantification performance.
This research introduces the Hierarchical Probabilistic Neural Network (HPNN), a single-network sampling-free method for both epistermic and aleatoric uncertainty quantification. As an evidential deep learning method, HPNN can efficiently estimate the uncertainties within a single deterministic forward pass of the neural network. Specifically, evidential deep learning treats the distribution parameters for the output variables as random variables that follow a learnable conjugate prior distribution. To learn accurate conjugate distribution, we introduce an additional regularization term to distill the knowledge from the Bayesian neural networks (BNNs) into HPNN. Different methods (Laplace approximation and ensemble ) are used to apprximate the BNN. We further propose to replace the Dirichlet prior distribution assmuption with the normalizing flow to obtain a more accurate estimation of the prior distribution. To relax assumptions, we propose a self-regularized training strategy for HPNN using Laplacian Approximation (LA). The self-distillation strategy avoids the heavy burden on additional BNNs, ensemble models, density models as well as out-of-distribution samples.
Epistemic uncertainty quantification (UQ) identifies where models lack knowledge. Traditional approaches to UQ, typically reliant on Bayesian frameworks, fall short when applied to pre-trained models outside the Bayesian paradigm. Our research tackles the challenge of quantifying epistemic uncertainty in any pre-trained model without necessitating access to the original dataset, alterations to the model, or constraints on the model's architecture and training methodology. We introduce a novel gradient-based method for epistemic UQ that evaluates model output gradients in relation to parameters, pinpointing where adjustments are needed for a more accurate input representation. Contrary to the belief that epistemic uncertainty can only be measured by comparing various models, our approach provides theoretical backing for the efficacy of gradient-based UQ. We enhance this method by applying class-specific gradient weights and layer-specific emphasis, improving the discernment of contributions across different network layers. Our method also incorporates a hybrid of gradient and perturbation techniques to refine the gradients. Tested on tasks like out-of-distribution detection, uncertainty calibration, and active learning, our approach outperforms existing UQ techniques for pre-trained models, showcasing its broad applicability and effectiveness.
Hanjing Wang and Qiang Ji. Diversity-enhanced Probabilistic Ensemble For Uncertainty Estimation. 39th Conference on Uncertainty in Artificial Intelligence (UAI), 2023. [PDF]
Hanjing Wang, and Qiang Ji. Epistemic Uncertainty Quantification For Pre-trained Neural Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.