Analytical Performance Evaluation of Computer Vision Algorithms

Computer vision algorithms need deliver not only the answer but also the reliability of the answer under various conditions in order for the algorithms to be of any practical use. Most existing vision research, however, focuses on solution development with limited effort on systematically characterizing the reliability (precision) of the solution under different conditions. It is especially important to identify conditions where the algorithm's performance can be optimized and conditions where the algorithm may fail. This objective can be achieved by performance evaluation.

Performance evaluation of a computer vision system is a fundamental issue in computer vision and receives increasing attention from computer vision researchers. In fact, it has been realized that a major factor that hinders the further development of computer vision field both scientifically and practically is a lack of systematic and scientific methodology to both characterize and compare the performance of different computer vision systems.

Research in performance evaluation can be partitioned into two groups: empirical performance evaluation and analytical performance characterization. The former evaluates the accuracy of a vision algorithm using real image data by comparing the output of the algorithm with some ground-truth data. The advantage of this approach is its realism. The major difficulties with this approach include difficulty with obtaining ground-truth data, sufficiency and representation of the selected image data, and reliability and generalization of the conclusion. The second approach focuses on analytically studying the sensitivity of an algorithm's output with respect to parameters affecting input data including perturbation, image resolution, illuminations, data spatial distribution, data amount, etc. as well as to tuning parameters. The major advantages of this approach include: no need of ground-truth data, offer insight to the algorithm, conclusions are predictable, generalizable, and quantifiable. The disadvantages include the error propagation theory may not be tractable for complex problems, the various simplifying assumptions (such as Gaussian noise, no systematic errors, and small and additive noises) may not hold.

Our research focuses on analytical performance characterization of computer vision algorithms. Specifically, we are interested in developing a genetic statistical and mathematical framework that allows to analytically study various issues related to the performance of computer vision algorithms. Issues we are interested in studying include sensitivity analysis, error propagation, numerical stability, failure modes, bounds of the estimated parameters, and degnereate conditions. We intend to study these issues under two conditions: 1) independent of the specific algorithms used and 2) with respect to a specific an algorithm.