Perceptual Annotation: Measuring Human Vision to Improve Computer Vision

• Published in: IEEE transactions on pattern analysis and machine intelligence Vol. 36; no. 8; pp. 1679 - 1686
• Main Authors: Scheirer, Walter J., Anthony, Samuel E., Nakayama, Ken, Cox, David D.
• Format: Journal Article
• Language: English
• Published: Los Alamitos, CA IEEE 01.08.2014; IEEE Computer Society; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Accuracy; Annotations; Applied sciences; Artificial intelligence; Biological and medical sciences; Case studies; citizen science; Computer science; control theory; systems; Computer vision; Data Curation; Data processing. List processing. Character string processing; Databases, Factual; Exact sciences and technology; Face; Face - anatomy & histology; Face detection; Female; Fundamental and applied biological sciences. Psychology; Human; Humans; Information systems. Data bases; Intelligence; Learning; Machine Learning; Male; Memory organisation. Data processing; Pattern Recognition, Automated - methods; Pattern Recognition, Visual - physiology; Pattern recognition. Digital image processing. Computational geometry; Perception; psychology; Psychology. Psychoanalysis. Psychiatry; Psychology. Psychophysiology; psychometrics; Psychophysics; regularization; Software; Support Vector Machine; Support vector machines; Training; Training data; Vision; Visual; visual recognition; Visualization; Computer vision; Image recognition; support vector machines; Face recognition; face detection; Psychology; regularization; Citizenship; Psychometrics; citizen science; visual recognition; Psychophysics; Annotation; Visual perception; Human error; Educational software program; Data integrity; Information quality; Facies; Machine learning; Vector support machine; Artificial intelligence
• ISSN: 0162-8828; 1939-3539; 2160-9292; 1939-3539
• DOI: 10.1109/TPAMI.2013.2297711

For many problems in computer vision, human learners are considerably better than machines. Humans possess highly accurate internal recognition and learning mechanisms that are not yet understood, and they frequently have access to more extensive training data through a lifetime of unbiased experience with the visual world. We propose to use visual psychophysics to directly leverage the abilities of human subjects to build better machine learning systems. First, we use an advanced online psychometric testing platform to make new kinds of annotation data available for learning. Second, we develop a technique for harnessing these new kinds of information-"perceptual annotations"-for support vector machines. A key intuition for this approach is that while it may remain infeasible to dramatically increase the amount of data and high-quality labels available for the training of a given system, measuring the exemplar-by-exemplar difficulty and pattern of errors of human annotators can provide important information for regularizing the solution of the system at hand. A case study for the problem face detection demonstrates that this approach yields state-of-the-art results on the challenging FDDB data set.