Sign language recognition using a combination of new vision based features

• Published in: Pattern recognition letters Vol. 32; no. 4; pp. 572 - 577
• Main Authors: Zaki, Mahmoud M., Shaheen, Samir I.
• Format: Journal Article
• Language: English
• Published: Amsterdam Elsevier B.V 01.03.2011; Elsevier
• Subjects: Appearance based features; Applied sciences; Exact sciences and technology; Hand gesture; Hidden Markov Model (HMM); Information, signal and communications theory; Kurtosis; PCA; Sign language recognition; Signal and communications theory; Signal processing; Signal representation. Spectral analysis; Signal, noise; Speech processing; Telecommunications and information theory; Sign language recognition; Appearance based features; Kurtosis; Hidden Markov Model (HMM); Hand gesture; PCA; Performance evaluation; Automatic classification; Prototype; Error rate; Locator; Aspect oriented; Gesture; Database; Language recognition; Probabilistic approach; Signal classification; Statistical method; Sign language; Dimension reduction; Flattening; Hidden Markov models; Speech recognition; Speech processing; Principal component analysis
• ISSN: 0167-8655; 1872-7344
• DOI: 10.1016/j.patrec.2010.11.013

► We present a combination of vision based features to enhance the recognition of ASL. ► Two features are newly introduced: kurtosis position and PCA as an image descriptor. ► Kurtosis position gave smallest error rate when the number of words is small. ► PCA is the most reliable feature when the number of words increases. ► Using a voting based combiner lead to additional reduction in the error rate. Sign languages are based on four components hand shape, place of articulation, hand orientation, and movement. This paper presents a novel combination of vision based features in order to enhance the recognition of underlying signs. Three features are selected to be mapped to these four components. Two of these features are newly introduced for American sign language recognition: kurtosis position and principal component analysis, PCA. Although PCA has been used before in sign a language as a dimensionality reduction technique, it is used here as a descriptor that represents a global image feature to provide a measure for hand configuration and hand orientation. Kurtosis position is used as a local feature for measuring edges and reflecting the place of articulation recognition. The third feature is motion chain code that represents the hand movement. On the basis of these features a prototype is designed, constructed and its performance is evaluated. It consists of skin color detector, connected component locator and dominant hand tracker, feature extractor and a Hidden Markov Model classifier. The input to the system is a sign from RWTH-BOSTON-50 database and the output is the corresponding word with a recognition error rate of 10.90%.