An eye–hand data fusion framework for pervasive sensing of surgical activities

• Published in: Pattern recognition Vol. 45; no. 8; pp. 2855 - 2867
• Main Authors: Thiemjarus, S., James, A., Yang, G.-Z.
• Format: Journal Article
• Language: English
• Published: Kidlington Elsevier Ltd 01.08.2012; Elsevier
• Subjects: Activity recognition; Applied sciences; Biological and medical sciences; Computational efficiency; Computerized, statistical medical data processing and models in biomedicine; Data fusion; Data integration; Endoscopes; Endoscopy; Exact sciences and technology; Eye–hand coordination; Feature selection; Information theory; Information, signal and communications theory; Investigative techniques, diagnostic techniques (general aspects); Medical management aid. Diagnosis aid; Medical sciences; Miscellaneous. Technology; Monitoring; Multi-objective BFFS; Multi-objective feature selection; Optimization; Pattern recognition; Recognition; Signal processing; Surgical workflow classification; Telecommunications and information theory; Activity recognition; Feature selection; Surgical workflow classification; Multi-objective BFFS; Eye–hand coordination; Multi-objective feature selection; Bayes estimation; Target tracking; Tree algorithm; Redundancy; Multiobjective programming; Gesture recognition; Instrumentation; Workflow; Activity levels; Eye-hand coordination; Search algorithm; Accuracy; Information processing; Signal processing; Feature extraction; Data fusion; Medical imagery; Endoscopy; Monitoring
• ISSN: 0031-3203; 1873-5142
• DOI: 10.1016/j.patcog.2012.01.008

This paper describes a generic framework for activity recognition based on temporal signals acquired from multiple input modalities and demonstrates its use for eye–hand data fusion. As a part of the data fusion framework, we present a multi-objective Bayesian Framework for Feature Selection with a pruned-tree search algorithm for finding the optimal feature set(s) in a computationally efficient manner. Experiments on endoscopic surgical episode recognition are used to investigate the potential of using eye-tracking for pervasive monitoring of surgical operation and to demonstrate how additional information induced by hand motion can further enhance the recognition accuracy. With the proposed multi-objective BFFS algorithm, suitable feature sets both in terms of feature relevancy and redundancy can be identified with a minimal number of instruments being tracked. ► We propose a generic eye–hand fusion framework for activity recognition. ► We propose a multi-objective BFFS with pruned-tree search algorithm for finding the optimal feature set(s). ► Endoscopic surgical episode recognition experiments are performed with a combined use of eye-tracking and motion sensing. ► Optimal feature sets, in terms of feature relevancy, redundancy and number of instruments being tracked, are identified. ► We validate the framework with surgical episode recognition experiments using various types of classifiers.