Development of a rescue system for agricultural machinery operators using machine vision

• Published in: Biosystems engineering Vol. 169; pp. 149 - 164
• Main Authors: Zhang, Yan, Gao, Pengbo, Ahamed, Tofael
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01.05.2018
• Subjects: Action recognition; Agricultural machinery; agricultural machinery and equipment; algorithms; Block Hankel-matrix; computer vision; data collection; Driver inattention; equipment operators; Rescue system; tractors; Action recognition; Agricultural machinery; Block Hankel-matrix; Driver inattention; Rescue system
• ISSN: 1537-5110; 1537-5129
• DOI: 10.1016/j.biosystemseng.2018.02.009

In this study, an automatic rescue system was proposed to monitor agricultural machinery operators using machine vision. The rescue system was developed to recognise the driver inattention status, that is, the distraction and fatigue by recognising the driver's actions. A Kinect sensor was used to collect image sequences of the operators, and the recognition system relied on the “player extraction” function of the Kinect sensor. A Hankel-based Kernel Mutual Subspace Method (KMSM) was developed to monitor tractor drivers and recognise driver inattention behaviours. To reduce the computational complexity for fulfilling the requirements of recognition, low-dimensional image vectors were used to generate low-dimensional block Hankel matrixes as representations for input action sequences. To evaluate the performance of the proposed KMSM, a driver action dataset was established that included 10 tractor drivers and 5 types of action that denote inattention. The drivers' inattention actions were classified into three danger levels, and the corresponding countermeasures for the actions at each danger level were similarly classified. Both offline and online experiments using similar subjects and different subjects were conducted to evaluate the designed inattention action recognition algorithm. In the offline experiment, the proposed Hankel-based KMSM achieved recognition rates of 91.18% and 86.18% when using similar and different subjects, respectively; and in the online experiment, the proposed method achieved 87.02 and 79.97% when using similar and different subjects, respectively. The average computation time of the Hankel-based KMSM was 0.07 s in the online experiment. Thus, the proposed Hankel-based KMSM method satisfies both the accuracy and the real-time requirements for a driver rescue system. •A rescue system was proposed to monitor the tractor's driver inattention action.•Kernel Mutual subspace algorithm proposed using Hankel matrix for the action images.•Developed algorithm had the advantage of processing the action images in real time.•Offline recognition rate 91% and 86% for similar/different subjects, respectively.•Online recognition rate 87% and 79% for similar/different subjects, respectively.