Group Dropout Inspired by Ensemble Learning

• Published in: Neural Information Processing Vol. 9948; pp. 66 - 73
• Main Authors: Hara, Kazuyuki, Saitoh, Daisuke, Kondou, Takumi, Suzuki, Satoshi, Shouno, Hayaru
• Format: Book Chapter
• Language: English
• Published: Switzerland Springer International Publishing AG 2016; Springer International Publishing
• Series: Lecture Notes in Computer Science
• Subjects: Dropout learning; Ensemble learning; Soft-committee machine; Teacher-student formulation
• ISBN: 3319466712; 9783319466712
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-319-46672-9_8

Deep learning is a state-of-the-art learning method that is used in fields such as visual object recognition and speech recognition. This learning uses a large number of layers and a huge number of units and connections, so overfitting occurs. Dropout learning is a kind of regularizer that neglects some inputs and hidden units in the learning process with a probability p; then, the neglected inputs and hidden units are combined with the learned network to express the final output. We compared dropout learning and ensemble learning from three viewpoints and found that dropout learning can be regarded as ensemble learning that divides the student network into two groups of hidden units. From this insight, we explored novel dropout learning that divides the student network into more than two groups of hidden units to enhance the benefit of ensemble learning.