Dreem Open Datasets: Multi-Scored Sleep Datasets to Compare Human and Automated Sleep Staging

• Published in: IEEE transactions on neural systems and rehabilitation engineering Vol. 28; no. 9; pp. 1955 - 1965
• Main Authors: Guillot, Antoine, Sauvet, Fabien, During, Emmanuel H., Thorey, Valentin
• Format: Journal Article
• Language: English
• Published: United States IEEE 01.09.2020; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Annotations; Apnea; Automated sleep stage classification; Automation; Brain modeling; Datasets; Deep learning; EEG; Electroencephalography; Feature extraction; Human performance; Inspection; inter-rater agreement; Machine learning; Manuals; open datasets; PSG; Sleep; Sleep apnea; Sleep disorders; State-of-the-art reviews; Technologists
• ISSN: 1534-4320; 1558-0210; 1558-0210
• DOI: 10.1109/TNSRE.2020.3011181

Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a single human scorer annotation despite an inter-rater agreement of about 85% only. The present study introduces two publicly-available datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients suffering from obstructive sleep apnea (OSA). Both datasets have been scored by 5 sleep technologists from different sleep centers. We developed a framework to compare automated approaches to a consensus of multiple human scorers. Using this framework, we benchmarked and compared the main literature approaches to a new deep learning method, SimpleSleepNet, which reach state-of-the-art performances while being more lightweight. We demonstrated that many methods can reach human-level performance on both datasets. SimpleSleepNet achieved an F1 of 89.9% vs 86.8% on average for human scorers on DOD-H, and an F1 of 88.3% vs 84.8% on DOD-O. Our study highlights that state-of-the-art automated sleep staging outperforms human scorers performance for healthy volunteers and patients suffering from OSA. Considerations could be made to use automated approaches in the clinical setting.