Using deep learning to predict outcomes of legal appeals better than human experts: A study with data from Brazilian federal courts

• Published in: PloS one Vol. 17; no. 7; p. e0272287
• Main Authors: Jacob de Menezes-Neto, Elias, Clementino, Marco Bruno Miranda
• Format: Journal Article
• Language: English
• Published: San Francisco Public Library of Science 28.07.2022; Public Library of Science (PLoS)
• Subjects: Appeals; Appellate procedure; Attorneys; Biology and Life Sciences; Computer and Information Sciences; Conflict resolution; Correlation coefficient; Correlation coefficients; Court decisions; Decision theory; Deep learning; Electronic filing; Judges & magistrates; Judiciary; Jurisdiction; Law; Learning algorithms; Legal research; Machine learning; Natural language processing; People and places; Performance prediction; Prediction (Psychology); Small claims courts; Social behavior; Social Sciences; State courts; Trials; Brazil
• ISSN: 1932-6203; 1932-6203
• DOI: 10.1371/journal.pone.0272287

Legal scholars have been trying to predict the outcomes of trials for a long time. In recent years, researchers have been harnessing advancements in machine learning to predict the behavior of natural and social processes. At the same time, the Brazilian judiciary faces a challenging number of new cases every year, which generates the need to improve the throughput of the justice system. Based on those premises, we trained three deep learning architectures, ULMFiT, BERT, and Big Bird, on 612,961 Federal Small Claims Courts appeals within the Brazilian 5th Regional Federal Court to predict their outcomes. We compare the predictive performance of the models to the predictions of 22 highly skilled experts. All models outperform human experts, with the best one achieving a Matthews Correlation Coefficient of 0.3688 compared to 0.1253 from the human experts. Our results demonstrate that natural language processing and machine learning techniques provide a promising approach for predicting legal outcomes. We also release the Brazilian Courts Appeal Dataset for the 5th Regional Federal Court (BrCAD-5), containing data from 765,602 appeals to promote further developments in this area.