Decoding Internal Decision Making During Reverse Engineering Tasks

• Published in: bioRxiv
• Main Authors: Marsh, Brianna, Rego, Jocelyn, Levy, Mia Matsuhisa, Sayer, Mitchell Peter, Waagen, Alexander, Barbieux, Aidan, Cranford, Edward A, Morrison, Donald F, Maldonado, Froylan, Johnson, Jeremy Phillip, DiVita, Joseph, Buch, Jonathan Michael, Kim, Tiffany Hyun-Jin, Lebiere, Christian, Fugate, Sunny, Bhattacharyya, Rajan
• Format: Paper
• Language: English
• Published: Cold Spring Harbor Laboratory 10.10.2023
• Edition: 1.1
• Subjects: Neuroscience; Neuroscience; Neural Decoding; Human Machine Teaming
• ISSN: 2692-8205
• DOI: 10.1101/2023.10.10.561734

Neural decoding is often limited to tasks with known stimuli and limited response options . Real world tasks, however, are often completely stimulus free with unconstrained user response possibilities. Real time decoding of internal decision making would allow for more complex and interactive Huma Machine Teaming in a way that is not currently possible. To address this problem, we present here a novel method of decoding moments of recognition and their associated internal value judgments in the context of highly complex software reverse engineering tasks. This is done through a combination of P300 detection (a neural marker of recognition) and the Engagement Index (a ratio of neural band powers) to determine whether an item has been identified as relevant to the task (to be further explored) or irrelevant to the task (to be quickly ignored). Artificial neural networks were trained to identify P300s in each subject during the reverse engineering tasks. Dimensionality reduction of neural data during the tasks showed the existence of separately clustering subgroups of P300s with differences in Engagement Index. Subgroups of P300s differentiated by Engagement were further verified as distinct groupings with pupil dilation and user behavior metrics. This decoded information could be used to aid in the reverse engineering process via cognitive offloading of the user’s own decision making on to the visual interface in a completely automated and personalized fashion. This represents a significant advance in domain of real-time neural decoding, and opens up many further possibilities for usage in a broad range of intelligent human systems integration applications.