In a bid to propel advancements in neuroscience, we are unveiling the complete training code for Brain2Qwerty versions 1 and 2. Additionally, our collaborator, the Basque Center on Cognition, Brain, and Language (BCBL), will be making available the v1 dataset. This initiative aims to significantly impact the lives of the millions grappling with brain lesions that hinder their communication abilities. While invasive techniques like stereotactic electroencephalography and electrocorticography have demonstrated that a neuroprosthesis can transmit signals to an AI decoder for speech restoration, these methods are challenging to implement on a wider scale. Our approach, which is non-invasive, seeks to address this limitation.
Last year saw the launch of Brain2Qwerty v1, a pioneering research project that utilizes AI to translate brain activity into text without the need for surgical intervention. We are now excited to present Brain2Qwerty v2, an advanced end-to-end pipeline capable of real-time sentence decoding from non-invasive brain recordings, achieving accuracy levels that were previously reserved for more invasive techniques.
Brain2Qwerty v2 was trained on roughly 22,000 sentences derived from nine participants, each contributing 10 hours of recordings through a magnetoencephalography (MEG) device while engaging in typing tasks. Unlike traditional methods that depend on manually crafted pipelines to identify neural activities, our approach employs an end-to-end deep learning model, decoding directly from unrefined brain signals.
By fine-tuning large language models with neural data, the system effectively harnesses semantic context, thereby connecting sporadic brain recordings with structured language. We also utilized AI agents to further enhance the decoding pipeline, with final configurations optimized by our engineering team.
The outcome is impressive: Brain2Qwerty v2 successfully extracts coherent sentences from noisy neural inputs, registering a word accuracy of 61%—a substantial leap from the mere 8% word accuracy achieved by other non-invasive techniques. Notably, for our best-performing participant, we recorded a remarkable 78% word accuracy, with over half of the sentences reconstructed containing one word error or fewer.
