Grand Challenge: Passive BCI Hackathon
|Ranking||Team||Organization & Country||Abstract Title|
|1||Swapnil Singh, Vartika Gupta, Tharun Kumar Reddy, Vipul Arora||IIT Kanpur & IIT Roorkee, India||Inter-session Subject Specific Mental Workload Multiclass Classification using Riemannian Methods|
|2||Sara Sedla, Johann Benerradi, Côme Le Breton||Inria, France & University of Nottingham, UK||Rank-1 CNN for mental workload classification from EEG|
|3||Marie-Constance Corsi, Sylvain Chevallier, Quentin Barthélemy, Isabelle Hoxha, Florian Yger||Inria, Paris, Univ. Paris-Saclay & Univ. Paris-Dauphine, France||Ensemble learning based on functional connectivity and Riemannian geometry for robust workload estimation|
|4||Divyesh Narayanan||Birla Institute of Technology and Science, Pilani, India||Inter-session Transfer Learning using Riemannian Geometry for Mental Workload Estimation
|5||Frank Bolton, Dov Te’eni, Eran Toch||Tel Aviv University, Tel Aviv, Israel||EEG Mental Workload Classification with Random Forest
|6||Srihari Madhavan & Vishnu K .N. Cota Navin Gupt||Indian Institute of Technology Guwahati, Assam, India||RNN Classification of Mental Workload EEG
|7||Mansi Sharma||DFKI Saarbrücken, Germany||Mental workload classification|
|8||Aneta Kartali, Kosta Jovanović, Luka Bojović||mBrainTrain LLC & Univ. of Belgrade, Belgrade, Serbia||Passive BCI Hackathon: Applying Deep Learning to Estimate Mental Workload
|9||Kunjira Kingphai and Yashar Moshfeghi||University of Strathclyde, Glasgow, Scotland||Mental Workload Prediction Level from EEG Signals using Deep learning Models
|10||Mario De Lorenzo, Mali Halac, Sukethram Sivakumar, Hasan Ayaz||Drexel University, Philadelphia, USA||Deep Neural Network modeling using CNN and BiLSTM for online Passive EEG Brain-Computer Interface to classify Mental Workload|
Join the challenge! The best submission will win a cash prize of 500€.
Your goal is to predict the mental workload for a given subject (intra-subject estimation) in one session, based on the EEG data from another session (inter-session adaptation).
Details for Participation
Your extended abstract (max. 1000 words; 2 figures/tables) will be published in our proceedings in Frontiers for Neuroergonomics.
Please read below for more information regarding participation rules, the timetable of the competition, a description of the dataset, the MATB task, its acquisition and preprocessing.
Do not hesitate to join our Discord channel HERE for questions
Participation & Submission Rules
- One submission per team
- One participant may only be part of one team
- Submission must include: results and abstract (same format as regular paper submissions for the conference)
15-Jun-2021: Official competition opening, publication of the database (2 sessions with labels)
01-Jul-2021: Release of dataset version 2 with the 3rd session included as a test set (i.e. without labels).
31-Jul-2021: Competition closing, deadline for code, predictions and abstract submission
31-Aug-2021: Evaluation by the hackathon organizing team
05-Sep-2021: Submission of presentation materials (Link)
13-Sep-2021: Announcement of results at the opening reception. Oral presentation of the winner and poster session presentations of all competing submissions.
Context & challenge:
Passive Brain-Computer Interfaces (BCIs) can estimate users’ states, e.g., their cognitive or affective states, from their brain signals, and use these estimations to adapt a human-computer interaction system accordingly [Zander2011]. As such, passive BCIs have been used for many applications, including to estimate users’ mental workload, in order to adapt education material to students’ cognitive resources [Yuksel2016] or to prevent airplane pilots from being overloaded and thus from missing alarms [Dehais2019]; or to estimate users’ affective states, in order to design adaptive video games maximizing users’ excitement or pleasure [Ewing2016], among many other.
As such, passive BCIs are a key element for Neuroergonomics [Lotte2019], for the design of numerous real-life studies and applications of neurotechnologies [Ayaz2018]. However, beyond promising proof-of-concepts, really using passive BCIs in everyday life still requires to face numerous challenges. One of them is the well-documented large within-subject variability affecting brain signals such as ElectroEncephaloGraphy (EEG) signals.
Indeed, EEG signals are highly non-stationary, and can change a lot across days, or even within day, even for the same user [Fairclough2020]. However, so far, the vast majority of passive BCI studies were conducted on a single day (a.k.a. session), making it unclear whether the designed BCI would still work on different days/sessions without re-calibration.
Similarly, existing public passive BCI data sets provide brain signals recorded on a single day/session [Hinss2021], preventing the design and evaluation of passive BCIs that would work across days/sessions. Ideally, a real-life passive BCI, as envisioned in Neuroergonomics, would need to be effective and efficient at all time, i.e., across sessions, without recalibration.
This competition aims at addressing this scientific challenge, by releasing the first (to the best of our knowledge) public passive BCI data set providing EEG signals across multiple sessions for each user, and by challenging the competitors to come up with the best algorithm to decode mental workload from EEG signals on a new unseen session, from a training data set comprising several sessions.
For this passive BCI hackathon, 15 participants (6 female; 9 average 25 y.o.) were invited to the lab for three independent experimental sessions each spaced one week apart (exactly 7 days). Each session involved a short training/warm up period. Following this, a resting state (one minute with eyes open) was recorded. Participants then completed an MATB-II task with three 5-minute blocks, each of a different difficulty level (i.e. different workload level) presented in a pseudorandom manner. By varying the number and complexity of the sub-tasks, 3 levels of workload were elicited (verified through statistical analyzes of both subjective and objective -behavioral and cardiac- data).
The project was validated by the local ethical committee of the University of Toulouse (CER number 2021-342). The dataset is part of a new open EEG database currently under development, and designed to answer a need for more publicly available EEG-based dataset to design and benchmark passive brain-computer interface pipelines (as detailed in [Hinss2021]).
The data are exported as a dataset from EEGlab under the .set and .fdt format. The OBS formatting guidelines were followed. Data are organized as following in the directory:
– One directory per subject
– Two sub-directories for each session
– Inside each session directory one sub-directory for the precise electrode locations (measured via the app) and one for the EEG data
– 5 (.set) files per session. Each of the epoched and preprocessed task conditions, the resting state as well as the raw file for the resting state.
– Each epoch is marked by events to show the condition (difficult, medium, easy, Resting State)
– The electrode locations are provided in a .txt file with the xyz coordinates for each electrode
This competition focuses on a renowned task that elicits various levels of mental/cognitive workload: the Multi-Atribute Task Battery-II (MATB-II) developed by NASA (https://matb.larc.nasa.gov/) assess task-switching and mental workload capacities (https://matb.larc.nasa.gov). Participants have to perform several subtasks simultaneously. Depending on the condition the number of subtasks and their respective difficulty differed.
In the easy condition (label 0) participants engaged in the TRACK and SYSTEM MONITORING task. TRACK is a simulation of manual control, and the participant has to keep a target at the center of a window. The SYSTEM MONITORING TASK demands monitoring of 4 gauges and 2 warning lights.
For the medium condition (label 1) a third task was added. RESOURCE MANAGEMENT presents the participant with a fuel management system where the goal is to maintain a certain fuel level by activating and deactivating a set of pumps that allow for the allocation of fuel to several reservoirs.
Finally to the difficult condition (label 2) COMMUNICATION was added to the three previous tasks: Here the participant has to respond to radio messages by changing the frequencies of different radios. Additionally, the TRACK task was made more demanding in the DIFFICULT condition.
Each of the conditions lasted for 5 minutes. The order was randomized with the other tasks meaning that participants did not necessarily start with the easy task first. Also participants may have performed other tasks in between the conditions.
The data acquisition was performed using a 64 active Ag-AgCl Electrode system (ActiCap, Brain Products Gmbh) and an ActiCHamp amplifier (Brain Products, Gmbh). One electrode could not be used and one electrode was dedicated to record cardiac activity, resulting in 62 electrodes, placed according to the international 10-20 system. In addition the precisie electrode location was obtained using a STRUCTURE (https://structure.io) 3D camera and the chanlocs plug-in developed specifically for electrode localisation purposes (https://github.com/sccn/get_chanlocs/wiki). The sampling frequency was set to 500Hz. Impedances were kept below 10 kΩ as much as possible. Markers of all events occurring during the tasks were recorded and synchronized using the LabStreamingLayer.
The preprocessing of the data was done in Matlab through the use of functions from the EEGlab toolbox. First the data from the resting state as well as the tasks was extracted from the overall recording. The electrode recording cardiac activity was removed and the remaining data was cut into two second non-overlapping epochs. Please see below for the preprocessing that was applied in order to clean the data:
– Epoching into 2-second non-overlapping epochs
– Referencing using right mastoid electrode
– High-pass filter 1 Hz (FIR Filter, pop-filtnew from EEGlab)
– Electrode rejection (average amplitude above 2 sd across channels) + spherical interpolation
– SOBI with subsequent automated IC_Label rejection (muscle, heart and eye components were rejected with a 95% threshold)
– Low-pass filter 40 Hz (FIR Filter)
– Average re-referencing (CAR)
– Down-sampling to 250 Hz
For more information, please contact Dr. Raphaëlle Roy by email.
Grand Challenge Team:
Dr Raphaëlle N. Roy studied cognitive science in France (Univ. Lyon and Grenoble-Alps) and Canada (UBC, Vancouver) and obtained her PhD in cognitive neuroscience and signal processing in 2015 from Grenoble INP (French engineering school and research institute). Since 2016, she is Assistant Professor in neuroergonomics and physiological computing at ISAE-SUPAERO, in the Department of Conception and Control of Aeronautical and Spatial Vehicles. She is also co-chair in the Artificial and Natural Intelligence Toulouse Institute (ANITI), co-founder and part of the steering committee of the French brain-computer interface association CORTICO, and associate editor for Frontiers in Neuroergonomics. She leads an interdisciplinary research at the cross-roads of cognitive science, neuroscience, machine learning and human factors, with a focus on how to better characterize operators’ mental state to enhance human-machine interaction and improve both safety and performance.
Mr Marcel Hinss: is an intern in Neuroergonomics at the DCAS department at the Institut Superieur de l’Aeronautique et de l’Espace (ISAE) in Toulouse. He is currently finishing his Research Master’s Degree at Maastricht University in cognitive and clinical neuroscience with a specialization in Neuroeconomics. His main objective for the duration of his internship at ISAE is to create an open EEG based database for passive Brain Computer Interfaces. He was responsible for the collection of the data presented in this competition.
Prof. Fabien Lotte: Dr. Fabien Lotte obtained a M.Sc., a M.Eng. (2005), and a PhD (2008) from INSA Rennes, and a Habilitation (HDR, 2016) from Univ. Bordeaux, all in computer science. His research focuses on the design, study and application of Brain-Computer Interfaces (BCI). In 2009 and 2010, Fabien Lotte was a research fellow at the Institute for Infocomm Research in Singapore. From 2011 to 2019, he was a Research Scientist at Inria Bordeaux Sud-Ouest, France. Between October 2016 and January 2018, he was a visiting scientist at the RIKEN Brain Science Institute, Japan. Since October 2019, he is a Research Director (DR2) at Inria Bordeaux Sud-Ouest. He is on the editorial boards of the journals Brain-Computer Interfaces (since 2016) and Journal of Neural Engineering (since 2016). He is also “co-specialty chief editor” of the section “Neurotechnologies and System Neuroergonomics” of the journal “Frontiers in Neuroergonomics”. He co-edited the books ”Brain-Computer Interfaces 1: foundations and methods” and ”Brain-Computer Interfaces 2: technology and applications” (2016) and the ”Brain-Computer Interfaces Handbook: Technological and Theoretical Advance” (2018). In 2016, he was the recipient of an ERC Starting Grant to develop his research on BCI.
Dr Ludovic Darmet: obtained a master from Ecole Centrale de Lille (France) in 2017 with a specialization in Decision and Data Analysis and a PhD from GIPSA-lab (France) in 2020, directed by François Cayre and Kai Wang. His PhD research interests included digital image forensics and computer vision. He joined ISAE-Supaero in 2021 as a post-doctoral fellow in Pr Frédéric Deshais neuroergonomics team. His research work now focuses on active BCI, mainly through SSVEP paradigms.
Dr Simon Ladouce: is a postdoctoral researcher in Neuroergonomics and Human Factors at the Institut Superieur de l’Aeronautique et de l’Espace (ISAE) in Toulouse. He obtained a PhD in cognitive neurosciences from the University of Stirling in 2018 for his work on the application of brain and body imaging methods to the study of cognitive processes underlying natural behaviours. His current work focuses on the development of SSVEP-based BCI that are visually unobtrusive to improve user’s experience.
[Ayaz2018] Ayaz, H., & Dehais, F. (Eds.). (2018). Neuroergonomics: The brain at work and in everyday life. Academic Press.
[Dehais2019] Dehais, F., Roy, R. N., & Scannella, S. (2019). Inattentional deafness to auditory alarms: Inter-individual differences, electrophysiological signature and single trial classification. Behavioural brain research, 360, 51-59.
[Ewing2016] Ewing, K. C., Fairclough, S. H., & Gilleade, K. (2016). Evaluation of an adaptive game that uses EEG measures validated during the design process as inputs to a biocybernetic loop. Frontiers in human neuroscience, 10, 223.
[Fairclough2020] Fairclough, S. H., & Lotte, F. (2020). Grand Challenges in Neurotechnology and System Neuroergonomics. Frontiers in Neuroergonomics, 1, 2.
[Hinss2021] Hinss, M., Somon, B., Dehais, F. & Roy, R. N. (2021) Open EEG Datasets for Passive Brain-Computer Interface Applications: Lacks and Perspectives. IEEE Neural Engineering Conference.
[Yuksel 2016] Yuksel, B. F., Oleson, K. B., Harrison, L., Peck, E. M., Afergan, D., Chang, R., & Jacob, R. J. (2016, May). Learn piano with BACh: An adaptive learning interface that adjusts task difficulty based on brain state. In Proceedings of the 2016 CHI conference on human factors in computing systems (pp. 5372-5384).
[Lotte2019] Lotte, F.* & Roy, R. N.* (2019) Brain-Computer Interfaces’ Contributions to Neuroergonomics. In Ayaz, H. & Dehais, F. (Eds), Neuroergonomics: The Brain at Work and in Everyday Life. Elsevier.
[Zander 2011] Zander, T. O., & Kothe, C. (2011). Towards passive brain–computer interfaces: applying brain–computer interface technology to human–machine systems in general. Journal of neural engineering, 8(2), 025005.