Addressing Cross-Subject and Trial-to-Trial Variability in EEG Physiology-Informed Data Augmentation PDF Free Download
1 / 123/123
100%
Addressing Cross-Subject and Trial-to-Trial Variability in EEG
Physiology-Informed Data Augmentation
vorgelegt von
M. Sc.
Oleksandr Zlatov
ORCID: 0009-0008-1784-2251
an der Fakultät IV - Elektrotechnik und Informatik
der Technischen Universität Berlin
zur Erlangung des akademischen Grades
Doktor der Naturwissenschaften
- Dr. rer. nat. -
genehmigte Dissertation
Promotionsausschuss:
Vorsitzender: Prof. Dr. Stefan Haufe
Gutachter: Prof. Dr. Benjamin Blankertz
Gutachter: Prof. Dr. Michael Tangermann
Gutachter: Fabien Lotte, PhD
Tag der wissenschaftlichen Aussprache: 23. November 2023
Berlin 2024
Abstract
Brain-computer interfaces (BCIs) based on electroencephalography (EEG) are designed to extract valuable
information about the user’s mental state, enabling various applications, including communication and de-
vice control. However, the utilization of this technology is hindered by challenges in EEG processing. BCI
systems typically require extensive training data due to the considerable variability in EEG signals among
individuals and even across sessions and trials. This challenge has become even more prominent with the
adoption of neural network classifiers, which demand even larger amounts of training data. To overcome
these limitations, data augmentation methods have been proposed to enhance the quantity and diversity of
the available data and improve the generalization of classification models. Existing data augmentation ap-
proaches either offer limited improvements through simple data transformations or employ complex deep
learning techniques that are challenging to train and unsuitable for small datasets.
In this dissertation, we introduce two novel data augmentation methods that offer substantial enhance-
ments in classification accuracy while maintaining simplicity and ease of implementation. Our approach
stands apart from previous methods by incorporating knowledge about the physiology and geometry of
the brain. Specifically, we utilize source localization techniques to identify task-relevant components and
incorporate backward and forward projection using a head model for data modification. We evaluate the
effectiveness of our methods in cross-subject and within-subject classification of motor imagery data using
various classifiers and scenarios with different amounts of training data. Our results demonstrate significant
improvements in classification performance across a wide range of scenarios, especially when applying our
data augmentation techniques for deep learning models. Moreover, we showcase that our methods generate
realistic and diverse EEG trials in a manner consistent with biological plausibility. These findings highlight
the high potential of physiology-informed data augmentation in EEG-based BCIs and present opportunities
for robust calibration-free utilization of this technology.
Zusammenfassung
Brain-Computer Interfaces (BCIs) auf Basis von Elektroenzephalographie (EEG) extrahieren wertvolle Infor-
mationen über den mentalen Zustand des Benutzers und ermöglichen verschiedene Anwendungen, darunter
Kommunikation und Gerätesteuerung. Die Nutzung dieser Technologie wird jedoch durch Herausforderun-
gen bei der EEG-Verarbeitung erschwert. BCI-Systeme erfordern üblicherweise umfangreiche Trainings-
daten aufgrund der Variabilität in den EEG-Signalen zwischen Individuen und Sitzungen. Diese Heraus-
forderung wird durch den Einsatz von neuronalen Netzwerk-Klassifikatoren, die noch größere Mengen an
Trainingsdaten erfordern, noch verstärkt. Um diese Begrenzungen zu überwinden, wurden Datenaugmen-
tierungsmethoden vorgeschlagen, um die Datenmenge und -vielfalt zu erhöhen und die Generalisierungs-
fähigkeit von Klassifikationsmodellen zu verbessern. Bisherige Ansätze zur Daten-Augmentation bieten
entweder nur begrenzte Verbesserungen durch einfache Daten-Transformationen oder setzen komplexe
Deep-Learning-Techniken ein, die schwierig zu trainieren sind und für kleine Datensätze ungeeignet sind.
In dieser Dissertation stellen wir zwei neuartige Daten-Augmentationsmethoden vor, die erhebliche
Verbesserungen der Klassifikationsgenauigkeit bieten und dabei Einfachheit und Umsetzbarkeit bewahren.
Unser Ansatz unterscheidet sich von bisherigen Methoden, indem er Wissen über die Physiologie und Ge-
ometrie des Gehirns einbezieht. Konkret nutzen wir Techniken zur Quellenlokalisation, um aufgabenspez-
ifische Komponenten zu identifizieren, und verwenden ein Kopfmodell für die Rückwärts- und Vorwärt-
sprojektion zur Datenmodifikation. Wir evaluieren die Wirksamkeit unserer Methoden bei der Klassifika-
tion von Motor-Imagination-Daten sowohl zwischen als auch innerhalb von Probanden unter Verwendung
verschiedener Klassifikatoren und Szenarien mit unterschiedlichen Mengen an Trainingsdaten. Unsere
Ergebnisse zeigen signifikante Verbesserungen der Klassifikationsleistung in einem breiten Spektrum von
Szenarien, insbesondere bei der Anwendung unserer Daten-Augmentationsmethoden für Deep-Learning-
Modelle. Darüber hinaus zeigen wir, dass unsere Methoden realistische und vielfältige EEG-Versuche erzeu-
gen, die biologisch plausibel sind. Diese Ergebnisse unterstreichen das enorme Potenzial von physiolo-
giebasierten Daten-Augmentationsmethoden in EEG-basierten BCIs und eröffnen Möglichkeiten für eine
stabile, kalibrierungsfreie Nutzung dieser Technologie.
i
Acknowledgements
I would like to acknowledge the invaluable support and contributions of numerous people
during the course of my thesis.
First and foremost, I would like to express my deepest appreciation to my supervisor, Prof.
Dr. Benjamin Blankertz. His guidance and the discussions we had about scientific concepts
greatly enhanced the quality of my work. Thanks to his support, I never felt alone on this long
journey.
I was fortunate to be a part of the Neurotechnology Group at TU Berlin, and be surrounded
by researchers from the Machine Learning Group and DAEDALUS RTG. I wish to express my
appreciation to Daniel, Vera, Nils, Gabriel, Noah, Park, Dominik, and Imke, among others, for
their significant contributions and support throughout my research.
My sincere thanks go out to my parents, who have supported me unconditionally through-
out my life. During these years, I always felt their presence and support, even when they were
physically distant.
I extend my deepest gratitude to my partner, Jean-Marie, who has been by my side through-
out these exciting yet challenging years, offering constant encouragement and support. I am
incredibly happy to have you in my life.
Many other friends and loved ones have played vital roles in supporting me on this journey.
I want to offer a special thank you to Kateryna Chernii, with whom I spent countless hours
in the library, and to Michael Rudi Koliatsis, who has always been there for me. I am also
thankful to all my friends from Odesa, Dresden, Tübingen, and Grenoble for their friendship
and encouragement.
Your collective support means a lot to me, and I truly appreciate having all of you in my life
during this special time.
ii
Contents
Abstract i
Zusammenfassung i
Acknowledgements ii
1 Introduction 1
2 Recording of brain activity 2
2.1 Nature of brain signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1.1 Neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.1.2 Brain structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2 Measurement of brain signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.1 Invasive methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.2.2 Non-invasive methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 BCI paradigms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.1 Motor imagery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 Event-related potentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.3 Steady-state evoked potential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.4 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3 Processing of EEG data 17
3.1 Preprocessing and artifact removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.1 Artifacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.1.2 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.1.3 Other preprocessing techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2 Feature extraction and selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2.1 Temporal methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2.2 Spectral methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2.3 Time-frequency methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.2.4 Spatial methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.2.5 Spatio-temporal methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2.6 Feature selection and dimensionality reduction . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3 EEG source localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
3.3.1 Linear model of EEG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.2 Head modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.3.3 Source localization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
3.4 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4 EEG Classification for BCIs 29
4.1 Traditional machine learning approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.1.1 Linear discriminant analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.1.2 Support vector machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
4.1.3 Other traditional classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2 Riemannian geometry classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2.1 On-manifold learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.2.2 Off-manifold learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3 Deep learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3.1 Discriminative models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
4.3.2 Representative models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3.3 Generative models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.4 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
iii
5 State-of-the-art in EEG data augmentation 42
5.1 Non-DL strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.1 Segmentation and recombination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.2 Sliding windows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.1.3 Transformations in temporal domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.4 Transformations in frequency domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.1.5 Geometrical transformations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.1.6 Empirical mode decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.2 Deep learning strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.2.1 Generative adversarial networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
5.2.2 Diffusion probabilistic models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
5.3 Comparison and summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6 Physiology-informed data augmentation. Material and methods 50
6.1 Material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
6.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.2.1 DA for cross-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
6.2.2 DA for within-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.3.1 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.3.2 Linear Discriminant Analysis (LDA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.3.3 Network architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
6.4.1 Cross-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
6.4.2 Within-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
6.4.3 Evaluation of generated trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
7 Physiology-informed data augmentation. Results 60
7.1 Cross-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7.1.1 Few-trial scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
7.1.2 Few-subject scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
7.1.3 Mixed scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
7.2 Within-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
7.3 Evaluation of generated data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
8 Discussion 74
8.1 Results interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.1.1 Cross-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
8.1.2 Within-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
8.1.3 Comparison to other methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8.1.4 Realistic augmented data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8.2 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
8.2.2 Classification models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.2.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
8.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
Appendices 81
A Methods implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
A.1 Dipole shift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
A.2 Choosing a particular strategy to shuffle components . . . . . . . . . . . . . . . . . . . . 81
B Cross-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
B.1 Few-trial scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
B.2 Few-subject scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
B.3 Mixed scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
C Within-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
D Evaluating generated EEG trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
iv
1 Introduction
The first electroencephalography (EEG) was published by Vladimir Pravdich-Neminsky in 1913 [Pravdich-
Neminsky, 1913], followed by Hans Berger’s discovery of the oscillatory activity of the human brain [Berger,
1929]. Since the beginning of research on brain-computer interfaces [Vidal, 1973], the field experienced a
tremendous development. Over the last two decades, advances in machine learning led to significant im-
provements in decoding of brain signals and translating them into commands for communication or control
of external devices. Recent deep learning methods provided further (even though limited) improvement.
The brain-computer interface (BCI) revolution, however, has yet to happen, and the practical applica-
tions beyond laboratory settings remain restricted. This is due to multiple difficulties associated with brain
activity decoding. They include fundamental limitations such as weak electroencephalography (EEG) am-
plitudes, low signal-to-noise ratio, and non-stationarity of the signals. Another crucial limitation is a high
variability in the data across subjects and even sessions within one subject. This leads to low accuracy of
classification and the need for long calibration time, preventing BCI systems from being broadly used. In-
creasing the size and diversity of the dataset with data augmentation methods has a potential for improving
the performance of BCI systems, and might be especially beneficial for promising neural network classifiers
that require a lot of data for training.
Data augmentation is a well-established approach in computer vision for increasing the diversity of
the training dataset and improving the generalization of the models. Multiple data augmentation meth-
ods were also proposed for EEG data [Lotte, 2015, Krell and Kim, 2017, Freer and Yang, 2020, Panwar et al.,
2020, Fahimi et al., 2021, Torma and Szegletes, 2023]. However, traditional (not using deep learning) ap-
proaches allow only limited improvement in classification accuracy, while methods based on deep learning
are difficult to train and interpret, and cannot be applied to small datasets that are common in the BCI field.
Moreover, both groups of the methods typically do not employ any expert knowledge about the data.
The purpose of this thesis is to contribute to the field of EEG data augmentation. First, we conduct a sur-
vey of the field and compare existing data augmentation methods. Then, we propose two novel approaches
that employ the information about brain physiology and head geometry. We demonstrate the effectiveness
of these methods on motor imagery (MI) classification by validating them with different classifiers in various
scenarios.
The first three chapters describe the recording, processing, and classification methods employed in BCI
systems. Specifically, in Chapter 2, we describe how the brain activity originates in neurons and how it can
be recorded in BCI systems. We review common techniques for neural activity recording and experimen-
tal paradigms utilized in BCIs. Chapter 3 is devoted to processing of measured EEG data. In particular, we
cover the artifact removal and filtering techniques, as well as feature extraction and selection methods. In
addition, we describe the process of EEG source localization, which we later employ in our methods. Chap-
ter 4 discusses trends and major challenges in EEG classification. We describe commonly employed ma-
chine learning techniques and compare traditional classification models with relatively new Riemannian
and deep learning classifiers.
The next chapters are focused on EEG data augmentation. In Chapter 5, we conduct a review of EEG data
augmentation as a solution to address high cross-subject and trial-to-trial variability in EEG data. We pro-
pose two novel data augmentation methods for cross-subject and within-subject classification in Chapter 6.
Finally, in the last two chapters, we present (Chapter 7) and discuss (Chapter 8) the results of the proposed
methods.
The key contributions of this thesis are two novel data augmentation methods for motor imagery EEG
that increase classification accuracy in BCI systems. These methods effectively address cross-subject and
trial-to-trial variability in the data, operate in a wide range of scenarios, including extremely limited amount
of training data, and facilitate calibration-free cross-subject classification.
1
2 Recording of brain activity
BCI systems allow to extract information about the mental state of a user from their brain signals. This
information is usually transferred to an external device for communication or control, but also for other
purposes [Blankertz et al., 2016, Krol et al., 2022]. For example, BCIs can be used to control a prosthetic
limb, to provide a communication tool for individuals with locked-in syndrome, or to assess pilots’ mental
state during flight.
A standard BCI system consists of six components (refer to Figure 1). The initial stage involves mea-
suring brain activity, followed by preprocessing of the recordings and extracting and classifying of relevant
features. Subsequently, commands are transmitted to an external device, smart equipment, or a computer
for practical application. Finally, the loop is closed with feedback.
Figure 1: A typical BCI system contains several stages from brain activity measurement until application,
and is closed by the feedback loop.
In this chapter, we focus on the first stage, namely the measurement of brain signals. In Section 2.1, we
provide a brief introduction to brain anatomy and neurophysiology for understanding the nature of brain
activity. Then, in Section 2.2, we compare commonly used types of recording devices that measure magnetic,
electrical, or metabolic brain activities. Lastly, in Section 2.3, we outline typical experimental paradigms
employed in BCI research.
2.1 Nature of brain signals
Gaining a comprehensive understanding of neural signals is crucial in the field of brain-computer interfaces.
The principles underlying the origin of brain activity are considered at every stage of BCI implementation.
Before performing measurements, decisions regarding recording equipment and experimental paradigms
must be made based on knowledge of where and how neural activity, corresponding to specific mental states
of the user, is generated. The later use of this information includes preprocessing, artifact removal, feature
extraction and selection, among others.
In this section, we briefly discuss the generation of neural activity. We provide an overview of the anatomy
of typical neural cells and their communication mechanisms, as well as the overall anatomical and func-
tional structure of the brain. For a more in-depth understanding of these topics, readers can refer to a neu-
roscience textbook, e.g., [Kandel et al., 2000].
2.1.1 Neuron
The brain receives signals from multiple sensors distributed throughout the body, processes them, and sub-
sequently sends commands to execute various actions, such as motor movements. The neuron, serving as
the fundamental computational unit of the brain, plays a central role in facilitating these processes. An av-
erage adult human brain is estimated to contain approximately 86 billion neurons [Azevedo et al., 2009],
actively engaging in communication with one another. Notably, a single neuron can establish up to 15000
direct connections with other neurons [Nguyen, 2013].
Anatomy of a neuron. Neurons exhibit diverse morphologies and sizes, which are tailored to their spe-
cific functional roles. A typical neuron consists of the soma, also known as the cell body, along with den-
drites and an axon that extend from it (see Figure 2). The dendrites possess extensive branching structures,
enabling them to receive signals from other cells and relay them to the soma. In contrast, the axon serves the
2
crucial function of transmitting neural signals away from the soma. Typically, a neuron possesses one axon
that is enveloped by a myelin sheath, promoting efficient signal propagation. At its end, the axon branches
into multiple extensions, each terminating in specialized structures called axon terminals, which facilitate
the transmission of signals to other neurons.
Figure 2: Schematic anatomy of a typical neuron. The neuron consists of the soma, the axon, and the den-
dritic tree. The axon is the longest myelinated filament that ends with axon terminals (based on the illustra-
tion by National Cancer Institute, SEER Training Modules).
The membrane of the neuron is made of a lipid bi-layer and contains ionic channels. These channels can
transport one or more types of ions through the membrane. Some ionic channels open and close depending
on the voltage across the membrane or on the presence of a chemical in the fluid around them. The ions,
transported through the membrane of the neuron, include sodium (Na+), chloride (Cl−), calcium (C a+2),
and potassium (K+). In the resting condition, the membrane has a steady-state potential of approximately
−70 mV, caused by the difference between intracellular and extracellular concentrations of the ions, which
is maintained by active pumps.
Action potential. When a neuron receives an electrical input from other neurons, the membrane po-
tential can be raised to a particular threshold, which generates a rapid electrical pulse called action poten-
tial (AP) or spike. APs propagate the electrical signal through the axon and play a central role in cell-to-cell
communication.
Figure 3: An action potential lasts several milliseconds. After the voltage reaches the threshold, the influx
of Na+causes membrane depolarization. The following efflux of K+repolarizes and even hyperpolarizes
the neuron. The process ends with the return to the resting state.
After the voltage threshold is reached, a sequence of events is triggered (see Figure 3). First, the mem-
brane potential rises from the rest value to 40 mV (depolarization) due to the influx of Na+. Thereafter, the
voltage-gated potassium channels open causing the efflux of K+, which leads to the return to the resting
3
potential (repolarization). Then, a slight overshoot to an even lower potential happens (hyperpolarization).
Finally, the voltage returns to the initial value (refractory period). During the refractory period, a new action
potential is impossible or very difficult to evoke. Typically, the entire cycle lasts several milliseconds.
An action potential propagates along the axon to the axon terminals. The signal can then be transmitted
to the dendrites of other neurons through synapses.
Synapse. A synapse is the structure that allows the transmission of the electrical signal from one neuron
to another. Synapses can be electrical or chemical. In electrical synapses, presynaptic and postsynaptic
cells are connected forming a gap junction, through which ionic currents can conduct. The majority of the
synapses, however, are chemical. Their transmission mechanism involves special signaling molecules called
neurotransmitters.
After an action potential arrives at the axon terminal of a chemical synapse, it causes the release of neu-
rotransmitters into the synaptic cleft, the gap between the presynaptic axon and postsynaptic dendrite (see
Figure 4). The neurotransmitters are then recognized by selective receptors on the postsynaptic cell mem-
brane. After the neurotransmitters bind to the receptors, they open ionic channels and cause a change in
the postsynaptic neuron’s membrane potential.
Figure 4: After the action potential arrives at the axon terminal of the presynaptic neuron, the neurotrans-
mitters are released into the synaptic cleft. They bind to specific receptors and generate postsynaptic po-
tential (based on the illustration by SimplyPhychology).
When the binding of neurotransmitters leads to an elevation in the membrane potential, the synapse is
considered excitatory. Conversely, if the membrane undergoes hyperpolarization, the synapse is classified as
inhibitory. These changes in the membrane potential are called excitatory postsynaptic potentials (EPSPs)
and inhibitory postsynaptic potentials (IPSPs), respectively. One EPSP is usually not enough to evoke a
spike in the postsynaptic cell. Therefore, EPSPs coming from multiple synapses nearly at the same time are
typically needed. At the same time, inhibitory synapses can prevent the action potential by IPSPs.
In Section 2.2.2.1, we will discuss that EPSPs and IPSPs constitute the brain activity that is measured by
EEG. In particular, EEG captures the collective activity of pyramidal neurons [Britton et al., 2016] situated in
layers three and five of the cerebral cortex (see Section 2.1.2). This aggregated activity can be presented as a
field with positive and negative poles (dipole). The dipole’s orientation corresponds to the alignment of the
pyramidal cells producing the activity. The location of such dipoles will play a key role in our proposed data
augmentation technique (described in Section 6.2.1).
2.1.2 Brain structure
Developing a BCI system often involves determining the specific region of the brain from which to record
neural activity. Consequently, BCI researchers require knowledge about the anatomical and functional
structures of the brain. This understanding enables them to accurately measure signals from targeted ar-
eas and differentiate them from potential artifact activity.
4
Brain organization. The brain is the largest part of the central nervous system (CNS) and consists of
three main areas: the brainstem, the cerebellum, and the cerebrum. The brainstem is a small brain structure
that connects the spinal cord with the cerebrum and transfers motor and sensory information from the rest
of the brain to the body and back. In addition, it plays a critical role in controlling heart rate and breathing.
The cerebellum lies just above the brainstem and is responsible, among other functions, for the coordination
of movements and balance. The cerebrum is the biggest part of the brain and serves numerous functions.
The cerebrum is divided into two nearly symmetrical hemispheres that are connected by the corpus
callosum. It is primarily composed of gray matter, consisting of cell bodies, dendrites, and supporting cells.
Beneath the gray matter lies the white matter, primarily comprised of myelinated axons, along with clusters
of gray matter surrounded by it. The outer layer of the cerebrum’s gray matter is known as the cerebral
cortex. The cerebral cortex is further divided into four lobes: frontal, temporal, parietal, and occipital (refer
to Figure 5). The cortex surface is characterized by folds, creating ridges called gyri and furrows known as
sulci, which significantly increase the cortical surface area.
Figure 5: Each brain hemisphere is divided into four lobes: frontal, parietal, temporal, and occipital.
Research on human cortical folding have reviled significant diversity among individuals [Van Essen et al.,
2019, Jalil Razavi et al., 2015]. For example, in the study [Van Essen et al., 2019], the group-averaged brain
atlas based on MRI scans of 210 subjects was used to demonstrate remarkable differences in brain folding
across individuals. This fact is significant for the data augmentation approach we intend to propose in Sec-
tion 6.2.
Functional systems. The brain is provided with a large amount of information. Its task is to process
and integrate this information, make decisions, and send the corresponding instructions to the rest of the
body. Depending on the function, the information is processed in particular brain areas, many of which are
located in the cerebral cortex (see Figure 6).
Figure 6: Functional areas of the human brain. The representation of most of the functions is symmetrical.
5
The motor cortex is part of the frontal lobe and is involved in planning and coordinating motor move-
ments. The primary motor cortex has sections responsible for the movements of different body parts and is
supported by two other areas, the premotor cortex and the supplementary motor cortex.
Sensory systems are responsible for the reception and processing of sensory information, such as vi-
sion, hearing, taste, smell, and touch. Typically, this information is converted into neural signals by special
receptor cells and then transferred through neural pathways to the parts of the brain involved in sensory per-
ception and processing. For example, vision is processed in the visual cortex in the occipital lobe, hearing
in the auditory cortex in the temporal lobe, and touch in the somatosensory cortex in the parietal lobe (see
Figure 6).
Interestingly, most of the functions are represented in both hemispheres symmetrically, but this does not
apply to language processing. Both Broca’s area, which is important for syntax, and Wernicke’s area, which
is responsible for language content, are located in the left hemisphere.
Most brain-computer interface (BCI) systems primarily rely on neural activity from the mentioned func-
tional systems, extracting brain signals from those specific areas. Nevertheless, some systems also incorpo-
rate signals related to emotions, concentration, or sleep processes. The choice of which signals to capture is
often influenced by the limitations of the measurement technologies, as certain technologies can only de-
tect activity from the cortical tissue. Furthermore, the recorded activity should align with the experimental
paradigm. We discuss both of these aspects in Section 2.2 and Section 2.3.
2.1.3 Summary
The neuron is the basic unit of communication in the brain. A typical neuron receives electrical input from
other neurons through the dendrites. This summed input, when strong enough to reach a particular thresh-
old, evokes the action potential, the short electrical pulse whose generation mechanism is based on the
influx and efflux of particular ions. The action potential propagates through the axon to the axon termi-
nals, where it can be transferred to another neuron through an electrical or chemical synapse, resulting in
postsynaptic potentials.
Utilizing networks of neurons, the brain performs numerous functions. The areas that are responsible
for the processing of information corresponding to different functions are located in particular brain areas.
In BCIs, this information can be used for choosing the recording equipment and experimental paradigm.
2.2 Measurement of brain signals
The first step of setting up a BCI is the measurement of brain activity. As we described in Section 2.1,
action potentials, sudden electrical impulses that occur when the membrane potential of a specific neu-
ron reaches a threshold, play a central role in cell-to-cell communication in the brain. Some of the first
recording technologies were directly measuring the changes in electrical potentials in single neurons with
invasive electrodes, or signals from large populations of neurons with non-invasive methods such as EEG.
Later, other technologies were proposed to measure neural activity indirectly. For instance, small changes
in blood flow that occur with increased brain activity can be measured with functional magnetic resonance
imaging (fMRI), or magnetic fields caused by brain electric currents are recorded by magnetoencephalogra-
phy (MEG).
In this section, we describe common measurement techniques grouped into two categories, invasive and
non-invasive. Invasive methods are using implanted electrodes that are placed under the skull or directly
into the cortex. In contrast, in non-invasive techniques, sensors are placed outside the body, on or very close
to the head. While being safer and easier to use, non-invasive methods measure brain activity further from
its sources, which results in lower-quality recordings with lower spatio-temporal resolution (see Figure 7).
2.2.1 Invasive methods
Invasive recording sensors are usually placed on the cortex surface, like in the case of electrocorticogra-
phy (ECoG), or implanted into the cortex, which is common for microelectrodes (ME) or microelectrode
arrays (MEA). In both cases, a surgery, which involves removing a part of the skull, placing the implant, and
replacing the removed part of the skull, is needed. While allowing measuring the brain activity in proximity
to the neurons, the need for surgery and potential brain tissue damage, which can happen if the sensor is
6
Figure 7: Techniques for measurement of brain activity differ in the spatial and temporal resolution. Invasive
methods typically have better spatial and temporal resolution (illustration based on [Easttom et al., 2021]).
implanted for a long enough time [Adeli et al., 2003, Villamar et al., 2018], are major limiting factors of wide
usage of invasive methods in BCIs.
2.2.1.1 Microelectrodes
Microelectrodes are electrodes that have a tip with dimensions of the order of one micrometer, small enough
to record neural activity from a single cell. They can be glass micropipettes filled with a conductive ionic
solution, metal electrodes that are typically made of platinum or tungsten, or, in more recent technologies,
silicon microelectrodes.
Intracellular recording. During intracellular recording, voltage or current across the membrane of a
single neuron is measured. One of the most common techniques is patch clamp recording (see Figure 8).
The membrane patch is isolated electrically by a tight seal of a glass pipette onto the cell membrane. This
seal is formed by placing the pipette tip close to the cell and applying a suction. The glass pipette is filled
with electrolyte solution and contains a chloride silver electrode, which is connected to an amplifier. Voltage
is measured with a reference to another electrode in the extracellular fluid outside the cell. Depending on
a research goal, one of many recording configurations can be chosen [Rubaiy, 2017]. Such recordings are
usually performed on brain tissue slices, and, more rarely, on the neurons of living animals. Therefore,
compared to extracellular recordings, this technique has very limited applicability in BCIs.
Figure 8: During the patch clamp recording, a glass pipette with electrolyte solution and a silver electrode is
sealed to the cell membrane (illustration based on [Veitinger et al., 2023]).
7
Extracellular recording. The extracellular recording is a less demanding way to measure the electrical
activity of a single cell. A tungsten or platinum-iridium microelectrode is placed close to the cell membrane,
and the voltage is measured with respect to a reference electrode. In this case, the potential fluctuations
at the cell membrane, but not across the membrane, are recorded. Therefore, the measured voltages are of
much lower magnitudes, and the shape of a measured action potential looks different (see Figure 9). Simpler
than during intracellular recordings, the procedure allows measuring from neurons in vivo in the brains of
animals, but also humans [Niediek et al., 2016].
Figure 9: Spikes measured with intracellular and extracellular electrodes. During the action potential, pos-
itive ions first flow inside the neuron (away from the extracellular electrode), and then back. This results in
different shapes of the recorded signals. In addition, the magnitude of the potential measured extracellularly
is much lower.
2.2.1.2 Microelectrode arrays
Microelectrode arrays consist of up to several hundred microelectrodes, that are typically organized in a
grid and implanted into cortical tissue (see Figure 10). Three main groups of MEAs include microwire,
silicon-based, and flexible microelectrode arrays. With MEA, it is possible to record action potentials not
only from one but also from multiple neurons simultaneously. In addition, local field potentials, that reflect
the synaptic currents and spiking activity of a local group of neurons, can be measured. Together with single
electrodes, microelectrode arrays provide the best spatio-temporal resolution among recording techniques.
Figure 10: The Utah electrode array is an example of a silicon-based MEA. It contains 100 microelectrodes
organized in a 10 by 10 grid [Campbell et al., 1991] (from [Kim et al., 2006]).
The technology has several disadvantages linked to its invasiveness, such as the need for surgery, im-
mune reactions and inflammations [McConnell et al., 2009], and a significant reduction in the measured
signal quality over time, caused by the formation of an insulating sheath around the electrodes and, as a re-
sult, increased impedance [Rao, 2013]. Nonetheless, there are successful examples of MEA applications for
communication [Kennedy et al., 2000], controlling a robotic arm [Downey et al., 2017], or moving a cursor
on a monitor [Hochberg et al., 2012].
8
2.2.1.3 Electrocorticography
Compared to microelectrodes, Electrocorticography (ECoG) is less invasive because its electrodes are not
implanted into the cortex, but placed on top of it. The cumulative activity of a large group of neurons, rather
than direct spike activity, is measured. ECoG, therefore, is perceived as a compromise between invasive mi-
croelectrode arrays and non-invasive EEG (described in Section 2.2.2.1). While being safer than MEAs, with
reduced risk of inflammation, rejection, and encapsulation of the electrodes, ECoG records neural activity
closer to the sources than EEG. This results in less distorted signals with higher amplitudes, broader spectral
bandwidth, and noticeably reduced influence of artifacts.
So-called microECoG sensors that consist of grids with smaller electrodes (a fraction of a millimeter
instead of several millimeters in diameter) provide much higher spatial resolution [Kellis et al., 2010]. This,
for instance, allows high-accuracy finger movement decoding [Wang et al., 2009].
ECoG is typically utilized in laboratory or clinical settings, such as pre-surgery monitoring in epilepsy
patients. However, there are some applications, where ECoG-based BCIs are also used outside the labora-
tory. For example, individuals with locked-in syndrome can use them at the bedside in clinics and even at
home [Vansteensel et al., 2016].
2.2.2 Non-invasive methods
Unlike invasive methods, non-invasive recordings do not require implantation surgery. The recording sen-
sors are placed outside the body. This makes them more attractive for usage in BCI systems and expands the
range of their applications beyond the laboratory setup.
2.2.2.1 Electroencephalography
EEG is the most convenient and widely used method for BCI recordings. The recording electrodes are placed
on the scalp. Like in ECoG, the activity from a large group of neurons is measured. As we described in
Section 2.1, after a spike occurs, neurotransmitters are released in synapses, causing postsynaptic potentials
in the dendrites. EEG measures the cumulative activity of postsynaptic potentials that are oriented radially
to the scalp. As mentioned in Section 2.1.1, the majority of this activity is generated by pyramidal neurons,
whose cell bodies are predominantly located in layers three and five of the cerebral cortex and are organized
in columns perpendicular to the cortical surface. This activity can be represented by a dipole. The activity
from a deep source is difficult to capture by EEG, since the voltage drops with the square of the distance.
The electrodes are positioned on the scalp according to a so-called 10-20 system (see Figure 11). The
front-back line is divided into parts corresponding to 10% or 20% of the total distance from nasion to inion
points. The first electrode is placed at 10% distance and the next ones at 20% from each other. The same
procedure is applied to the left-right line, and, similarly, to the parallel lines. Electrode names contain letters
that refer to the brain lobe or area, such as Pfor parietal or Ofor occipital lobes, and numbers, odd for the
left hemisphere and even for the right. There are different variations of EEG electrode configurations, which
can include from 1 up to 256 electrodes, that are based on the 10-20 system and have similar logic.
Depending on the brain activity, oscillatory signals with different frequencies can be captured by EEG (see
Table 1). Each wave type has its characteristic frequency band and is associated with a functional state of
the brain. This information, together with the knowledge about spatial representations of brain functions,
is used in EEG-based BCIs during the preprocessing step.
Table 1: Frequency bands in EEG signals and corresponding mental states.
Wave type Frequency band (Hz) Mental state
Delta 0.3−4 Unconsciousness, deep sleep
Theta 4−8 Drowsiness, deep meditation
Alpha 8−13 Awake, relaxed, eyes closed
Beta 13−30 Active thinking, eyes opened
Gamma 30−100 Cognition, information processing
On the way from the source to the electrode, the signal passes through different tissues, such as the brain,
cerebrospinal fluid, skull, and scalp, that act as conductive volumes and low-pass filter the original signal.
The recorded signal has low amplitude and can be corrupted by numerous artifacts. They include, among
9
Figure 11: Standard 10−20 system of positioning of EEG electrodes on the scalp.
others, physiological artifacts, for example, eye movements, eye blinking, heartbeat, sweat, or chewing; and
technical artifacts, such as electrode pop, cable movements, or line noise. Overall, it leads to a low signal-
to-noise ratio (SNR) and the need for a complex preprocessing procedure.
However, low SNR and low spatial resolution of EEG are compensated by low cost, high temporal resolu-
tion, easy setup, and portability. As a result, EEG became the most popular measurement method for BCIs.
EEG-based BCIs are used for various medical applications, including motor rehabilitation after a stroke or
other neurodegenerative diseases [Mulder, 2007, Silvoni et al., 2011], assistive communication devices [Al-
Saleh et al., 2016], control of a prosthetic device or a wheelchair [Bandara et al., 2018, Ru¸sanu et al., 2020],
or medical diagnosis [Houmani et al., 2018]. Besides medicine, they have been used in other fields [Van Erp
et al., 2012], such as education [Gomarus et al., 2009], entertainment [Tangermann et al., 2008, Liao et al.,
2012], marketing [Rawnaque et al., 2020], and others [Portillo-Lara et al., 2021].
2.2.2.2 Magnetoencephalography
Magnetoencephalography (MEG) records brain neural activity by measuring the magnetic field produced
by brain electric currents (see Figure 12). For that, very sensitive magnetometers, typically, arrays of su-
perconducting quantum interference devices (SQUIDs) are used. The recorded magnetic field reflects the
postsynaptic currents in the dendrites, the same measured by EEG. However, the magnetic field is orthogo-
nal to these currents. Therefore, MEG is sensitive only to the tangential to the scalp currents that commonly
flow in cortical sulci rather than gyri, whereas EEG detects both of them.
The measured magnetic field is on the order of a few femtoteslas and interferes with external magnetic
signals, including the earth’s magnetic field. Therefore, shielding, such as a magnetically shielded room
constructed of aluminium and mu-metal, is required. In addition, SQUIDs have to operate at very low tem-
peratures to maintain superconductivity, which is achieved by cooling them with liquid helium. Together
with shielding, it leads to high costs for the whole recording system. As an alternative, a highly sensitive spin-
exchange free relaxation (SERF) magnetometer of much lower cost can be used. Partially, the cost of a SERF
magnetometer is reduced by shielding it using mu-metal cylinders instead of magnetically shielded rooms,
which is possible due to the smaller size of the sensors. While having also several disadvantages compared
to SQUID [Wakai, 2014], SERF magnetometers are being investigated for future cheaper and more portable
applications [Boto et al., 2018].
Alike EEG, MEG has a high temporal resolution. In addition, magnetic fields measured by MEG are not
distorted by the skull and scalp, which leads to better spatial resolution. However, the BCI usage of MEG
remains limited because it requires a magnetically shielded room, a bulky and non-portable cooling system,
and has considerably higher costs than EEG.
10
Figure 12: Magnetoencephalography recording setup (illustration by National Institute of Mental Health,
Department of Health and Human Services).
2.2.2.3 Functional magnetic resonance imaging
Functional magnetic resonance imaging (fMRI) records neural activity indirectly by measuring the changes
related to blood flow. A local increase in neural activity creates a greater demand for oxygen, which is sup-
plied by an increase in highly oxygenated blood flow [Gore, 2003]. Oxygen is carried by the hemoglobin
molecule in red blood cells. fMRI records the magnetic differences between oxygenated hemoglobin and
more magnetic deoxygenated hemoglobin. This recorded signal is known as the blood oxygenation level-
dependent (BOLD) response. As a result, fMRI generates scans of brain cross-sections representing in-
creased neural activity, for example, as a response to external stimuli. During the recording, a subject is
placed into the fMRI scanner (see Figure 13a), where different stimuli, such as sounds or visual scenes, can
be presented.
(a) fMRI scanner (b) fNIRS cap
Figure 13: Recording technologies based on the detection of hemodynamic response to neural activity.
(a) fMRI scanner offers high spatial resolution and whole brain measurement (illustration by Economic-
sUZH, licensed under CC BY-SA 4.0). (b) fNIRS is inexpensive, portable, and allows body movements during
the measurements (illustration by Mentorklaspsy, licensed under CC BY-SA 4.0).
The major advantage of fMRI is high spatial resolution. In this aspect, fMRI works better than other non-
invasive techniques such as EEG or MEG. However, the change of BOLD signals is relatively slow, it starts
several hundred milliseconds after the actual neural activity and peaks at 3 to 6 seconds [Rao, 2013]. Thus,
fMRI offers low temporal resolution. In addition, the technology is expensive, produces operational noise,
and allows recording only from a subject that stays completely still.
11
2.2.2.4 Functional near-infrared spectroscopy
Alike fMRI, functional near-infrared spectroscopy (fNIRS) is designed to detect changes in oxygenated and
deoxygenated hemoglobin concentrations [León-Carrión et al., 2012]. But instead of using the paramag-
netic properties of hemoglobin, it relies on the different absorption properties of biological chromophores,
the atoms within a molecule that absorb the light at different wavelengths and generates its color. The near-
infrared light can pass through the scull and enter a few centimeters into the cortex, while being absorbed
only by a few biological chromophores, including hemoglobin [Scarapicchia et al., 2017]. Its spectrum varies
in the oxygenated state, thus, the concentrations of oxygenated and deoxygenated hemoglobin can be mea-
sured by near-infrared light transmitted through the tissue.
The fNIRS cap (see Figure 13b), which looks similar to EEG, contains emitters and detectors placed on
the head at a fixed distance from each other. However, recording with fNIRS is less sensitive to muscle
artifacts than EEG, since it relies on optical instead of electrical measurements.
fNIRS is a small, inexpensive, and portable alternative to fMRI, offering recording in freely moving par-
ticipants. Compared to fMRI, fNIRS has similar temporal, but rather poor spatial resolution. Moreover, it
enables recording of cortical tissue only, while fMRI can scan the entire brain’s neural activity.
2.2.2.5 Positron emission tomography
Positron emission tomography (PET) measures brain activity indirectly based on metabolic activity detec-
tion (see Figure 14). It uses radioactively labeled substances, known as radiotracers, that are injected into the
bloodstream and transported to the brain. They end up in different brain areas depending on the metabolic
activity caused by neural activity. Then, the emissions of the radiotracers are measured by PET. Typically, a
labeled form of glucose is used as a radiotracer [Rao, 2013].
Figure 14: Positron emission tomography scanner ("PET/CT scanner" by Frank Kehren, licensed under CC BY-NC-
ND 2.0.).
The main advantages of PET in comparison to fMRI are that it can trace pathways of the substances
in the brain, and allows small movements during the measurement. The spatial and especially temporal
resolution of PET is inferior to fMRI. In addition, PET requires the injection of radioactive substances, which
can be used on a participant only a few times until it is unsafe. Furthermore, the radioactivity decays rapidly
and, therefore, limits the recordings to monitoring only short-time tasks.
2.2.3 Summary
Invasive methods offer direct measurement of neural activity from relatively small groups or even single
neurons. They allow recording action potentials and generally provide higher information transfer and more
accurate BCI control than non-invasive techniques. In addition, they offer excellent spatial and temporal
resolution and signal recording at higher frequencies. Unfortunately, all these advantages come at the cost
of implantation surgery and poor biocompatibility of the implants, which leads to degradation of recording
quality with time and possible health risks for subjects. Accordingly, before a way to handle these problems
is found, broad usage of invasive recording in BCIs is not possible.
12
In contrast, non-invasive methods do not require operation and provide a safer and easier alternative for
measuring brain activity, which makes them more suitable for human BCIs. However, many of those meth-
ods measure neural activity indirectly, which significantly decreases their temporal resolution. In addition,
methods like MEG, fMRI, and PET are non-portable, expensive, and require special magnetic shielding (in
the case of MEG) or the usage of radioactive substances (in the case of PET). Overall, most of these methods
find their applications in BCIs, but rarely go beyond the laboratory setup.
While being a safe non-invasive method, EEG directly measures electrical activity, has good temporal
resolution, is inexpensive and portable. Due to these factors, EEG-based BCIs are most commonly used and
have many medical and nonmedical applications. Therefore, from now on, our main focus will be on BCIs
that use EEG as a neural activity recording technique.
2.3 BCI paradigms
To implement an EEG-based BCI for a particular application, an experimental paradigm with a correspond-
ing control signal has to be chosen. Based on the selected paradigm, in the training phase, a user performs a
specific task while their EEG signal is captured. The recorded data is used to train a neural decoder designed
for the paradigm. Subsequently, in the utilization phase, the user performs the same task while the decoder
interprets the recently measured control signals and employs them for BCI purposes.
In this section, we describe common paradigms for EEG-based BCIs. We also discuss their advantages
and disadvantages considering how they match the expected control for a particular application, and the
physical state of a participant.
2.3.1 Motor imagery
Motor imagery (MI) is a process of imagining a movement of a part of the body, e.g., left or right hand
or foot, without moving it physically. Imagining a motor movement leads to a similar neural activation
pattern to the one observed during the actual movement [Pfurtscheller and Neuper, 1997]. In particu-
lar, it causes event-related desynchronization (ERD), while relaxation results in event-related synchroniza-
tion (ERS) [Pfurtscheller and Neuper, 2001].
During ERD/ERS, a relative power decrease/increase in EEG occurs in certain frequency bands. Imag-
ining motor movements causes ERD in mu (8 −12 Hz) and beta (18 −26 Hz) rhythms (the mu band is the
alpha band recorded from the sensorimotor cortex [Hassanpour et al., 2019]), which corresponds to cortical
neurons firing in a less synchronized way compared to the rest condition. ERD and ERS are most visible in
EEG signals acquired from EEG channels corresponding to brain areas where a particular motor task origi-
nates (see Figure 15). For example, MI of the left hand causes changes in channel C4 and of the right hand
in channel C3 (see Figure 16), while ERD/ERS during imagery movements of feet are most visible in chan-
nel Cz. This information can be employed by BCI systems to decode the commands sent by a user through
imagination.
While the functional regions remain uniform across individuals (see Figure 15), there exists notable di-
versity in brain folding (discussed in Section 2.1.2). Consequently, even the same groups of pyramidal neu-
rons, which can be represented as source dipoles, might be spatially shifted, introducing a challenge in the
analysis of recorded EEG signals. We will address this challenge by proposing a data augmentation method
in Section 6.2.
Various applications of MI-based BCIs have been investigated. They include after-stroke rehabilita-
tion [Cheng et al., 2018], control of prosthetic devices [Elstob and Secco, 2016, Williamson et al., 2017], and
gaming [Wang et al., 2019, Liao et al., 2012].
MI-based BCIs do not require any external stimulus, are intuitive to use, and allow users to initiate the
command at their will. However, it can be difficult for participants to find the correct imagination strategy
and learn how to send the corresponding command. In addition, the number of commands that can be
decoded by the BCI system is limited to 3 or 4 imagery motor movements. The performance of the system
declines rapidly with the number of commands [Kronegg et al., 2007].
2.3.2 Event-related potentials
An event-related potential (ERP) is a recorded EEG response to a stimulus, which can be a sensory, cognitive,
or motor event. The brain produces signals in fix-time relation to the event. ERPs typically have low ampli-
13
Figure 15: Primary motor cortex area in the right hemisphere with approximate locations of the correspond-
ing EEG channels. Activity measured by channels C4 and Cz is associated with imagery movements of the
left hand and feet, respectively (adapted from [Lotte et al., 2015]).
Figure 16: Power spectral density (PSD) of EEG signals measured during motor imagery tasks. In channels
C3 (on the left) and C4 (on the right), ERD in mu (8−12 Hz) rhythm can be observed. ERD occurs in channel
C3 while performing imagery movement of right hand versus left hand. The opposite happens in channel
C4, where ERD occurs during left-hand imagery movements.
tudes compared to background EEG activity. Therefore, ERPs are obtained by averaging multiple recorded
trials.
Some ERPs are called according to an event that triggered them, such as somatosensory evoked poten-
tial (SEP) (tactile event) or error-related potential (ErrP) (error event). However, most of them are referred
to by the letter P (for positive deflections) or N (for negative deflections), followed by a number, that re-
flects the timing of ERP. For example, N100 indicates that the response appears 100ms after the stimulus
and is negative. The early responses, which peak 20 to 100ms after the stimulus, are automatic responses,
also called exogenous, whereas later ERPs, occurring after 200ms, reflect cognitive processing and are called
endogenous.
Various stimuli evoke ERPs at different timings [Sur and Sinha, 2009]. Among these, the most commonly
utilized ones are visual P300 and ErrP.
Visual P300. The P300 component is one of the most studied and used. It is an endogenous event-
related potential that is derived by averaging EEG trials of a specific event type. To record a P300, the oddball
14
paradigm is typically used, in which a low-probability trigger event is abruptly presented in the sequence
of high-probability standard events. The trigger event causes the P300 wave, which is a positive peak with a
latency of roughly 250 to 500ms. The highest increase in measured amplitudes of this ERP can be observed
at midline locations, in particular, in channels Pz, Cz, and Fz.
Since 1988, the visual P300 has been used in BCIs for so-called P300 spellers [Farwell and Donchin, 1988],
offering a tool of communication for disabled patients. Speller devices [Pan et al., 2022] usually use a matrix
of letters, numbers, or symbols, which rows and columns are flashed in sequence. The user has to focus
their attention on the intended character. The P300 is triggered every time after a row or column with the
intended symbol is flashed, which is used by the BCI to identify and type the correct symbol. In order to
improve the accuracy and speed of typing, many paradigm modifications have been proposed, including
different forms of character presentations [Townsend et al., 2010, Shi et al., 2012] and usage of language
models [Moghadamfalahi et al., 2015]. Besides visual P300, spellers also employ visual and tactile P300
waves [Chang et al., 2014a, Höhne et al., 2011, van der Waal et al., 2012].
Apart from being used in spellers, the P300 component has also been used to control a computer cursor
in 2D [Citi et al., 2008], navigate a wheelchair [Iturrate et al., 2009], control a virtual hand in VR [Edlinger
et al., 2009], and for other specific tasks [Aydın et al., 2016, Abiri et al., 2019].
The big advantage of P300 BCIs is the fact that most people can use them with high accuracy after rela-
tively short calibration. It does not involve complicated training procedures and can be quickly used. How-
ever, it requires a high level of attention [Treder and Blankertz, 2010] and visual/auditory focus.
Error-related potential. Another commonly used ERP is error-related potential (ErrP), which is typi-
cally used to improve BCI performance [Chavarriaga et al., 2014]. ErrP occurs when there is a mismatch be-
tween the user’s expectation and the actual response of the BCI system. It is recorded mainly in the frontal
and central lobes, in particular, FCz and Cz channels, with a latency of 200 to 700ms after the erroneous
event.
After an ErrP is detected during the usage of BCI, it can be integrated into its control loop, which was also
shown in an online study [Schmidt et al., 2012]. For example, while using a P300 speller, ErrP can be used to
cancel the preceding wrong letter [Ferrez et al., 2023], or replace it with the best second letter [Visconti et al.,
2008]. Both strategies have been shown to improve speller performance. However, the error signal is often
subject to delays, which can hinder the real-time correction process and pose challenges for implementation
in some BCI systems. Additionally, the ErrP lacks specificity regarding the nature of the error, making it
unclear how corrections should be applied.
ErrPs have been successfully used in different BCIs for robot reinforcement learning [Iturrate et al., 2013],
cursor control [Iturrate et al., 2015], 2D reaching task [Omedes et al., 2015], and other purposes [Kumar et al.,
2019].
The usage of ERPs in BCI has its pros and cons. On the one hand, ERPs are produced by most users without
any particular training and appear at relatively short latencies. On the other hand, they have low amplitudes
compared to the background activity and often require a high level of user attention. In addition, the laten-
cies and waveforms of some ERPs vary a lot from subject to subject, which makes it more difficult to train
the BCI system to recognize individual ERPs.
2.3.3 Steady-state evoked potential
Steady-state evoked potentials (SSEPs) are recorded when a subject perceives a stable frequency oscillatory
stimulus. In the case of steady-state visual evoked potential (SSVEP), a visual stimulus, commonly a picture
flickering at the frequencies between 3 Hz and 75 Hz, is presented, while the subject is instructed to focus
their gaze and attention on it. The increased EEG power at the frequency of the stimulus, typically measured
in the occipital cortex, characterizes the SSEP. Interestingly, the magnitude of the measured SSEP correlates
with the level of the user’s attention. BCIs also employ auditory and somatosensory SSEP, where amplitude-
modulated sound and vibrotactile sensors, respectively, can be used as stimuli. However, SSVEP is by far the
most used SSEP for BCI design.
In BCIs, multiple visual stimuli flickering at different frequencies can be presented to the user. Each of
them is associated with a particular command. To send the intended command, the user attends to the
corresponding stimulus, and the SSVEP with the frequency of the stimulus can be detected. The possibility
of using many frequencies results in a BCI system with many commands and degrees of freedom. Since the
15
stimuli are exogenous, no particular training is needed, and therefore, the paradigm is available for many
users. In addition, SSVEPs can be classified more accurately than ERPs [Abiri et al., 2019]. The main limiting
factor is the need for visual stimuli and dependence on the user’s attention. The flickering stimuli could also
cause fatigue in subjects, especially when presented at low frequencies [Chang et al., 2014b]. In addition,
the paradigm is not well suited for users with visual impairments.
The possibility of having multiple commands and reliable detection of SSVEPs is employed in many BCIs.
Among many other applications, SSVEP BCIs proved themselves to be effective in wheelchair control [Li
et al., 2013], gaming [Chen et al., 2017], and high-speed spellers designs [Chen et al., 2015].
2.3.4 Summary
A paradigm determines how and which control signals are recorded to transfer commands to the BCI system
in the most efficient way. Besides aiming at high accuracy performance, the BCI system should be reliable
and convenient for the target group of users.
In this section, we described the most widely used and researched paradigms for EEG-based BCIs, dis-
cussed their benefits, drawbacks, and common applications. Even though ERPs and SSEPs can be more
reliable and easier to use, and proved to be especially efficient in particular applications, the motor imagery
paradigm is the most used and investigated [Aggarwal and Chugh, 2022]. The main reason for this is that
MI-based BCIs do not require an external stimulus and their operation can be initiated voluntarily by the
user, which offers a wider range of applications in both medical and nonmedical fields.
2.4 Chapter summary
In order to provide an understanding of the principles of BCI systems implementation, this chapter covered
the path of neural activity from its origin in single neurons to the signal recordings, obtained with measure-
ment equipment.
In Section 2.1, we described the structure of the neuron and the mechanism of spike generation and
transmission. In addition, we shortly introduced the anatomical and functional structure of the brain.
After providing a brief introduction into neuroanatomy and neurophysiology aspects that are relevant
to BCIs, we reviewed common techniques for neural activity recording. Although invasive measurement
methods provide high spatio-temporal resolution and allow recording close to the sources, they come with
health risks and problems with long-term usage. Among safe non-invasive methods, the most employed is
EEG, which provides the best combination of resolution, portability, costs, and user-friendliness.
The implementation of a BCI system also requires a choice of experimental paradigm, which will define
the exact type of the setting, measurement, and assisting equipment, i.e., devices producing stimuli or pro-
viding feedback. The MI paradigm assumes sending commands by imagining a motor movement, ERPs are
measured as a response to a trigger event, which can be visual, auditory, and somatosensory, or, for exam-
ple, an erroneous event, while SSEPs occur as a response to a rather prolonged stimulus. An independence
of the external stimulation and the possibility to send a command at user’s will make the MI paradigm the
most used and suited for multiple applications. Considering this fact, in the following chapters, we focus in
particular on motor imagery EEG-based BCIs.
16
3 Processing of EEG data
Preprocessing of EEG data is a crucial step in preparing the data for analysis, as it helps to eliminate un-
wanted noise and artifacts that can contaminate the signal and obscure the underlying neural activity. The
primary goals of preprocessing include applying digital filters to remove noise, identifying and correcting
artifacts such as eye blinks and muscle activity, segmenting the signal into epochs corresponding to specific
experimental conditions or events, changing the reference electrode to improve signal quality, and adjusting
the signal to remove baseline drift or offset. These preprocessing methods are described in Section 3.1.
In EEG analysis, feature extraction and selection aim at identifying the most informative features from
the EEG signal to study the underlying neural activity. Feature extraction transforms raw EEG data into fea-
tures that capture essential characteristics, while feature selection identifies a subset of relevant features
to improve classification accuracy and interpretability. The goal is to reduce dimensionality while retain-
ing essential information. Section 3.2 describes various methods used to extract and select features from
temporal, spectral, and spatial domains in EEG data.
Section 3.3 is focused on the topic of EEG source localization, which we employ in our method (see Sec-
tion 6.2.1). The process of source localization consists of decomposing measured EEG data into individual
components, choosing or constructing a head model that estimates the potentials on the scalp based on the
underlying sources of neural activity, and solving the inverse EEG problem where the location, strength, and
orientation of the sources are identified based on the measurements. Locations of EEG sources, besides be-
ing of scientific interest themselves, have been used for removing artifact-related data and selecting relevant
components for further classification.
3.1 Preprocessing and artifact removal
3.1.1 Artifacts
As we mentioned in Section 2.2.2, measured EEG data has low amplitudes and can be corrupted by multi-
ple artifacts, which leads to a low signal-to-noise ratio. All the artifacts can be combined into two groups:
physiological and non-physiological. Physiological artifacts are generated by the human body, and non-
physiological artifacts are caused by external sources.
Physiological artifacts. Physiological artifacts are caused by electrophysiological activity from the hu-
man body, excluding the brain. The most common physiological artifacts in EEG are ocular-related, e.g., eye
movements and blinking, related to muscle activity, and produced by the heart.
Ocular-related artifacts such as eye blinking and eye movements influence EEG recordings because dif-
ferent parts of the eye have different electric charges. The cornea is charged positively, and the retina is
charged negatively. Eye blinking causes the upward and outward movement of the eye [Collewijn et al.,
1985], which produces a positive waveform in frontal electrodes. Similarly, lateral eye movements yield
large electrical potentials close to the temples, in particular, in electrodes F7 and F8 (see Figure 11 for elec-
trode positions on the scalp). The amplitude of the ocular-related artifacts is typically much higher than the
amplitude of background EEG [Croft and Barry, 2000]. However, such artifacts can be easily detected during
the recording and removed. Placing electrodes below and above the eyes can help to identify ocular-related
activity. Blind source separation techniques, such as independent component analysis (ICA), are also used
to remove these artifacts.
Muscles, when contracted, also produce electrical activity. High-frequency artifacts appear in EEG record-
ings during head movements, forced eye closures, jaw clenching, chewing, and bruxism. The amplitude of
the noise depends on the force of the corresponding muscle contraction [McMenamin et al., 2010]. It is
difficult to measure isolated muscle-related activity directly. During the recording, subjects are typically
instructed to relax, avoid movements, and minimize muscle artifacts. In addition, multiple approaches in-
cluding filtering, linear regression, and source decomposition [Minguillon et al., 2017] are applied to reduce
the effect of muscle artifacts on EEG recordings.
Another source of physiological artifacts is cardiac activity. The electrical activity of the heart causes low-
amplitude artifacts with a very specific waveform and frequency. They are more likely to occur in the elec-
trodes on the left hemisphere and depend on the body type of the participant. Similarly to ocular-related,
cardiac activity can be measured separately from brain signals. Hence, parallel recording of the electrocar-
diogram enables the recognition and removal of the heart-related artifacts.
17
Physiological artifacts can be avoided, rejected, or removed. Proper instruction of participants helps
to avoid some artifacts, especially ones related to eye and muscle movements. However, some artifacts
inevitably appear in EEG recordings. Physiological artifacts that have characteristic electrical fields can be
relatively easily identified and removed. Nevertheless, sometimes the whole segments of corrupted EEG
measurements have to be rejected.
Non-physiological artifacts. Non-physiological artifacts are caused by external sources located any-
where near the EEG recording setup. Typical external artifacts include mains interference, electrode pop,
and cable movements.
One of the most common external artifact sources is the alternating mains electricity of 60 Hz in North
America and 50 Hz in most of other countries. Mains interference artifact causes frequent, monotonous
waves at the corresponding frequencies. The influence of this artifact can be reduced by using a shielding
cable, connecting a participant to the ground, using short electrode wires, or moving the EEG system far
away from the source of interference in the room. In addition, mains interference artifacts can be removed
with a notch filter [Urigüen and Garcia-Zapirain, 2015].
Another common external artifact is an electrode pop, which can occur when the electrode is poorly
placed on the scalp and becomes loose. This artifact is characterized by a sudden shift to the new offset (DC)
or measured signals going out of range. To prevent this artifact, the electrodes have to be properly applied
and maintained. Tight fixation on the scalp before recording and checking and replacing the electrodes
where the artifacts appear during the measuring helps to reduce the artifact.
Cable movements change their conductive properties and cause artifacts in EEG recordings [Syme-
onidou et al., 2018]. Depending on the movement, it can cause different distortions to the measured sig-
nals. Common cable swinging produces oscillations at the frequency of the swing, which may overlap with
the frequencies of interest. Active electrodes allow pre-amplification modules close to the electrode, which
amplifies the signal before the artifact may occur. In addition, cable swinging artifacts can be removed by
filtering, but only if the swinging frequency does not overlap with the frequency of EEG oscillations.
Non-physiological artifacts can be minimized by adjusting the environment, for example, shielding the
cables, connecting a participant to the ground, or properly placing the electrodes. Additionally, using active
electrodes can reduce the influence of external artifacts. In contrast to physiological artifacts, many non-
physiological artifacts do not have particular characteristics and are more difficult to identify. Therefore,
they are more likely to cause corruption of the EEG data.
3.1.2 Filtering
A typical BCI system contains a preprocessing step (see Figure 1). In this step, one of the goals is to remove
physiological and non-physiological artifacts from measured EEG data. Filtering is the most commonly used
method in the preprocessing of EEG signals. Temporal and spatial filters applied to the measured data can
significantly improve the performance of BCIs.
Temporal filters. Temporal filters are used to remove artifacts and retain the frequencies correspond-
ing to the signal of interest. Depending on which frequencies are removed or retained, a temporal filter can
be low-pass, high-pass, band-pass, or band-stop (see Figure 17).
The low-pass filter remains low frequencies below the particular threshold and removes or attenuates
higher frequencies. For example, it can be used to remove high-frequency artifacts caused by muscle con-
traction. In contrast, a high-pass filter removes low-frequency signals such as slow drifts. Applying a band-
pass filter keeps the signals between lower and upper bounds. In the MI paradigm (see Section 2.3), it can be
used to retain signals in the range between 8 and 30 Hz. It contains both the mu and beta frequency bands
that are associated with motor-related tasks and contains corresponding ERDs (see Section 2.3.1). Finally,
for a band-stop filter, frequencies between the lower and upper bound are removed or attenuated. For ex-
ample, the notch filter, that we mentioned in Section 3.1.1, is a particular band-stop filter with a narrow stop
band. It is often applied to EEG data to remove mains interference artifacts.
Applying temporal filters can cause unwanted filtering artifacts at the edges of the signal. Therefore, it is
recommended to apply them to continuous rather than segmented EEG data. Otherwise, this edge artifact
may in some cases last even longer than the corresponding segment [Cohen, 2014].
18
Figure 17: Depending on the type of the temporal filter, different frequencies are removed and remained.
Spatial filters. Some background noise can also be eliminated by spatial filters. If the noise is present
in two neighboring electrodes, but the signal of interest is measured only by one of them, then the difference
between the two channels will represent the signal, but not the noise. Therefore, linear combinations of the
electrodes, known as spatial filters, can remove the noise from EEG measurements. Simple spatial filters
include bipolar, Laplacian, and common average reference (CAR) filters (see Figure 18).
Figure 18: Three spatial filters were applied to F z,C3,C z, and C4electrodes. The values of correspond-
ing (shown in grey) electrodes are subtracted from them. One neighboring electrode is used for bipolar
filtering, four electrodes for Laplacian, and all measured electrodes for CAR.
Applying a bipolar filter means applying a simple subtraction of one electrode signal from another. In a
simple variant of the Laplacian filter, the average value of four neighboring channels is subtracted from the
target channel [McFarland et al., 1997]. CAR refers to a filter where the average of all channels is taken as
zero, and each channel is re-referenced to it.
19
3.1.3 Other preprocessing techniques
Besides filtering, many other preprocessing steps, including segmenting, baseline correction, removal of
corrupted data, artifact removal with ICA, normalizing, and downsampling, can be applied to EEG data [Al-
Saegh et al., 2021, Hu and Zhang, 2019].
In the MI paradigm, EEG data is time-locked to the particular event of interest, the onset of the motor
imagery task. For further processing and classification, continuous EEG data has to be segmented, i.e.,
cut into epochs that contain corresponding MI tasks. After the epoching, the dimensionality of EEG data
changes from 2D(electrodes ×time) to 3D(electrodes ×time ×epochs), where the length of one epoch is
typically much smaller than the length of the measured continuous data.
Measurements of some electrodes can be corrupted due to multiple reasons, such as no contact be-
tween the electrode and the scalp, bridged electrodes, or saturated channels. Such electrodes are usually
completely removed from the analysis. After channel removal, we might have data with different channels
across trials or participants, which can cause inconveniences for further classification. In this case, these
channels have to be removed also in other trials, which leads to loss of information. Alternatively, corrupted
channels can be interpolated, for example, by spherical splines [Greischar et al., 2004]. Occasionally, the
entire epoch is rejected if all or many electrodes are corrupted. This can happen due to a significant con-
tamination by artifacts or a participant error.
Among other commonly used preprocessing techniques are artifact removal with different source de-
composition methods, clipping and normalizing data to speed up the convergence and avoid trapping by
local minima, downsampling the signal for speeding up the computation and reducing memory storage,
and others [Al-Saegh et al., 2021].
3.2 Feature extraction and selection
Motor imagery EEG captures a lot of raw data measured from many channels with a relatively high sam-
pling rate. The goal of the feature extraction step is to transform this data into numerical features. Ideally,
obtained features lead to the best classification performance by preserving all the information needed for
the classification, and, at the same, not being redundant. MI feature extraction methods can be divided into
five groups: temporal, spectral, time-frequency, spatial, and spatio-temporal methods (see Table 2). In this
section, we briefly describe these methods. More information is available in extensive reviews on feature
extraction methods for all types of BCIs [Khosla et al., 2020, Rashid et al., 2020, Stancin et al., 2021], and for
MI EEG in particular [Singh et al., 2021].
Table 2: Common feature extraction methods for MI EEG
Features
Temporal methods
Statistical features
Hjorth features
RMS, IEEG
Autoregressive modeling
Entropy
Spectral methods
Band power
Spectral entropy
Spectral statistical features
Time-frequency methods
STFT
CWT
DWT
Spatial methods
BSS
CSP
SSD
Spatio-temporal methods Sample covariance matrices
20
3.2.1 Temporal methods
Temporal features represent the change of an EEG signal as a function of time. They are extracted from each
channel separately and are merged into one feature vector, which is used for classification.
The simplest temporal features are statistical features such as mean, standard deviation, variance, root
mean square (RMS), integrated EEG (IEEG), and others [Khosla et al., 2020]. Based on the variance of EEG
signal x(t), a group of features called Hjorth parameters can be calculated:
Act i vi t y =v ar (x(t)),
Mobi li t y =sAc ti vi t y(x′(t))
Act i vi t y(x(t)) ,
Compl exi t y =Mobi l i t y(x′(t))
Mobi li t y(x(t)) .
Hjorth activity, mobility, and complexity represent signal power, mean frequency, and similarity of the shape
of the signal to a pure sine wave, respectively. Information theory-based features such as entropy are also
used to represent the complexity of EEG signals.
Another common approach for feature extraction in the time domain is autoregressive (AR) modeling.
The EEG signal for each channel is modeled as a weighted sum of the previous values:
xt=
p
X
i=1
αixt−i+nt,
where pdenotes the number of time points in the past used for the modeling, ntis zero-mean noise, and
αiare AR coefficients. After these AR coefficients are estimated, they are concatenated into feature vectors.
AR modeling and its adaptations have been used in many recent studies [Rodríguez-Bermúdez and García-
Laencina, 2012, Lawhern et al., 2012, Chai et al., 2017].
In addition, peak-valley modeling, fractal dimension, quaternion modeling, and other methods [Singh
et al., 2021] have been used for the extraction of temporal features.
3.2.2 Spectral methods
Analysis in the frequency domain has also been utilized for EEG feature extraction. Most of the spectral
methods calculate the power of EEG signals in different frequency bands. Typically, these methods use fast
Fourier transform (FFT) [Bousseta et al., 2018, Yang et al., 2018] or power spectral density (PSD) [Nguyen
et al., 2017, Chakladar and Chakraborty, 2018]. PSD is calculated with FFT and Welch’s method [Oikonomou
et al., 2017].
Signal band power can also be estimated by the variance of the data, like in Hjorth Activity. After the
band-pass filtering, the variance of the signal is calculated within frequency bands of interest. A logarithm
transformation spreads the resulted features, shifts their distribution closer to Gaussian, and makes covari-
ance matrices of the classes more similar. Subsequently, feature vectors can be well classified with linear
models [Dickhaus et al., 2009], i.e., linear discriminant analysis (LDA). Interestingly, the log transform has
also been applied to other features, such as RMS and IEEG, to make more separable feature vectors [Hamedi
et al., 2014, Momen et al., 2007]. For better separability of the classes, it is also important to apply spatial fil-
tering, i.e., Laplacian or common spatial pattern (CSP) filters, before calculating log band-power [Blankertz
et al., 2008b].
Similarly to the time domain, information theory and statistical methods can be applied in the frequency
domain. Spectral entropy calculated based on PSD [Zhang et al., 2015], and statistical methods applied to
spectral features [Samuel et al., 2017] have been used for MI EEG feature extraction.
3.2.3 Time-frequency methods
Temporal feature extraction methods do not focus on the spectral characteristics of the signal, which can
be important for the classifier. On the other hand, spectral methods can be ineffective without temporal
information. Time-frequency features, therefore, are suggested to overcome these limitations and integrate
both domains. They are especially powerful in BCIs since they can represent the change of spectral activity
in non-stationary EEG data.
21
The most common approaches are short-time Fourier transform (STFT) [Tabar and Halici, 2016, Tian
and Liu, 2019], continuous wavelet transform (CWT) [Borisoff et al., 2004, Ieracitano et al., 2020], and dis-
crete wavelet transform (DWT) [Guo et al., 2015, Lin and She, 2020]. STFT divides the trial into overlapping
segments and computes FFT on each of them separately. It results in a spectrogram that represents how
spectral characteristics change over time. Alternatively, time-frequency spectra can be generated with CWT,
which decomposes the signal into wavelets. STFT and CWT have also been used to extract features in form
of spectral images for deep learning classification [Mammone et al., 2020, Dai et al., 2019]. DWT is a time-
frequency decomposition technique that decomposes a signal into additive components according to their
time-frequency properties. It is a powerful method for MI EEG analysis, since different frequency bands of
EEG signals contain different information about motor imagery states.
3.2.4 Spatial methods
Unlike previously described approaches, spatial methods generate features using the information not from
a single channel, but a combination of channels. Widely used spatial feature extraction methods for BCI
systems include blind source separation (BSS) [Ortiz-Echeverri et al., 2019], common spatial pattern (CSP),
and spatio-spectral decomposition (SSD).
BSS methods assume that measured EEG signal x(t) is a linear combination of sources s(t), which in-
cludes clean EEG sources and some artifacts. Under this assumption, the EEG signal is represented as fol-
lows:
x(t)=As(t),
where Ais a mixing matrix. The goal of BSS methods is to find a matrix Wthat allows us to infer the sources:
s′(t)=Wx(t).
Such algorithms include independent component analysis (ICA) and principal component analysis (PCA).
As we mentioned in Section 3.1.3, these methods have been used not only for feature selection but also for
artifact removal.
Common spatial pattern. BSS methods are unsupervised, which means that class labels correspond-
ing to the EEG trials remain unknown for the algorithm. Unlike them, CSP is a spatial feature extraction
method that relies on labeled data. CSP aims to maximize the variance of the spatially filtered signal for one
class while minimizing it for the other class.
Mathematically, for the data of two classes with estimated covariance matrices C1and C2, CSP is learning
a matrix of spatial filters Wto maximize the objective function:
J(w)=w⊺C1w
w⊺C2w,
which can be optimized with the Lagrange multiplier method. However, CSP may suffer from overfitting and
poor generalization performance when applied to real-world EEG signals. To address this issue, different
regularization techniques have been proposed to improve the performance of CSP [Lotte and Guan, 2011].
In BCI studies, regularization strategies penalize the objective function J(w) [Blankertz et al., 2007, Williams,
2006, Lotte and Guan, 2011] or regularize the estimate of covariance matrices [Cheng et al., 2017, Kang et al.,
2009, Lotte and Guan, 2010]. For example, one of the options how to regularize an initial class covariance
matrix Cis: ˜
C=(1−γ)ˆ
C+γI,
ˆ
C=(1−β)sC+βG,
where Iis the identity matrix, γ,β∈[0,1] are regularization parameters, and sand Gare the scalar and a
generic matrix, respectively [Cheng et al., 2017]. The CSP objective function can be regularized as follows:
J(w)=w⊺C1w
w⊺C2w+αP(w),
where αis a user-defined positive constant, and P(w) is a penalty function, which enforces specific solutions
for filters.
CSP and its variations have been shown to be effective in EEG analysis [Blankertz et al., 2008a, Blankertz
et al., 2008b, Liao et al., 2007, Lemm et al., 2005, Tomioka and Müller, 2010] and are the most common
feature extraction methods in MI-based BCIs [Padfield et al., 2019].
22
Spatio-spectral decomposition. Another popular spatial feature extraction method is SSD [Nikulin
et al., 2011]. It maximizes the signal power in the particular predefined frequency bin of interest around
frequency fwhile minimizing it in the surrounding frequency bins around f−∆fand f+∆f. This maxi-
mizes [Nikulin et al., 2011] signal-to-noise ratio (SNR) under the assumption that the signal frequency lies in
the first bin, and the noise sources produce signals with a relatively broad frequency range around it (see Fig-
ure 19). Since different cognitive, perceptual, and motor processes generate EEG oscillations in particular
frequency bands [Buzsáki and Draguhn, 2004], the SSD is effective in detecting functional EEG components
including motor imagery activity.
Figure 19: SSD applied to real MI EEG data. SSD maximizes the variance of the signal in the central frequency
bin while simultaneously minimizing the variance of the noise in the left and right frequency bins (adapted
from [Nikulin et al., 2011]).
Mathematically, the measured EEG matrix Xcontains signal Sand noise N:
X=S+N.
The columns of Xare then filtered twice. Firstly, a band-pass filter around frequency fis applied to get
matrix Xsthat contains both signal and noise. Secondly, Xis band-pass filtered separately around f−∆f
and f+∆f, and the two results are summed up to obtain matrix Xnthat contains noise only. After that, the
corresponding covariance matrices Csand Cnare calculated resulting in the following objective function:
J(w)=w⊺Csw
w⊺Cnw,
which is similar to the one in CSP and can be optimized in the same manner. Besides the purpose of feature
extraction, SSD can be used for source separation, which we employ in our methods (Section 6.2).
3.2.5 Spatio-temporal methods
Spatio-temporal methods extract features in both time and space (channel) domains. In BCIs, common
spatio-temporal features are sample covariance matrices and features extracted using deep learning [Singh
et al., 2021].
Sample covariance matrix Ciof a single trial is calculated based on the corresponding (assumed to be
zero-mean) measured data matrix Xi:
Ci=XiXT
i
N−1,
where Nis the epoch duration expressed in the number of samples. Such features are commonly used
in Riemannian geometry classification [Barachant et al., 2013]. We describe this class of classifiers in Sec-
tion 4.2.
Other spatio-temporal features are usually extracted with deep neural networks, such as representative
models described in Section 4.3.2.
23
3.2.6 Feature selection and dimensionality reduction
As we described, a big diversity of feature extraction methods is available for EEG analysis, which potentially
leads to the classification problem with high-dimensional feature vectors. The main reasons to reduce the
number of input variables for the classifier are improving classification accuracy, decreasing training and
prediction time, reducing overfitting, and improving the interpretability of the model [Tangermann, 2007].
Feature selection and dimensionality reduction methods reduce the quantity of the features, but they op-
erate in different ways [Torres et al., 2020]. Dimensionality reduction combines features to reduce their
number, while feature selection excludes some according to particular criteria. In this section, we provide
examples of feature selection and dimensionality reduction approaches including statistical, filter bank, and
evolutionary techniques.
Widely used statistical dimensionality reduction techniques for MI EEG data are PCA [Yu et al., 2014]
and ICA [Guo et al., 2013]. PCA aims to find orthogonal components that explain the maximum variance
in the data, while ICA aims to identify statistically independent components. They have also been utilized
with other signal processing methods for better classification results [Wang et al., 2014, Yu et al., 2014]. In
addition, as we already mentioned earlier, these methods can be utilized for artifact removal and feature
extraction.
The filter bank approach for feature selection is a technique that involves decomposing the EEG sig-
nal into multiple sub-bands using a set of band-pass filters. The goal of this approach is to capture the
frequency-specific characteristics of the EEG signal, as different brain activities are associated with different
frequency bands. After decomposing the EEG signal into sub-bands, features can be extracted from each of
them. In the case of motor imagery EEG, the filter bank approach is typically utilized in systems that use
CSP for feature extraction [Padfield et al., 2019]. In filter bank CSP (FBCSP), CSP is applied to each sub-band
separately. The resulting features can then be selected based on the mutual-information criteria [Ang et al.,
2008]. In one study [Chatterjee and Sanyal, 2020], the performance of different variants of the filter bank
approach was compared on an MI EEG dataset, concluding that traditional filter bank ranges from 4 to 24
Hz with a 4 Hz frequency band separation provides the most discriminating features.
Another class of feature selection approaches, evolutionary methods, have been shown to be effective in
finding optimal solutions in large feature spaces [Baig et al., 2017]. Typically, during each iteration, an evolu-
tionary algorithm randomly generates a set of candidate feature subsets, evaluates their fitness, and selects
the best performing subsets to generate the next generation of candidates. The process continues until a
stopping criterion is met, such as a maximum number of iterations or a convergence threshold. The fitness
of the subset is commonly estimated by classification accuracy. Evolutionary feature selection methods for
EEG [Baig et al., 2017, Atyabi et al., 2012, Saibene and Gasparini, 2023] apply genetic algorithm [Yang and
Honavar, 1998], particle swarm optimization [Kennedy, 2010], and other evolutionary optimization meth-
ods [Padfield et al., 2019].
Multiple studies [Singh et al., 2021, Al-Nafjan, 2022, Torres et al., 2020, Al-Saegh et al., 2021] reviewed
feature selection and dimensionality reduction methods for BCIs and provide further information on the
topic.
3.3 EEG source localization
EEG measures brain activity through electrodes placed on the scalp. Due to the volume-conduction averag-
ing of the signal sources [Brette and Destexhe, 2012] and a relatively small number of sensors, the recorded
signals have low spatial resolution compared to other measurement techniques (see Section 2.2). Source lo-
calization methods aim at reconstructing the sources of the recorded activity and identifying their locations.
Source localization is widely used in medical applications for monitoring and diagnosis of multiple dis-
eases, including epilepsy and various psychiatric and neurological disorders [Michel and Murray, 2012]. In
MI EEG analysis, source localization can help to identify the sources that are most relevant to the motor task,
which can improve the accuracy of classification [Edelman et al., 2016, Asadzadeh et al., 2020]. In addition,
sources corresponding to artifacts can be identified and removed from EEG [Hallez et al., 2006, Harmening
et al., 2022].
Assuming the linear model of EEG, the process of source localization generally contains the following
steps: (i) decomposition of the mixed EEG signals into individual source components, (ii) choice or design
of a head model, and (iii) localization of the found source components using the head model. In this section,
we go through these steps and briefly describe commonly used methods.
24
3.3.1 Linear model of EEG
The general linear model of EEG [Parra et al., 2005] assumes that measured EEG signals x(t) are a linear
superposition of the source signals s(t):
x(t)=As(t)+n(t),
where Ais the propagation matrix of the forward model and n(t) represents the noise, often assumed to
be Gaussian with zero mean. Each column of Ais a spatial pattern that defines how the corresponding
source signal is projected to the EEG channels. The counterpart is the backward model, which describes the
decomposition of EEG signals with spatial filters:
s′(t)=W⊤x(t),
where each column of W(resp. each row of W⊤) is a spatial filter that extracts one component from the
EEG. As mentioned in Section 3.2.4, a matrix of spatial filters W(backward model) for data x(t) can be ob-
tained, for example, with blind source separation methods such as ICA or PCA. Given a matrix W, one can
calculate [Haufe et al., 2014] the corresponding matrix of spatial patterns (forward model):
A=CXW(W⊤CXW)−1,
where CXis the covariance matrix of the data. For invertible W, this implies A=(W⊤)−1.
Based on the extracted components, source localization methods aim at identifying the locations of the
underlying sources. That generally requires a head model that describes how the electrical field propagates
from the sources to the electrodes on the scalp, providing a solution for the forward problem.
3.3.2 Head modeling
Head modeling aims at finding the EEG forward solution that determines how each electrical source in the
brain contributes to the signal measured by each EEG electrode on the scalp (see Figure 20). The elec-
trical activity generated by synchronized postsynaptic potentials (see Section 2.1) does not propagate ho-
mogeneously through the brain. The brain, cerebrospinal fluid, skull, and scalp have different geometries
and conductivities and impact the current propagation to a different extent. Knowing these characteristics,
Poisson’s equation can be used to calculate the potentials at each electrode [Malmivuo and Plonsey, 1995]
generated by sources, which are typically modeled as equivalent current dipoles [Michel and He, 2019]. In
head modeling, in the forward problem, a different notation for the propagation matrix Ais commonly used:
X=LS +N,
where matrix Lis called the leadfield.
Various models are used to solve the forward problem. Simple head volume conductor models such
as a single-layer sphere or multi-shell sphere allow simple and fast computations, but are not as accurate
as realistic head models. While being computationally more complex, realistic head models including the
boundary element method (BEM) and finite element method (FEM) better represent the head shape.
Simple head models Historically, the first head models assumed a spherical shape of the head and uni-
form conductivity throughout the brain, which allows for an analytical solution of the forward problem [Yao,
2000]. However, this oversimplified modeling leads to limited source localization accuracy [Michel et al.,
2004].
A slightly more realistic multi-shell sphere represents the head as a set of three or four nested concentric
spheres, typically representing different layers such as brain, cerebrospinal fluid, skull, and scalp. These
layers have different conductive properties (recent studies have shown that the conductivity ratio between
the skull and brain is about 20:1 [Lai et al., 2005]); therefore, such models attribute different conductivities to
each sphere. Assuming homogeneous and constant properties of each layer, the forward solution can also
be calculated analytically and increases the accuracy to a certain degree.
Realistic head models For a more accurate volume conductor model, more realistic parameters of
conductivity and geometry have to be considered. Ideally, the precise individual head shape and geome-
try of corresponding tissues are obtained by MRI. In cases when performing an MRI scan is not possible
25
Figure 20: Forward and inverse EEG problem. The forward problem aims at predicting the electrical poten-
tials at EEG channels that would be generated by given neural activity sources, while the inverse problem
refers to estimating the distribution of sources from the measured scalp potentials.
or not practical, a generic (averaged) head geometry can be used. Commonly utilized realistic head mod-
els include the boundary element method (BEM) [Haemaelaeinen and Sarvas, 1989] and the finite element
method (FEM).
The BEM assumes electrically homogeneous and isotropic brain tissue with different conductivity val-
ues and accounts for anatomic information of the head. The tissue surfaces are modeled by closed tri-
angle meshes with a limited number of nodes. The accuracy of the BEM is roughly proportional to the
number of nodes. However, the computation time is also growing linearly with the number of nodes and
electrodes [Fuchs et al., 2007]. While accurate enough and relatively computationally simple, the BEM is
a compromise between oversimplified spherical models and very complex methods that require detailed
information about the brain tissue and heavy computations.
The FEM provides an even more realistic and accurate head model. In contrast to the BEM, it considers
inhomogeneous and anisotropic conductivities of brain tissue [Zhang et al., 2006, Lee et al., 2009]. In ad-
dition, the FEM meshes the whole volume of the brain, which makes it a volume-based method, contrary
to the surface-based BEM. However, the method requires an accurate segmentation of MRI. Moreover, indi-
vidual conductivity and anisotropy parameters are rarely available [Hu and Zhang, 2019, Michel and Brunet,
2019]. Overall, the FEM generates an accurate head model, but requires detailed information about the
brain and is computationally expensive.
3.3.3 Source localization
After a head model is built and a leadfield is constructed, the final step is solving the inverse problem, i.e.,
determining the neural activity sources given the measured EEG. In contrast to the deterministic forward
problem, the inverse problem is under-determined, since the number of current sources is much larger than
the number of sensors. Therefore, additional constraints and regularization based on a priori information
about the sources are needed to solve the inverse problem.
The earliest approaches to solving the inverse problem assume a limited number of equivalent dipoles [Ka-
vanagh et al., 1978]. The a priori assumption, in this case, is that only one or a few active brain areas generate
the scalp potentials. More recent distributed source models do not require a priory information about the
number of sources. Instead, many equivalent dipoles distributed in a 3-D grid or 2-D surface are considered
as sources, and the strength of each of these dipoles is estimated. We briefly discuss commonly used source
localization methods in this section.
Minimum norm (MN) methods. Currently, the most widely used source localization methods are based
on the minimum norm approach [Hämäläinen and Ilmoniemi, 1994], which was one of the earliest attempts
to solve the inverse problem for EEG and MEG data. MN assumes the minimum overall intensity, i.e., the
26
smallest L2-norm, of the solution S. The solution has the simplest expression:
S=L⊤(LL⊤)†X,
where †denotes conjugate transpose. The MN algorithm is biased towards weak superficial solutions since
they can generate stronger fields with less energy due to proximity to the electrodes. Various weighting ap-
proaches have been proposed to decrease this bias. For instance, the weighted MN (WMN) algorithm [Wang
et al., 1992] employs the weighting based on the norm of the columns of the leadfield matrix L. The low
resolution electromagnetic tomography (LORETA) [Pascual-Marqui et al., 1994] is a variation of WMN that
minimizes the norm of the second-order spatial derivative of the current source distribution for spatially
coherent and smooth solutions. Standardized LORETA (sLORETA) incorporates additional assumptions re-
garding the smoothing and weighting of the values, which leads to higher spatial accuracy [Pascual-Marqui,
2002]. MN family of methods is one of the most successful approaches for EEG source localization. The
main disadvantage of MN methods, however, is overly smooth and widespread solutions.
Beamforming and scanning methods. Beamforming methods originated from radar and sonar signal
processing, but have been later shown to be effective also in EEG and MEG source localization [Van Veen
et al., 1997, Sekihara et al., 2001]. The inverse problem solution is chosen according to the constraints that
enhance the contribution of the neural activity from the location of interest and suppress the contributions
from other locations. The beamformer and its variations [Sekihara et al., 2001, Robinson, 1999] demon-
strate high accuracy results compared to other source analysis methods [Dalal et al., 2008]. However, the
performance can degrade in the case of correlated sources [Robinson, 1999] or inaccuracies of the forward
model.
Another approach is to use a scanning strategy. All candidates for source locations are scanned and,
assuming a dipole source at each location, the forward solution is calculated based on the leadfield. The
forward solution is then used to quantify how likely it is for the candidate dipole to be in the signal subspace
as opposed to the noise subspace. Such a method is implemented in the multiple signal classification (MU-
SIC) algorithm [Mosher et al., 1992]. The accuracy of the MUSIC algorithm, however, can be poor in the case
of correlated sources [Mosher and Leahy, 1999]. This limitation is partially resolved in several variations
of MUSIC, including the recursively applied and projected MUSIC (RAP-MUSIC) [Mosher and Leahy, 1999]
and the first principle vector (FINE) localization method [Xu et al., 2004].
Other source localization methods. Many more source localization methods [He et al., 2018, Hu and
Zhang, 2019] were proposed, including Bayesian methods [Friston et al., 2008, Trujillo-Barreto et al., 2004]
that maximize the posterior distribution of sources given measurements while assuming a prior proba-
bilistic distribution of the sources; and methods for sparsity solutions [Gorodnitsky et al., 1995, Matsuura
and Okabe, 1995, Ding and He, 2008] that consider that the main neural electric activity is sparsely local-
ized [Gorodnitsky et al., 1995].
3.4 Chapter summary
In this chapter, we reviewed methods for preparing EEG data for classification by performing preprocessing,
artifact removal, feature extraction and selection. Additionally, the chapter discusses source localization ap-
proaches based on head modeling. These techniques are important for accurately identifying and analyzing
neural activity recorded by EEG.
In order to remove multiple physiological and non-physiological artifacts and prepare the data for classi-
fication, the data has to be preprocessed. The preprocessing includes artifact removal, filtering, segmenting
the data into epochs, baseline correction, downsampling, and other techniques.
Feature extraction and selection are important steps in EEG analysis, as they aim to identify informative
features that capture essential characteristics of the underlying neural activity. In MI EEG studies, various
temporal, spectral, time-frequency, spatial, and spatio-temporal features have been extracted for classifica-
tion. The CSP algorithm is the most commonly used feature extraction method. Statistical, filter bank, and
evolutionary methods are then applied on top of the extracted features for feature selection and dimension-
ality reduction.
Source localization has also been used to remove the artifact-related EEG components and select rele-
vant sources for classification. Typically, it is conducted in several steps. Firstly, the measured data is decom-
posed into individual components, which often involves blind source separation techniques such as PCA or
27
ICA. Then, a forward problem is solved based on head modeling, resulting in the leadfield that estimates
the scalp-recorded EEG signal that would be expected from a particular neural activity of the brain sources.
Finally, the leadfield is used to identify the location, strength, and orientation of the sources corresponding
to EEG components.
28
4 EEG Classification for BCIs
A lot of recently developed models have been introduced and applied to MI-EEG classification over the last
two decades. The BCI field follows the general trends of machine learning, but also adapts them for the
specific case of brain signal analysis. In this chapter, we discuss these trends together with major challenges
of EEG classification. In addition, we describe the most common machine learning methods used in MI-
based BCIs and discuss their advantages and disadvantages.
The goal of a classification task is to identify to which class yfrom a given set of classes Yan object x∈X
belongs. In MI-based BCIs, object xcan be, for example, a feature vector extracted from a measured EEG
trial, a spectral image, a topological map, or even a vector/matrix of raw signal values. Set Ycontains motor
imagery events such as left/right hand or foot movements. Given a training set of labeled pairs (x,y), a clas-
sifier learns a mapping fw:X→Yby optimizing a cost function J(y,ˆ
y) on an independent validation set
over parameters w, where yis the true and ˆ
yis the predicted class label. In EEG-based BCIs, the most used
metric is classification accuracy, i.e., the percentage of objects that were correctly labeled [Altaheri et al.,
2021]. Offering different forms of mappings fwand approaches how to learn parameters wfrom available
training data, a variety of methods solve the EEG classification problem.
Although a considerable improvement has been achieved in MI-EEG classification, the task remains
challenging due to many reasons. Because of multiple biological, environmental, and electronic equip-
ment artifacts, EEG signals have a low signal-to-noise ratio [Goldenholz et al., 2008]. The signals are also
non-stationary over time, within and between subjects [Saha and Baumert, 2020]. This makes the classifi-
cation more difficult and requires a calibration process that takes about thirty minutes before each session
and needs the assistance of an expert. Another problem is a generally small amount of training data, since
it is expansive and time-consuming to collect them. Moreover, feature extraction from the measured data is
often based on hand-engineered techniques and relies on human expertise in the particular area. All these
difficulties result in overall low reliability and performance of BCIs.
Over the last 15 years, most of the newly proposed algorithms were addressing one or more of these
challenges [Lotte et al., 2018]. Some methods were improved by utilizing an idea of adaptation [Lotte et al.,
2015], where classifiers update their parameters with time to track the changes in signal distribution and deal
with EEG non-stationarity. Other approaches generate artificial signals to increase the size and variability of
training datasets and improve classification [He et al., 2021]. One of the most promising research directions
is developing a so-called end-to-end classifier, which can automatically preprocess raw EEG data, extract
features, and perform classification.
In this chapter, we describe the methods that are used in MI-based BCIs starting with traditional ap-
proaches, which still demonstrate competitive performance, and following with two groups of models, ge-
ometry classifiers and deep learning methods, that were applied to MI-EEG data later, yet both reached a
state-of-the-art level of classification. We employ several of these classification models later (see Section 6.2)
for testing the performance of our data augmentation methods. In addition, deep generative models, which
are also described in this chapter, play a central role in EEG data augmentation and are further discussed in
Section 5.2.
4.1 Traditional machine learning approaches
Conventional machine learning models have been extensively used for classifying MI EEG. These models are
favored due to their simplicity, ease of understanding, ability to learn from limited data, and generally strong
performance. As a result, they are often the primary choice for MI-EEG classification. In this context, we
primarily focus on two widely used classification methods: linear discriminant analysis (LDA) and support
vector machine (SVM). These methods are still heavily utilized and can be challenging to outperform with
newer approaches. Additionally, we briefly describe several other traditional techniques that may not be as
popular but have demonstrated promising results on specific datasets.
4.1.1 Linear discriminant analysis
LDA uses a hyperplane to separate the data of different classes. After the model is trained, a new feature
vector is classified depending on which side of the hyperplane it is located (see Figure 21).
LDA assumes that the data from different classes are derived from a Gaussian distribution with equal
covariance matrices. In the case of two classes, the feature vectors have distributions N(µ1,C) and N(µ2,C),
where µ1and µ2are corresponding mean vectors and Cis the common covariance matrix. The optimal
29
Figure 21: The LDA hyperplane separates the data representing two different classes.
LDA hyperplane is found by maximizing the distance between the means of two classes and minimizing the
within-class variability [Fukunaga, 1990]. The resulting hyperplane is defined by the normal vector wand
bias b:
w=C−1(µ2−µ1), b=w⊺(µ1+µ2)/2.
For a new feature vector x, a class label is assigned according to the rule:
(cl ass1 if w⊺x−b<0,
cl ass2 if w⊺x−b≥0.
The covariance matrix Cis estimated empirically using the feature vectors from the training dataset.
However, this estimation tends to be error-prone when the dataset is relatively small. A regularized LDA was
introduced to deal with this problem. For example, in shrinkage LDA, the regularized covariance ˆ
Cis shrunk
towards identity matrix:
ˆ
C=(1−λ)C+λνI,
with λ∈[0,1] and νbeing regularization and scaling parameters.
In BCIs, the estimated distribution parameters can also change with time due to non-stationarity of
EEG signals. To address this problem, several adaptive strategies for LDA were proposed [Schlögl et al.,
2010, Vidaurre et al., 2011a, Vidaurre et al., 2011c, Vidaurre et al., 2011b], including one based on Kalman
Filtering to track the distribution of each class [Hsu, 2011].
LDA is a simple and fast classifier, which demonstrates good results on MI-EEG data. It has been success-
fully used in numerous BCI applications. The regularized LDA is also effective on small datasets. However,
LDA depends on the preprocessing and feature extraction steps, and because of its linearity, can also per-
form poorly on complex nonlinear EEG data.
4.1.2 Support vector machine
SVM also finds a hyperplane that separates two classes. From both classes, it takes the closest to the candi-
date hyperplane feature vectors, which are called support vectors, and calculates the distance between these
vectors and the hyperplane, the margin. The algorithm founds the optimal hyperplane that maximizes this
margin (see Figure 22).
In the case of linear non-separability, it is impossible to find such an optimal hyperplane. Regulariza-
tion allows errors on the training set and makes the algorithm less sensitive to outliers. Moreover, SVM
can also learn nonlinear separation boundaries using the “kernel trick”. The idea is to map the data into a
high-dimensional space, where ideally it is linearly separable. The “kernel trick” allows calculating the dot
30
Figure 22: The optimal SVM hyperplane is found by maximizing the margin.
products of the transformed feature vectors and optimizing the higher dimensional decision boundary with-
out actually performing the transformation. In BCI applications, the Gaussian kernel with a parameter σis
usually used:
K(x,y)=exp(−∥x−y∥2
2σ2).
The algorithm utilizing this kernel, commonly referred to as Gaussian SVM, has exhibited noteworthy per-
formance in the context of BCIs [Kaper et al., 2004].
SVM generalizes well, is effective in high-dimensional spaces, and, when used together with a kernel
function, can learn nonlinear boundaries. It often outperforms other classifiers on MI-EEG datasets [Lotte
et al., 2007] and has several adaptive variants [Li and Zhang, 2010, Woehrle et al., 2015] to deal with the
non-stationarity of signals. However, the required training time is relatively high, and the regularization
and kernel parameters have to be predefined or estimated from training data in an additional optimization
process.
4.1.3 Other traditional classifiers
LDA and SVM have been the most popular traditional classifiers for BCI applications, yet not the only ones in
use. Among other used models, there are also k-nearest neighbors (kNN), quadratic Bayesian, and decision
tree (DT) classifiers.
For a target feature vector, KNN calculates the distances to all the feature vectors from the training
dataset and assigns the class label that is the most common among the closest kneighbors. While being one
of the simplest classification algorithms, with a sufficient amount of data, KNN can approximate complex
nonlinear functions. The algorithm can be effective in BCIs with low-dimensional feature vectors [Borisoff
et al., 2004] while showing rather poor results otherwise. This, together with high outlier sensitivity, difficulty
with finding an optimal parameter k, and slow performance in the case of bigger datasets, makes KNN an
unpopular choice for BCI applications.
Bayesian classifiers employ Bayes rule to compute the posterior probability, which determines the prob-
ability of a feature vector to belong to a specific class. Subsequently, the maximum a posteriori (MAP) deci-
sion rule is applied to estimate the class. Quadratic Bayes assumes a Gaussian distribution for the features
within each class, resulting in a quadratic decision boundary. While this classifier is characterized by its sim-
plicity and efficiency, it assumes conditional independence among the features, a condition that is typically
violated in the context of Brain-Computer Interfaces. Consequently, except for a few rare exceptions [Sol-
hjoo and Moradi, 2004, Lemm et al., 2004], other methods often outperform this classifier.
DT is the classifier based on simple decision rules, which are learned from the training data. Starting
from the root node of the tree, the feature vector is checked for the decision rule. Depending on the result,
it follows the corresponding brunch and is checked for the next rule, ending up at the terminal node with a
class label. To avoid overfitting, a technique known as bagging is used to create a so-called random forest,
an ensemble of DTs. The dataset is randomly divided into subsets, which are used to create separate DTs.
31
Later, the class is assigned using voting or averaging. Even though the algorithm is not that widely used, it
was reported [Lemm et al., 2004] to outperform regularized LDA on some MI-EEG datasets.
4.2 Riemannian geometry classifiers
Riemannian geometry classifiers have emerged as a recent development in EEG classification. These clas-
sifiers have demonstrated superior performance compared to other cutting-edge methods in various BCI
competitions [Congedo et al., 2017]. Moreover, they offer the advantage of not requiring spatial filtering or
feature extraction steps, making Riemannian algorithms an active area of research in motor imagery EEG
classification. In this section, we provide a general overview of the methodology, following [Lotte et al.,
2018]. For more in-depth information, readers can refer to comprehensive reviews on Riemannian methods
in BCIs [Yger et al., 2017, Congedo et al., 2017].
Riemannian geometry studies Riemannian manifolds. These manifolds are smooth curved spaces that
can be locally, i.e., at each point, approximated with a linear tangent space which is equipped with an inner
product. The distance on the Riemannian manifold is defined as the length of a curve that is the shortest
path between two points on the manifold. This shortest curve is called a geodesic (see Figure 23).
In the case of BCI applications, geometry classifiers typically take sample covariance matrices estimated
from EEG trials as input data. With proper regularization, they lie in the space of symmetric positive def-
inite (SPD) matrices [Bhatia, 2007], which forms a Riemannian manifold. The EEG trials, which, thus, are
the points on the Riemannian manifold, can be classified in two ways. On-manifold learning, e.g., Rieman-
nian minimum distance to mean (RMDM) or Riemannian support vector machine (RSVM), can be directly
applied by calculating the distances and kernels on the manifold. In off-manifold learning, the points are
projected onto the tangent space, where a standard classifier, such as LDA or SVM, can be employed.
Figure 23: The tangent space to the Riemannian manifold in point Gis drawn. The distance between
points Gand C2is the length of the corresponding geodesic.
4.2.1 On-manifold learning
As we already mentioned, the Riemannian distance between two points is the length of the corresponding
geodesic. For two SPD matrices C1and C2, the square of the distance can be calculated by:
δ2(C1,C2)=X
n
l og 2λn(C−1
1C2),
where λn(M) is the n-th eigenvalue of the matrix M. The distance between points C1and C2is non-negative,
symmetric, and equals zero if and only if C1=C2. The geometric mean Gof KSPD matrices {C1,...,CK} is
the unique solution of the corresponding optimization problem:
argmin
GX
k
δ2(Ck,G).
The problem does not have a closed-form solution, yet has a fast and robust iterative algorithm for comput-
ing G[Barachant et al., 2013].
On-manifold learning involves calculating the distances and kernels directly on the manifold. For exam-
ple, in the case of binary classification, the RMDM classifier calculates the geometric means C1and C2of
32
both classes on the training dataset. Subsequently, the new EEG trial is assigned with the label correspond-
ing to the closest geometric mean.
RMDM is simpler than traditional machine learning methods, does not need spatial filtering, feature ex-
traction, and parameter tuning, and has good generalization capability. Among the disadvantages, there are
probable numerical problems with computing the Riemannian distance between two SPD matrices and less
accurate distance values in the case of larger dimensions. In addition, computational complexities of geo-
metric mean and Riemannian distance are growing cubically with the number of EEG electrodes. However,
as long as the number of electrodes is not large, the RMDM classifier shows competitive results to previous
state-of-the-art methods [Lotte et al., 2018].
RMDM is just one of the techniques utilized on Riemannian manifolds. Another approach known as the
Riemannian support vector machine (RSVM) introduces a different method, where the SVM kernel function
is formulated using the geodesics on Riemannian manifolds [Yun et al., 2013].
4.2.2 Off-manifold learning
In order to benefit from both, Riemannian geometry and traditional classification methods, such as LDA
or SVM, it was proposed [Barachant et al., 2010, Barachant et al., 2012] to project the SPD matrices onto
the tangent space. In the tangent space, the points can be vectorized and classified with different methods,
which could not use an original SPD matrix on the Riemannian manifold as an input. First, the geometric
mean Gof the whole set of SPD matrices is computed. Thereafter, each SPD matrix Ckis projected onto the
tangent space, located at the point G, and vectorized:
ck=upper (G−1
2logG(Ck)G−1
2),
where upper operator keeps the upper triangular part of a matrix and vectorizes it by multiplying each di-
agonal element by 1 and non-diagonal by p2. In order to decrease the dimensionality of the vector ckwith
dimension n(n+1)/2, which can be bigger than the number of training trials in some BCIs, a variable se-
lection is applied. Finally, the obtained vectors are fed to a classification algorithm (usually LDA or SVM).
This approach is reported to significantly outperform the classical CSP+LDA combination [Barachant et al.,
2012].
Like the RMDM classifier, the methods based on the projection onto tangential space do not require
spatial filtering. While these methods perform better than RMDM in terms of accuracy, they have increased
algorithmic complexity and additionally inherit disadvantages of the used classifier.
4.3 Deep learning
Recently, deep learning (DL) methods have gained immense popularity as a valuable tool for EEG classi-
fication. Deep neural networks offer the advantage of establishing an efficient end-to-end classification
framework by performing data preprocessing and feature extraction automatically. This distinguishes them
from traditional classifiers, where such steps, particularly for MI-EEG data, can be challenging and time-
consuming. Although constrained by the limited size of EEG datasets, DL methods have already surpassed
other advanced classification techniques on numerous datasets [Altaheri et al., 2021].
In this section, we describe the most common DL models used in BCI research, combined into three
groups. Discriminative models are typically used for classification, representative models for learning low-
dimensional representations and feature extraction, and generative models for data generation and aug-
mentation of training datasets.
4.3.1 Discriminative models
The main goal of discriminative DL models is data classification. By employing non-linear transformations,
they can extract useful features from signals, learn non-linear decision boundaries, and assign to a new
sample a class (or probability to belong to a particular class) from a pre-defined set. Discriminative modes
that are widely used in MI-EEG classification include convolutional neural networks (CNN), recurrent neural
networks (RNN) and hybrid neural networks.
Multi-layer perceptron. MLP is, perhaps, the simplest neural network architecture. Even though later
it was replaced by more complex and effective deep learning models [Al-Saegh et al., 2021], only 15 years
ago MLP was still the most popular neural network used for BCIs [Lotte et al., 2007], and was applied to
33
many EEG classification problems [Haselsteiner and Pfurtscheller, 2000, Palaniappan, 2005, Balakrishnan
and Puthusserypady, 2005]. Therefore, it is valuable to provide a description of it, not only due to its histori-
cal significance but also because it offers perspectives to understand the main neural network principles.
The basic unit of MLP is a so-called perceptron or artificial neuron, which takes a feature vector xas
an input and calculates the output ˆ
y(see Figure 24). The input features are summed with corresponding
weights w, and then a non-linear activation function gis applied to the result:
ˆ
y=g(
i
wixi+b),
where bis a bias.
Figure 24: Perceptron predicts the output ˆ
yby applying an activation function gto the weighted sum of
input features.
There are a lot of activation functions (see Figure 25) that one can use in a neural network architecture.
Due to non-vanishing gradients and good empirical results, ReLu and GELU are the most popular choices
for hidden layers, while the softmax function is often used for the output layer.
(a) Sigmoid: g(x) =1
1+e−x(b) Tanh: g(x) =t anh(x)(c) Softmax: g(xi)=exi
jxj
(d) ReLu: g(x) =max(0, x) (e) lReLu: g(x) =max(αx,x) (f) GELU: g(x) =xP (X<x)
Figure 25: Examples of non-linear activation functions. ReLu and GELU are the most popular activation
functions used in the hidden layers. In subfigure (f), X∼N(0,1), P(X<x)=Φ(x) is the standard Gaussian
cumulative distribution function.
Perceptron alone can be already used for classification. However, even though it could represent some
non-linearities thanks to activation functions, some other non-linear functions, such as XOR, cannot be
represented. To tackle this problem, MLP was introduced. In addition to input and output layers, MLP also
has at least one hidden layer where many neurons are stacked. The output values of each neuron in the
hidden layer serve as an input for the next hidden layer or, in the end, for the output layer. MLP consists of
fully connected layers, i.e., every neuron from the previous layer is connected to every neuron from the next
layer (see Figure 26).
34
Figure 26: Example of an MLP architecture, with 5 input neurons, two hidden layers with 8 neurons in each,
and one output neuron.
During training, the weights of the network are optimized in order to minimize a predefined cost func-
tion J(ˆ
y,y), where ydenotes the correct label, using so-called backpropagation [Rumelhart et al., 1986]. In
each iteration, the values are forwarded through the network, and the output value ˆ
ytogether with the cost
function J(ˆ
y,y) are computed. Subsequently, all the gradients for the weights are calculated using a chain
rule, in which the gradients on layer ldepend on the gradients on layer l+1. Therefore, the process of the
calculation of the gradients is going backward from the output to the input layer and is called backpropa-
gation. After all gradients are computed and the weights are updated, the algorithm proceeds with the next
iteration until convergence.
MLP is a universal approximator, meaning it can approximate any continuous function given enough
neurons and layers. However, for the classification of noisy and non-stationary EEG data, a careful selection
of the neurons and layers number, as well as regularization, is needed to avoid overfitting. MLP is not used
that often for MI-EEG classification anymore, mainly because it shows poorer results than more complex
neural network architectures.
Convolutional neural network. CNNs are by far the most used neural networks for MI-EEG classifica-
tion. As standalone models or, less often, in a combination with other deep learning methods, CNNs are
currently used in 76% of all deep learning MI BCI studies [Altaheri et al., 2021].
CNN is a feedforward neural network with at least one convolutional layer [Fukushima, 1980]. Typically,
it also contains pooling and fully connected layers. The convolutional layer is the basic unit of a CNN. Let
us assume a 2Dinput X, which, in the case of MI-EEG, can be a timesteps ×channels matrix or a spectral
image. A 2Ddiscrete convolution HN+K−1,M+L−1of the input XN,Mand a filter WK,L, which is also called a
kernel, will be:
Hi,j=X
nX
m
Xn,mWi−n+1,j−m+1,
where nand mrange over all legal subscripts of Xn,mand Wi−n+1,j−m+1. Several filters are usually applied in
parallel to extract different features. It is also common to design specific kernels to separate EEG temporal
and spatial feature extraction [Schirrmeister et al., 2017]. After the convolution, a nonlinearity and a so-
called pooling are usually applied. The pooling layer aggregates the local values by average or max operators.
Convolution, nonlinearity, and pooling together form a building block of CNN, and can be repeated, usually
with modifications, several times (see Figure 27).
CNNs were shown to outperform other deep learning architectures [Tayeb et al., 2019, Wang et al., 2018b]
and became one of the state-of-the-art methods in MI-EEG classification. It happened, among other rea-
sons, due to their ability to take input data in different formats, such as 2D raw EEG data, spectral images, or
topographic maps, and extract useful features. The small size of most MI-EEG datasets is a strong limiting
factor, that does not allow employing all the potential of CNNs. Therefore, shallow network modifications,
35
Figure 27: A typical convolutional neural network with two building blocks of convolution and pooling, that
are followed by fully connected layers to produce the output ˆ
y.
that have fewer parameters to learn, often show better results [Schirrmeister et al., 2017] and CNNs with
only 2 or 3 convolutional layers are currently used the most [Altaheri et al., 2021].
Recurrent neural network. Even though RNNs are not used in BCIs as often as CNNs, they have also
been successfully applied for MI-EEG classification. Their specific feature is the ability to use the informa-
tion not only from the particular input value but also from the context surrounding it. This is achieved by
introducing a recurrent internal state and enables these types of networks to deal with time-series signals
analysis.
The simplest form of RNN contains a loop that transmits the information from previous values and uses
them to process the following (see Figure 28). Let us consider a simple version of RNN where context cfis
just the hidden state from the previous unit ht−1. In this case, the current hidden state is determined by
the current input xtand the previous hidden state; the output otis computed only from the current hidden
state:
ht=f(ht−1,xt,θ),
ot=f(ht,θ).
This allows taking into account already seen values, although in practice, RNNs are not able to extract useful
information if the gap between a current and a relevant unit in the past is relatively big [Bengio et al., 1994].
Figure 28: Unfolded RNN loop. Outputs on top depend on hidden units in the middle. The hidden units are
determined by both, inputs and a one-way context loop.
Special RNN architectures that can handle this problem are long short-term memory (LSTM) [Hochreiter
and Schmidhuber, 1997] and gated recurrent unit (GRU) [Cho et al., 2014] networks. They are capable of
learning long-term dependencies, i.e., to compute the output considering what happened relatively far in
the past. We will focus on LSTM here since it is applied to MI-EEG data much more often than GRU [Altaheri
et al., 2021].
LSTM units use the following five elements. The input gate decides what information to store in the cell
state:
it=σlog (Wixt+Uiht−1+bi).
36
The forget gate determines the parts of the context that have to be forgotten:
ft=σlog (Wfxt+Ufht−1+bf).
The output gate indicates how relevant the input and the cell state are to the output:
ot=σlog (Woxt+Uoht−1+bo).
The cell state stores the relevant context information. It is not changed much and can contain important
information for a large number of time steps:
ct=f◦ct−1+it◦t anh(Wcxt+Ucht−1+bc).
Finally, the output of LSTM is computed in the hidden state:
ht=ot◦t anh(ct).
W,U, and bare parameters to learn, operator ◦is the element-wise product, and σl og represents logistic
sigmoid functions.
Even though there are successful RNN applications for MI-EEG data classification, they remain sec-
ondary after CNNs because of the bigger time and memory consumption, as well as generally poorer per-
formance [Altaheri et al., 2021].
Hybrid neural networks. Hybrid deep learning models combine two or more neural networks into
one framework. A lot of such approaches for MI-EEG classification propose to combine two discriminative
networks, in particular, CNN with LSTM [Zhu et al., 2019, Zhang et al., 2021b, Freer and Yang, 2020], but also
CNN with GRU [Li et al., 2020] and CNN with MLP [Amin et al., 2019]. Discriminative NNs are also being
used in combinations with networks from two other groups, the representative [Tabar and Halici, 2016] and
generative [Zhang et al., 2020], which we will describe in the following sections. Moreover, CNN and LSTM
were applied together with non-DL methods, mainly with SVM [Taheri et al., 2020, Kumar et al., 2021].
4.3.2 Representative models
Representative DL models learn useful low-dimensional representations of the input data. In BCIs, it is usu-
ally used for feature extraction. Unlike discriminative models, they operate in an unsupervised manner, i.e.,
learn from unlabeled data. The extracted features can later be used for different tasks, such as clustering or
classification. In BCIs, commonly used representative DL models are autoencoders (AEs), restricted Boltz-
mann machines (RBMs), and deep belief networks (DBNs). DBNs consist of a series of AE or RBM networks
stacked together and have been extensively applied in BCI research [Hassanpour et al., 2019, Chu et al.,
2018, Lu et al., 2017, Xu et al., 2020].
Autoencoder. Autoencoder is a neural network, which aims at learning efficient encodings, i.e., repre-
sentations, of the input data, typically, for dimensionality reduction and feature extraction. The simplest AE
consists of three layers: the input, hidden, and output layers. An Encoder compresses the high-dimensional
input into lower-dimensional encoding, which is also called a bottleneck, and a decoder tries to reconstruct
the data back from its compressed form (see Figure 29).
Let us consider input x∈X, representation h∈H, and corresponding encoder Ew:X→Hand de-
coder Dθ:H→X, parametrized by wand θ, respectively. For each input x, the corresponding represen-
tation h=Ew(x), and the decoded output x′=Dθ(h) are calculated. To train an AE, one has to define a
measure d(x,x′) of how different xand x′are. Then, the following cost function, which shows how well the
input can be reconstructed from the hidden variables, is minimized during the training phase:
J(w,θ)=X
i
d(xi,Dθ(Ew(xi))).
The ideal AE should be sensitive enough to the input to be able to reconstruct it accurately, but also not
to memorize the inputs to avoid overfitting. To achieve this balance, several variants of AEs with different
architectures and regularization techniques were proposed, including undercomplete, sparse, contractive,
and denoising autoencoders. A special variational autoencoder (VAE) is used for data generation and will
be discussed in Section 4.3.3.
37
Figure 29: In the hidden layer, the autoencoder learns low-dimensional representations of the high-
dimensional input. These representations are used to reconstruct the data in the output layer.
Restricted Boltzmann machine. RBM shares a similar idea with AE, but uses a stochastic approach.
It has only two layers: visible and hidden. All the visible nodes are connected to all the hidden nodes by
undirected edges (see Figure 30).
Figure 30: Restricted Boltzmann machine with four visible and three hidden units.
RBM is an energy-based model that tries to minimize a predefined energy function. For RBMs, the energy
function for visible units xand hidden units his defined as:
E(x,h)=−X
i
aixi−X
j
bjhj−X
i,j
xihjwi j ,
where ai,bjare biases, and wi j are the weights between all the units. The marginal probability of a visible
vector xis:
p(x)=Phe−E(x,h)
Px,he−E(x,h)
.
The derivative of the log probability of a training vector with respect to a weight is:
∂log p(x)
∂wi,j=〈xihj〉d a t a −〈xihj〉mod el ,
where the angle brackets are used to denote expectations under the distribution specified by the subscript
that follows. So the learning rule for the optimization step with learning rate αwill be:
∆wi,j=α(〈xihj〉d at a −〈xihj〉model ).
38
Getting an unbiased sample for 〈xihj〉d at a is easy, unlike for 〈xihj〉model . That is why the second term is re-
placed by 〈xihj〉recon through a reconstruction procedure. More information on that, as well as the detailed
derivation of the equations, is provided in [Hinton, 2012]. The learning works well for many applications,
even though it uses the crude approximation of the gradient of log probability.
Deep belief network. DBNs are deep neural networks that consist of multiple layers of hidden units.
They can be viewed as several RBMs or AEs stacked together, with hidden units of the previous model being
used as visible units to the next one.
The training method for RBMs, described in the previous section, is usually applied to train DBN that
consists of stacked RBMs. After the first RBM is trained, the next one is stacked atop it, and the last trained
layer is perceived as the visible layer for the new RBM.
AE and, especially, RBM-based DBNs are used in many BCI studies [Altaheri et al., 2021]. Often, after
DBN is trained in an unsupervised manner, it gets trained again, but this time with labeled data, and is used
as a classifier [Yin and Zhang, 2017, Lu et al., 2017, Chu et al., 2018]. Several classification approaches use
a DBN applied together with CNN in a hybrid model [Yang et al., 2020b, Wang et al., 2018b]. A traditional
classifier, usually SVM, can also be used on top of DBNs for MI-EEG classification [Chu et al., 2018, Xu et al.,
2020].
As we mentioned, DBNs are often utilized in a two-step procedure. First, on unlabeled data, they are
employed to learn useful representations. Second, a classifier, which can be a DBN or any other model, is
trained on top of the extracted features. This gives a possibility to use it as a powerful feature extraction
unit in a combination with other models. However, lately, the use of DBNs in BCI research has been de-
creasing [Roy et al., 2019]. It probably happens because more and more studies prove good hierarchical
feature learning capabilities of CNNs, which makes them more attractive models since they also provide an
end-to-end classification framework.
4.3.3 Generative models
Generative models aim at learning the true data distribution based on the training set and generating new
data with some variations. In DL, neural networks are used to approximate the true distribution function.
The most common and successful generative DL models are variational autoencoders (VAEs) and generative
adversarial networks (GANs). Both are used to generate MI-EEG data, while GANs usually demonstrate
better results [Zhang et al., 2020, Fahimi et al., 2021]. Commonly, the generated data is used to augment and
improve the training dataset and, thus, classification results.
Variational autoencoder. VAE is associated with a standard autoencoder, described in Section 4.3.2,
which can learn low-dimensional representations of the input. However, such representations can be only
mapped to a corresponding input with a decoder, but cannot be used to generate similar outputs with some
variability. VAE, instead, learns the probability distribution of the training data to achieve this goal.
VAE assumes that there is some latent (hidden) variable hthat generates an observation x. We want
to maximize the likelihood of the data with a probability distribution p(x), which is connected with the
prior p(h), the likelihood pθ(x|h), and the posterior pθ(h|x) according to the Bayes theorem:
pθ(h|x)=pθ(x|h)p(h)
p(x).
We can infer pθ(h|x) using a method called variational inference. For that, the pθ(h|x) is approximated
with a simpler distribution qw(h|x). It can be derived [Kingma and Welling, 2013] that maximizing the log-
likelihood of the observed data, together with minimizing the difference between the approximate posterior
and the exact posterior, is equivalent to maximizing the so-called evidence lower bound (ELBO):
Lθ,w(x)=E(ln(pθ(x|h))) −DK L[qw(h|x)||p(h)].
Instead of encoding the latent variables directly, the encoder qw(h|x) outputs parameters describing a
distribution of latent variables. In VAEs, the normal distribution is assumed for the prior; thus, the encoder
will output two vectors for the mean and variance (covariance matrix is assumed to be diagonal) of the
distribution. The decoder pθ(x|h) is used to generate data based on the hvalue (see Figure 31).
During the training, we cannot backpropagate through hbecause it is sampled randomly. To solve this
problem, a so-called reparametrization trick is used. The idea is, instead of sampling hdirectly from N(µ,C),
39
Figure 31: VAE architecture with an encoder that outputs mean and variance parameters, and a decoder that
is used to generate data. Latent variables are computed according to the reparametrization trick: h=µ+Lϵ,
where C=LL⊺.
to sample ϵfrom N(0,I) and then calculate h=µ+Lϵ, where C=LL⊺. With such an implementation, we
can backpropagate through the node hsince its randomness is shifted to ϵ.
VAEs are relatively easy to train, can be used to learn complex distributions, and generate a wide variety
of samples. VAEs have a clear objective function to optimize, which makes it easy to compare two VAE
models with each other. However, VAEs usually perform worse than GANs on MI-EEG datasets [Zhang et al.,
2020, Fahimi et al., 2021], probably because they do not generate high-quality samples due to injected noise;
and generally may struggle to generate samples from high-dimensional distributions.
Generative adversarial network. GAN is the most commonly used generative model in deep learning
in general, and in the BCI field in particular. Instead of learning an explicit probability distribution esti-
mation as variational autoencoders do, GANs utilize a game theory approach and are based on the idea of
competition between two neural networks, a Generator and a Discriminator.
The training is based on the principle of adversarial learning. The Generator tries to generate realistic
data from noise, while the Discriminator aims at recognizing which data is fake, i.e., produced by the Gen-
erator, and which is real. During the learning, both players improve. The Generator tries to trick the Dis-
criminator by improving the quality of the output data, while the Discriminator gets better at differentiating
between fake and real samples in order not to get tricked (see Figure 32).
Figure 32: The Generator takes a random noise as input and tries to output the most realistic samples to trick
the Discriminator. The Discriminator learns how to improve its discriminative capability of distinguishing
fake samples from real ones. Its output is the probability pof the input sample to be real. After the training
is finished, one can keep only the Generator network and generate new realistic data from random noise.
Given noise z, the Generator outputs samples G(z). The Discriminator predicts the probability of the
40
input to be real. If the sample is real, D(x) should output 1, and if the sample is fake, D(G(z)) should output 0.
During the training, the Generator tries to minimize the following function, while the Discriminator, instead,
tries to maximize it:
Ex[l og (D(x))] +Ez[l og (1 −D(G(z)))].
The expectations Exand Ezare over all real samples, and over all random inputs to the generator, respec-
tively. After the training is finished, one can use the Generator to produce new realistic samples.
GANs can generate a wide variety of high-quality samples of complex data and are the most powerful
deep generative models. Training datasets augmented with GANs have been shown to improve classifica-
tion results significantly on several BCI datasets [Abdelfattah et al., 2018, Fahimi et al., 2021, Zhang et al.,
2020, Panwar et al., 2020]. However, this improvement varies greatly in different datasets and preprocessing
modes, and a clear conclusion regarding their effectiveness in the BCI field cannot be made so far [He et al.,
2021]. Among other disadvantages, the difficulties with training and sensitivity to hyperparameter choice
and initialization have to be mentioned.
4.4 Chapter summary
A big variety of machine learning methods is used for MI-EEG data analysis. While being simple, compu-
tationally efficient, and easy to train, traditional classifiers, such as LDA and SVM, are still widely used for
signal classification. Moreover, in the case of small training datasets, combined with efficient feature extrac-
tion techniques, they find themselves among the best models in terms of classification accuracy. The main
drawback of these classifiers is the need for separate preprocessing, feature extraction, and feature selection
steps.
Geometry classifiers, and in particular Riemannian classifiers, solve this problem by operating in a par-
ticular Riemannian manifold, a space of covariance matrices of raw EEG signals. They demonstrate high-
accuracy performance and allow using a variety of different algorithms by projecting data onto tangential
to the Riemannian manifold space. However, computing Riemannian distances between datapoints is chal-
lenging, especially in the case of the big number of EEG electrodes.
Deep learning models also offer an end-to-end classification framework. They can automatically learn
complex features from raw EEG data using deep architectures and scale well with the size of the training
dataset. CNNs, alone or combined with other discriminative, representative or generative neural networks,
appear the most successful in MI-EEG classification. Relatively hard training procedures and poor perfor-
mance on small training datasets remain big challenges for deep learning architectures in BCIs.
41
5 State-of-the-art in EEG data augmentation
EEG signal classification is a difficult task due to several factors such as weak EEG amplitudes, low signal-to-
noise ratio, and non-stationarity of the data. Recently, deep learning models have emerged as a promising
approach for EEG classification and have demonstrated comparable or even superior performance to tra-
ditional classifiers on multiple datasets (see Section 4.3.1). Nevertheless, the potential of neural network
classifiers is hindered by the limited size of most available EEG datasets, which constrains further improve-
ment of the performance. Moreover, there is high variability in EEG recordings across different subjects and
sessions, which causes poor generalization to the independent test data of both traditional and deep learn-
ing models. This leads to weak performance and the need for a long calibration time, which restricts the
usage of EEG-based BCIs.
Data augmentation (DA) is a well-established approach in computer vision that expands the size and
diversity of the training dataset, thereby improving the generalization of the models [Shorten and Khoshgof-
taar, 2019, Yang et al., 2022]. This is achieved by applying various transformations, such as rotation, scaling,
and cropping, to the original data, while preserving corresponding labels.
While effective for data augmentation in image processing, these transformations may not be directly ap-
plicable to EEG data due to the continuous time-series nature of EEG recordings. For example, simple trans-
formations such as rotation or shifting can disrupt the time domain structure of the signals. To overcome this
limitation, different strategies have been utilized for EEG data augmentation [He et al., 2021]. While some
basic transformations such as adding or multiplying noise can be directly applied to EEG recordings, other
DA techniques involve manipulating EEG representations in the form of spectral images or other extracted
features. DA has been applied in BCIs not only to expand the available dataset and enhance classification
accuracy, but also to mitigate the issue of class imbalance and reduce or suppress calibration time.
In this chapter, we aim at reviewing and comparing DA methods for EEG data. These methods can be
divided into two major groups based on their use of deep learning (see Table 3). The first group of non-DL
methods includes segmentation and recombination, sliding windows, empirical mode decomposition, as
well as time domain, frequency domain, and geometrical transformations. The second group includes DL-
based approaches such as GANs and their variants, as well as diffusion models. Our primary focus is on MI
data, but we also refer to studies proposing methods for other BCI paradigms.
5.1 Non-DL strategies
The majority of non-deep learning data augmentation techniques involve basic modifications to the orig-
inal data to expand the size of training datasets. Certain methods apply direct manipulations to EEG time
series, while others convert the measured signals into the frequency domain, extract features, decompose
the signals, or alter sensor positions.
5.1.1 Segmentation and recombination
To augment EEG data, one approach is to segment and recombine measured trials in either the time or time-
frequency domain [Lotte, 2015]. In the time domain, a training EEG trial can be divided into segments and
then combined with segments from other random trials of the same class to create a new synthetic trial (as
shown in Figure 33a). In the time-frequency domain, each training trial can be transformed into a time-
frequency representation using a short-term Fourier transform (STFT), and the recombination process is
applied to STFT windows from different trials. The resulting synthetic trial is obtained by inverse STFT (as
shown in Figure 33b). This augmentation technique has been applied to MI and workload data, resulting
in up to 15% improvement [Lotte, 2015] in LDA classification, particularly when the number of available
training trials is limited.
A similar strategy was utilized [Dai et al., 2020] to augment MI data for CNN classification. Compared
to the performance without DA, which was already superior to several other state-of-the-art classifiers, the
accuracy increased by 2% after the augmentation.
It was proposed [Xie and Oniga, 2023] to combine the time and time-frequency segmentation and re-
combination in one DA pipeline. In this approach, three random EEG training trials were selected. The first
two trials were segmented and recombined in the time domain into the intermediate sample. This sample
was then recombined with the third selected trial in the time-frequency domain by swapping two frequency
bands of 7 −13 Hz and 14 −30 Hz. Such augmentation resulted in the improvement of 4.41% of average
accuracy, and up to 11.15% for individual subjects in CNN-based classification.
42
Table 3: Common EEG data augmentation methods
Method group DA method Reference Paradigm DA input Accuracy gain
Non-DL
Segmentation-
recombination
time and freq SR [Lotte, 2015] MI, WL TS up to 15%
time and freq SR [Dai et al., 2020] MI TS up to 2%
combined time and freq SR [Xie and Oniga, 2023] MI TS up to 4.4%
noise SR [Zhang et al., 2021a] MI TS 2.8 - 3.6%
Sliding windows SW [Ko et al., 2019] MI TS NA
SW [Majidov and Whangbo, 2019] MI TS NA
Temporal
domain
noise addition [Cecotti et al., 2015] ERP TS AUC +0.38
noise, flip [Freer and Yang, 2020] MI TS up to 5.5%
noise, shift, masking [Mohsenvand et al., 2020] ER, SD TS 2-6%
averaging [Yang et al., 2020a] VEP TS 3-5.6%
noise, flip, masking, reverse [Rommel et al., 2022b] MI, SD TS up to 25%
Frequency
domain
amplitude perturbation [Li et al., 2019] MI TS up to 3.8%
amplitude perturbation [Schirrmeister et al., 2017] MI TS NA
bandstop filter [Cheng et al., 2020] MI TS 3.7%
frequency shift [Rommel et al., 2022a] SD TS up to 20%
FT surrogates [Schwabedal et al., 2019] SD TS up to 7%
Geometrical
transformations
image transformation [Shovon et al., 2019] MI SI NA
channel rotation [Krell and Kim, 2017] ERP TS up to 1.5%
channel rotation [Rommel et al., 2022b] MI TS up to 2%
channel shuffle, dropout [Saeed et al., 2021] AE TS NA
cov matrix interpolation [Kalunga et al., 2015] ERP, VEP EF 1.5%
differential entropy [Wang et al., 2018a] ER EF up to 40.8%
Empirical mode
decomposition
EMD [Dinarès-Ferran et al., 2018] MI TS NA
EMD [Zhang et al., 2019] MI TS up to 1.9%
graph EMD [Kalaganis et al., 2020] WL TS up to 9%
DL
Generative
adversarial networks
WGAN [Hartmann et al., 2018] MT TS NA
WGAN-GP [Panwar et al., 2020] RSVP TS NA
cWGAN, sVAE, sWGAN [Luo et al., 2020] ER SI up to 10.2%
DCGAN [Zhang et al., 2020] MI SI 17-21%
cDCGAN [Fahimi et al., 2021] MI TS 3.5-7.3%
WGAN, DCGAN, VAE [Aznan et al., 2019] SSVEP TS up to 35%
RGAN [Abdelfattah et al., 2018] MI, MT TS up to 39.1%
Diffusion models DPM [Torma and Szegletes, 2023] MI, ERP SI NA
DPM [Tosato et al., 2023] ER SI 1.5%
Abbreviations: SR, segmentation and recombination; SW, sliding windows; EMD, empirical mode decomposition; GAN, generative adversarial network;
WGAN, Wasserstein GAN; WGAN-GP, WGAN with gradient penalty; cWGAN, conditional WGAN; sWGAN, selective WGAN; VAE, variational autoencoder; sVAE,
selective VAE; RGAN, recurrent GAN; DPM, diffusion probabilistic model; NA, not applicable; MI, motor imagery; WL, workload; ERP, event related potential;
ER, emotion recognition; SD, sleep data; AE, abnormal EEG; VEP, visual evoked potential; SSVEP, steady-state VEP; MT, motor task; RSVP, rapid serial visual
presentation; TS, time-series; SI, spectral images; EF, other extracted features.
43
In the MI EEG classification study by [Zhang et al., 2021a], the authors assumed that meaningful brain
signals exist in the frequency band from 0−100 Hz, while the frequencies above 100 Hz consist of noise and
artifacts. Thus, the method recombined an above 100 Hz noise signal from one trial with a below 100 Hz
signal from another trial, which resulted in 2.8% and 3.6% improvement in two and four-class CNN-based
classification, respectively.
(a) Segmentation and recombination in time domain
(b) Segmentation and recombination in the time-frequency domain
Figure 33: Segmentation and recombination method for EEG data augmentation (adapted from [Lotte,
2015]). (a) EEG trials are segmented and recombined to obtain new artificial trials. (b) EEG trials are trans-
formed into time-frequency representations by STFT and recombined in that domain.
5.1.2 Sliding windows
A simple strategy that has been used for increasing the size of EEG datasets is the sliding windows ap-
proach. Multiple segments of length twind ow with a shift of tshi f t can be extracted from a single training
EEG trial (see Figure 34). In [Ko et al., 2019], this strategy was applied with twi ndow =2sand a stride of one
time point, resulting in 189 segments from a single trial. A DL model RSTNN [Ko et al., 2018] was tested only
on the resulting augmented dataset, therefore, the impact of the augmentation is unknown. In [Majidov and
Whangbo, 2019], this approach was applied to extract four segments of twi ndow =3sand shift tshi f t =0.3s
from each training trial. Multiple classifiers were trained on the augmented dataset for MI classification, but
no comparison to the classification without augmentation was provided.
It is often assumed [Ko et al., 2019, Majidov and Whangbo, 2019] that DL classifiers cannot effectively
learn from small datasets, therefore, the performance prior to DA is not reported. However, it is important
to validate a classifier before DA since there are instances [Dai et al., 2020] where the accuracy decreases
after implementing sliding windows, possibly due to the loss of motion-related information caused by using
smaller windows.
44
Figure 34: Sliding windows approach allows extracting multiple segments from a single EEG trial.
5.1.3 Transformations in temporal domain
Various transformations can be applied directly to EEG recordings in the time domain. It was proposed [Ce-
cotti et al., 2015] to increase the number of training trials by simply adding Gaussian noise to the electrode
values. On an ERP classification task with only few available training trials, the AUC of the Bayesian LDA
classifier after DA improved from 0.53 to 0.91. Besides noise addition, several other DA methods, i.e., noise
multiplication and flip of the signal, were also tested [Freer and Yang, 2020] on the convolutional long-short
term memory network (CLSTM) for MI EEG binary classification. The results indicated that classification
accuracy could gain up to 5.5% after the transformations in the temporal domain.
In [Mohsenvand et al., 2020], besides applying noise addition and temporal shift, a zero-masking ap-
proach was proposed. A portion of the signal was randomly picked and set to zero in order to improve the
performance of several DL classifiers. The effect of multiple DA methods was analyzed, concluding that re-
moving some of them from the processing pipeline leads to the decrease in the accuracy of classification of
emotion recognition and sleep data by 2 to 6%. Similar zero-masking method was compared [Rommel et al.,
2022b] with noise addition, sign flip, and time reverse [Rommel et al., 2022a] approaches on MI and sleep
datasets. The results of the classification on the particular CNN model [Chambon et al., 2018] indicated that
the time reverse strategy could improve MI classification in the case of small training sets for up to 25%.
With larger training datasets, zero-masking was the most effective among the tested methods.
The averaging technique has been applied [Yang et al., 2020a] for visual evoked potential (VEP) classifi-
cation with the RNN with attention mechanism. With this technique, the authors proposed to address the
effect of signal-to-noise ratio (SNR) across trials. Several random trials of the same class were chosen to cal-
culate the average potential, which was then considered a new artificial EEG trial. The number of averaged
trials khad to be chosen. Low kled to insufficient diversity, while large kproduced meaningless artificial
data. The optimal value of kallowed achieving 3.0 to 5.6% gain in classification accuracy.
5.1.4 Transformations in frequency domain
Instead of manipulating the EEG potentials directly, different methods applied transformations in the fre-
quency domain. A so-called amplitude perturbation technique was applied [Li et al., 2019] for the MI clas-
sification task. The time-series signals were transformed into spectral images by STFT. Then, the amplitude
values were perturbated by adding Gaussian noise, and, finally, the inverse STFT was performed to obtain
time-series signals again. This DA method boosted the performance of the proposed DL model, gaining up
to 3.8% accuracy. Earlier, this technique was also utilized to investigate the causal effect of the amplitude
perturbation on the deep CNN [Schirrmeister et al., 2017]. In addition, the researchers demonstrated that
small amplitude perturbations did not cause the CNN to misclassify the trials.
The study [Cheng et al., 2020] on biosignal analysis compared different DA methods for EEG and pro-
posed a bandstop filtering approach. The trials were filtered at a randomly selected frequency band using a
bandstop filter. With this method, the classification improvement of 3.7% was achieved on a MI dataset.
It was also proposed [Rommel et al., 2022a] to randomly translate all channel power spectral density (PSD)
by a small shift. A time domain modulation was used to shift the frequencies in EEG signals, which led to up
45
to 20% classification gain on sleep data.
The FT surrogate transform [Schwabedal et al., 2019] has been proposed as a promising technique for
frequency data augmentation. The method involved replacing the phases of Fourier coefficients with uni-
formly sampled random values from [0,2π], assuming that EEG signals can be approximated by linear sta-
tionary processes characterized by their Fourier amplitudes. It was suggested to address the class imbalance
and to increase the size of the training dataset. Empirical results indicated that this approach was able to
enhance classification accuracy by up to 7%.
5.1.5 Geometrical transformations
It is commonly believed that the geometrical transformations frequently used in computer vision, such as
rotations and flipping, do not apply to EEG data due to their unique temporal structure. Nonetheless, such
strategies were applied [Shovon et al., 2019] to STFT images extracted from MI data. In particular, spectral
images were rotated, flipped, zoomed in, and zoomed out to generate additional 1000 artificial samples for
a CNN model training. However, in this study, the impact of such DA was not reported.
Another group of methods applied geometrical transformations in the spatial domain of sensors. A sen-
sor rotation method has been initially proposed [Krell and Kim, 2017] for P300 data. It was motivated by the
fact that the EEG cap can slightly move over the head of a subject between different sessions. The method
approximated what would have been recorded by a BCI system with slightly shifted positions of channels.
A random angle was sampled from a given interval, and the 3Dcoordinates of all EEG channels were ro-
tated by this angle around the picked axis. The values of the channels were interpolated with a radial-basis
function interpolator. The augmented data improved SVM classification by 1.5%. This method, but with
spherical splines for the interpolation, was also tested [Rommel et al., 2022b] on MI classification, achieving
an improvement of 2%. Another study proposed [Saeed et al., 2021] to shuffle and dropout EEG channels.
During the training process, certain channels were shuffled, and a random binary mask was implemented
over the channels to eliminate some of them, thus enhancing the model’s robustness.
Geometrical transformations can also be applied in feature domains. For example, in [Kalunga et al.,
2015], it was proposed to interpolate between covariance matrix representations on the Riemannian man-
ifold (see Section 4.2). The interpolation was done on the geodesic connecting each pair of original trials
of the same class, which guaranteed that a new artificial trial lay on the manifold. This strategy boosted
the performance of the MLP classifier by 1.5%. Another approach [Wang et al., 2018a] applied a shift to
the extracted differential entropy features by adding Gaussian noise to the feature matrices. The method
was tested on the classification of emotion recognition data with three different classifiers, namely, two DL
models and SVM. Results demonstrated that augmented data could significantly improve the performance
of DL models (by up to 40.8%), but had little effect on the SVM classifier.
5.1.6 Empirical mode decomposition
Recently, a novel method for EEG data augmentation, based on the empirical mode decomposition (EMD),
has been proposed [Dinarès-Ferran et al., 2018]. EEG trials were decomposed into a finite number of intrin-
sic mode functions (IMFs) [Huang et al., 1998]. Each of these functions represented a non-linear oscillation
of the signal. Thus, the method was more suitable for non-stationary EEG signals than FFT or wavelets. Af-
ter the IMFs were calculated, the signal could be recovered by adding all the IMFs and the residue. A new
artificial trial was generated by swapping the IMFs between the original training trials of the same class.
In particular, it was proposed to take the first IMF from the first original trial, the second IMF from the
second trial, and so on. The results showed that, with this strategy, up to 50% of the original trials could
be replaced by artificial samples without accuracy loss in LDA classification of MI data. This method was
also tested [Zhang et al., 2019] in combination with DL models, which could enhance the performance and
achieve up to 9.1% classification accuracy gain.
The EMD algorithm has been extended [Kalaganis et al., 2020] to graph signals. When combined with
graph convolutional neural network (GCNN), the method has been shown to be effective in predicting
driver’s responses and classifying passive vs. attentive conditions in game-like BCIs. The results show that
both classical and graph EMD improve GCNN classification, while graph EMD boosts the performance by
another 9% compared to the classical EMD.
46
5.2 Deep learning strategies
Traditional data augmentation methods have demonstrated promising results in EEG research, but they
have achieved limited accuracy gain and often performed augmentation solely on the input space without
taking into account the inherent features of the signals. In contrast, neural networks have been shown to
be effective in leveraging the underlying characteristics of EEG. Variational autoencoders (VAEs) and gen-
erative adversarial networks (GANs) are two typical deep learning strategies for DA. As we described in Sec-
tion 4.3.3, VAE learns the probability distribution of the training data, which is then used to generate artificial
trials. In contrast, GAN employs a game theory approach and is based on adversarial learning. In most of
the recent EEG studies on DA, GANs and their variants outperformed VAEs [Zhang et al., 2020, Luo et al.,
2020]. Therefore, in Section 5.2.1, our primary focus is GAN-based models, while providing some VAE re-
sults for comparison. Another promising deep learning strategy based on diffusion models is described in
Section 5.2.2.
5.2.1 Generative adversarial networks
Originally introduced for computer vision, GANs have been extended to other types of data, e.g., audio
data [Oord et al., 2016, Engel et al., 2023]. Data augmentation strategies based on GANs typically utilized
class-conditioned learning [Mirza and Osindero, 2014, Odena et al., 2017] to increase the amount and diver-
sity of the labeled training data. Recently, multiple GAN models have been proposed for EEG data augmen-
tation [He et al., 2021, Hartmann et al., 2018, Panwar et al., 2020, Luo et al., 2020]. DA has been applied to
both time-series data and extracted spectral images.
Wasserstein GAN. Wasserstein GAN (WGAN) [Arjovsky et al., 2017], which, compared to the original
GAN, uses a new loss function derived from the Wasserstein distance, has been utilized in many BCI studies.
It was shown [Hartmann et al., 2018] that it was possible to generate realistic EEG data with a modification of
the improved WGAN [Gulrajani et al., 2017]. The generated signals strongly resembled the real EEG record-
ings in the time and frequency domain. In [Hartmann et al., 2018], however, only single-channel EEG was
synthesized, suggesting that multichannel EEG generation should be explored in future works. Soon, a novel
WGAN with gradient penalty (WGAN-GP) was introduced [Panwar et al., 2020] to generate 64-channel EEG
trials for rapid serial visual presentation (RSVP) data. The quality of generated data was evaluated via vi-
sual inspection and log-likelihood score from a Gaussian mixture model. In addition, it was shown that the
class-conditioned WGAN-GP outperformed EEGNet on the classification task. Further variants of WGAN
were applied [Luo et al., 2020] to enhance the performance of the classifier in the emotion recognition tasks.
Realistic EEG data in the form of power spectral density and differential entropy were generated using con-
ditional WGAN (cWGAN), selective WGAN (sWGAN), and selective VAE (VAE). The training dataset was aug-
mented with different amounts of generated data and tested on the classification task with SVM and DNN
models. Experiments indicated that the proposed methods enhanced the performance of the classifiers
and outperformed other DA methods including VAE, noise addition, and channel rotation, while sWGAN
in combination with DNN gained 10.2% accuracy, which was the biggest improvement among the tested
models.
Deep convolutional GAN. A commonly used GAN variant for EEG data augmentation has been deep
convolutional GAN (DCGAN) [Radford et al., 2016]. In contrast to the original GAN where the Generator
uses a fully connected neural network, DCGAN employs a transposed CNN to upsample the images. It was
proposed [Zhang et al., 2020] to apply DCGAN to the spectral images extracted from two MI datasets. Im-
provements in the classification accuracy of 17% and 21% were observed after DA, which outperformed
all other tested methods including channel rotation, shifting, noise addition to the spectrogram, and VAE.
Instead of generating spectral images, [Fahimi et al., 2021] proposed a conditional DCGAN architecture to
generate subject-specific EEG trials in the form of time-series. A subject-specific feature vector extracted
from half of the target subject’s samples was passed to the Generator in addition to random noise. Using
the proposed DA method yielded a significant improvement of 3.57% to 7.32% on different data. The use of
three different DA methods, i.e., DCGAN, WGAN, and VAE was explored on the SSVEP cross-subject classi-
fication. Several strategies were implemented for DA varying the portion of artificial data in the training (or
pre-training) datasets, boosting the performance of classification by up to 35%.
47
Recurrent GAN. [Abdelfattah et al., 2018] proposed a DA model based on recurrent GAN (RGAN) [Mo-
gren, 2016] that utilizes a recurrent neural network in the generator component. Due to its ability to capture
time dependencies, RGAN is a well-suited model for generating time-series data. The method was tested
on the MI classification task, improving the accuracy by up to 39.1% on small dataset sizes and showing
superiority over AE and VAE methods.
5.2.2 Diffusion probabilistic models
Recent advances have proven that diffusion probabilistic models (DPMs) could outperform GANs in both
image [Dhariwal and Nichol, 2021] and audio generation [Kong et al., 2023]. A diffusion process progres-
sively transforms input data from any complex distribution into Gaussian noise. By training a diffusion
model, a way of performing the reverse process is learned and can be used to generate artificial data (see
Figure 35). DPM employs a parameterized Markov chain trained using variational inference for data gener-
ation.
Figure 35: The process of generating EEG spectral images from Gaussian noise with a diffusion model. The
images are generated upside down, with the x-axis representing the channels and the y-axis representing the
frequencies (from [Tosato et al., 2023], licensed under CC BY 4.0).
The capabilities of DPMs in multichannel EEG data generation were investigated [Torma and Szegletes,
2023] on ERP and MI data. The data in the form of spectral images was generated conditioned on class
labels. It was shown that the proposed model could learn the main frequency characteristics and subject-
specific features. Compared to the GAN model [Panwar et al., 2020], DPM generated signals that were more
useful for the pre-training of EEGNet, before re-training on the original set. A similar strategy was employed
in [Tosato et al., 2023] for emotion-labeled EEG classification. The input to the classifier were images, i.e.,
EEG-based electrode-frequency distribution maps (EFDMs) [Wang et al., 2020]. These are grayscale images
that plot the intensity values in each channel at each frequency, each transform step. The improved diffusion
model [Nichol and Dhariwal, 2021] was used to generate EFDMs, that were squared, flipped upside-down,
and converted to RGB images to meet the input format the model required. The DA model yielded an im-
provement of 1.5% in DL-based classification.
Diffusion models are a powerful tool, yet computationally expensive. They need many iterations dur-
ing sampling to generate data, making them significantly slower than GANs. Recent studies [Luhman and
Luhman, 2021, Kong and Ping, 2021, Salimans and Ho, 2022] proposed multiple ways to fasten the slow
distillation process, which remains the main limitation of the method.
48
5.3 Comparison and summary
Numerous studies addressed EEG data augmentation over the last eight years. However, most of them were
tested on different datasets with different classification models, which makes it difficult to compare DA
approaches with each other. Nonetheless, some comparative results for MI data were reported.
Three different papers compared several DA methods on the BCI Competition IV 2a dataset [Lee et al.,
2020]. The segmentation-recombination method was shown [Dai et al., 2020] to be more effective than slid-
ing windows and noise addition on the classification with a hybrid-scale CNN [Dai et al., 2020]. [Freer and
Yang, 2020] tested noise addition, signal flip, noise multiplication, frequency shift, as well as different com-
binations of these DA methods on three different classifiers. The best data augmentation methods for the
Riemannian MDM [Barachant et al., 2012], Shallow CNN [Schirrmeister et al., 2017], and introduced C-LSTM
classifiers were noise multiplication, noise multiplication combined with noise addition, and noise addition
combined with signal flip, respectively. Finally, the effect of multiple transformations in time, frequency, and
spatial domain listed in Table 3 was compared by [Rommel et al., 2022b] for the Deep CNN [Schirrmeister
et al., 2017]. The results demonstrated that MI could benefit from all three groups of DA. On smaller training
sets, time reverse, and FT surrogates were the most effective, while channel dropout led to the best results
on larger training sets.
In [Cheng et al., 2020], it was shown that on the ResNet [He et al., 2016] classification of MI PhysioNet
dataset [Goldberger et al., 2000], zero-masking was the most effective transformation among other simple
time and frequency transformations. Few papers compared deep learning DA with non-DL models. [Fahimi
et al., 2021] implied superiority of DL methods, showing that cDCGAN demonstrated significantly better re-
sults than VAE, which, in its turn, outperformed segmentation-recombination approach on MI classification
with DCNN [Fahimi et al., 2019]. Independently, DCGAN was compared by [Zhang et al., 2020] with VAE and
several traditional augmentation methods such as geometric transformation and noise addition. The results
revealed that both DCGAN and VAE outperformed the traditional methods, while the DCGAN demonstrated
the best performance.
Given the limited number of studies, providing a definitive conclusion about the most effective DA
method is challenging. Researchers have to select a suitable method for each specific task, taking into ac-
count several critical factors, such as the classification model employed, the representation of EEG features,
the BCI paradigm utilized, and the amount of training data available.
When comparing DA methods, it is important to mention their limitations. Traditional methods may
destroy data labels [Shorten and Khoshgoftaar, 2019] and lose the motion-related information, while pro-
viding limited improvement in the diversity of generated data [Zhang et al., 2020]. GANs have instability
during training that may result in meaningless output. In addition, the training process of a GAN requires
a considerable amount of measured data, which might not be available. Another challenge that GANs usu-
ally face is the low quality of the generated samples [Salamon and Bello, 2017]. Recently introduced DPMs
are computationally expensive, which makes the process of data generation significantly slower than with
GANs.
We propose two novel data augmentation approaches for MI EEG data that distinguish themselves from
existing methods by incorporating physiological and geometric information about the brain. Conventional
DA methods offer limited enhancements in classification accuracy, while deep learning approaches require
complex training procedures and substantial amount of original data. Our proposed methods aim to over-
come both drawbacks, offering a relatively simple solution with a significant improvement in classification
accuracy. Overall, our research focuses on investigating the use of physiological information about the brain
to improve motor imagery EEG classification within the challenging context of high cross-subject and trial-
to-trial variability, coupled with a limited amount of available training data.
49
6 Physiology-informed data augmentation. Material and methods
In this chapter, we present our two novel techniques for augmenting EEG data, which differ from previous
approaches by incorporating physiological information about the brain. In Section 6.1, we begin with a de-
scription of the dataset used for evaluation. Then, in Section 6.2, we provide detailed explanations of our
approaches for cross-subject and within-subject classification. Additionally, we describe employed classifi-
cation models (Section 6.3) and the evaluation procedure (Section 6.4).
6.1 Material
We evaluated the proposed methods on a large-scale BCI study with motor imagery as a control paradigm.
Several papers analyzing the data set have been published, e.g., [Blankertz et al., 2010, Hammer et al., 2012,
Sannelli et al., 2019]. The study was approved by the Ethical Review Boards of the Medical Faculty, University
of Tübingen. 80 participants, who did not take part in any BCI study before, took part in the study, 39 male,
41 female (aged 29.9±11.5y, with a range of 17-65).
In the present analysis, we considered only the runs in which participants performed kinesthetic motor
imagery [Neuper et al., 2005] according to visual cues in offline mode, i.e., without BCI feedback. Addition-
ally, we made the following two restrictions:
(1) Restriction to motor imagery of the hands: While three types of motor imagery were used in the study,
we consider only the two classes of left-hand and right-hand motor imagery. The reason for this restriction
is the following: The third class of motor imagery involves either one foot or of both feet, according to the
participant’s preference. Different from the hands, the effect of motor imagery of a foot is fundamentally
different between participants. Few have a decrease in the amplitude of the sensorimotor rhythm (SMR)
over the foot area (which would be expected and analogue to the effect of hand imagery), many have an
increase of the SMR amplitude over the hand areas (probably due to inhibition, [Neuper and Pfurtscheller,
2001]), and some have an increase in rhythmic activity of the foot area, often in the beta range. These largely
different correlates of foot imagery between participants make learning participant-independent models
more complex. This is a challenge that is not addressed in the evaluation of the proposed methods.
(2) Restriction to participants with good baseline classification: Moreover, for cross-subject classification,
we included only data from those participants, for whom the (participant-specific) left- vs. right-hand mo-
tor imagery classification yielded an accuracy of at least 80 % (with the baseline shrinkage LDA approach).
While gaining better classification accuracies for participants in which baseline methods fail is an important
goal of method development, it is not the primary goal of the method proposed here. We want to demon-
strate that physiology-informed data generation helps to obtain similar or better classification results with
less training data. Applying this restriction, we retained 18 participants. For within-subject classification,
we included individuals who achieved a minimum classification accuracy of 65%. This was motivated by
the fact that, compared to cross-subject classification, within-subject classification generally tends to be an
easier task. Consequently, our dataset for within-subject classification consisted of 37 subjects.
Figure 36: Design of the trial for MI runs. The trial starts with a warning cue in the center of the screen in the
form of a cross at t= −2 s (baseline or pre-stimulus interval). After 2s (i.e., at t=0), the stimulus appears in
the form of an arrow. The direction of the arrow indicates the task to execute: left for left-hand movement,
right for right-hand movement, and down for foot movement. After 4s, the cross and arrow disappear, and
the screen stays blank for 2s. Then a new trial starts.
50
Brain activity was recorded from the scalp with multichannel EEG amplifiers (BrainAmp DC by Brain
Products, Munich, Germany) using 119 Ag/AgCl electrodes (reference at nasion; manufacturer EasyCap,
Munich, Germany) in an extended 10–20 system, and sampled at 1000 Hz with a band-pass filter of 0.05 Hz
to 200 Hz. For the present analysis, which is concerned with frequencies below 30 Hz, signals have been
subsampled at 100 Hz. From each participant, 75 trials of each motor imagery class (left hand, right hand,
and foot) have been recorded in pseudo random. Each trial (see Figure 36) started with the presentation of
a fixation cross for 2 s, which was then replaced for 4 s by an arrow pointing to the left, right, or downwards
to cue the respective motor-imagery task. Subsequently, the arrow disappeared, and a blank screen was
shown for 2 s. Thereafter, the next trial started. More details about the experimental setup can be found
in [Blankertz et al., 2010].
6.2 Methods
We introduce two MI-EEG data augmentation methods that take advantage of knowledge about neuro-
physics and geometry of the head. The two methods aim to address variability across subjects and variability
across trials within one subject, respectively. We add physiology-plausible variability in the data that can be
exploited by classification models and allows them to generalize reasonably from little data.
Our first method (see Section 6.2.1) aims at improving cross-subject classification by taking into account
individual physiological characteristics. As we discussed in Sections 2.1.2 and 2.3.1, even though areas rep-
resenting a certain function, like motor control of the left hand, are in similar locations across individuals,
there are noticeable differences in the recorded EEG. The signals that are acquired as EEG are mainly gener-
ated by pyramidal neurons in the cortex, which are oriented perpendicular to the cortical surface (see Sec-
tion 2.1.1). However, due to the folding of the cortical surface, the location and orientation of the respective
neurons are different. This can lead to substantial differences in the projection of the signals to the scalp.
While the resulting large inter-participant differences observed in the EEG patterns pose a considerable
challenge for participant-independent classifiers, we can also exploit this fact to modify data of one partici-
pant to mimic data of another (imaginary) participant.
Our second method (see Section 6.2.2) is focused on the classification within subjects. Although the
brain activity related to MI remains consistent in EEG trials, the impact of noise and artifacts may differ. To
enhance the robustness of classifiers against noise, we propose to augment the training dataset by swapping
motor-related and noise-related components from different trials within the same class.
6.2.1 DA for cross-subject classification
We propose a DA method to enhance the accuracy of cross-subject classification and allow a more robust
training of complex models. To achieve this, we suggest altering the measured EEG data of one subject in
order to simulate data of an imaginary subject in a physiology-informed manner. More specifically, we use
a source decomposition method on the given data, localize the dipoles of the current sources, randomly
change the location of those dipoles, and recombine the source signals using a forward model with the
modified dipoles to produce new natural-like EEG data (see Table 4 and Figure 37).
Signal decomposition. Our method relies on the general linear model of EEG (see Section 3.3.1) that
assumes that the measured EEG data X∈Rc×Tconsists of a linear superposition of source signals S∈Rp×T,
and Gaussian noise N∈Rc×T:
X=AS +N,
where A∈Rc×pis the propagation matrix of the forward model with cbeing the number of channels, pbeing
the number of modeled sources, and Tbeing the number of time points. Signal decomposition aims at
finding a matrix of spatial filters W∈Rc×pthat decomposes EEG signals back into components S′∈Rp×T:
S′=W⊤X.
We use the spatio-spectral decomposition (SSD) method [Nikulin et al., 2011] to split the EEG signal into
components. SSD takes measured data Xas an input and determines a matrix of spatial filters Wby max-
imizing the ratio of power in the frequency band of interest and the power in flanking frequency bands in
the extracted components (see Section 3.2.4). For the application to motor imagery data, we chose 8−13 Hz
51
as the band of interest and 5−8 Hz and 13 −16 Hz as flanking frequencies. Given a matrix W, one can cal-
culate (see Section 3.3.1) the corresponding matrix A, which implies A=(W⊤)−1in the case of invertible W.
Each column aiof the matrix Ais a spatial pattern:
A=¡a1a2... ai... ap¢.
Table 4: Data generation for cross-subject classification
Step Input Parameters Output
Notation Meaning Notation Meaning Notation Meaning
Source
decomposition
XEEG data [f−,f+]ccentral band Aspatial patterns
[f−,f+]lleft band Wspatial filters
[f−,f+]rright band S′source components
Source
localization
Aspatial patterns icomponent index ρdipole location
Lleadfields qdipole orientation
yi(t) source signal
Dipole shift ρdipole location rdistance ˜
ρnew location
Lleadfields ˜
aiaugmented pattern
Data generation ˜
aiaugmented pattern ˜
Xaugmented EEG
Source localization. The MUSIC algorithm [Mosher et al., 1992] is used to find a dipole fit for a given
SSD pattern aiusing a head model with corresponding leadfields L. We use the generic four-shell BEM head
model (see Section 3.3.2) with about 1000 voxels. MUSIC applies a scanning technique (see Section 3.3.3)
and outputs the location ρ(voxel of the head model), orientation q(normalized dipole moment) of the best
fitting dipole, and the corresponding source signal yi(t). Note that the localized source yi(t) can slightly
differ from the SSD source s′
i(t) due to the error of the algorithm.
The particular SSD pattern ai(or several patterns) has to be chosen for modification. There are several
options for selecting the components that are modified. For example, only the strongest SSD components,
or only the strong components with dipoles in the sensorimotor area can be used. Alternatively, those com-
ponents which can be fitted well with a single dipole can be modified. In our method, we opted for altering
all components (that are provided by SSD). This is motivated by our assumption that groups of neurons (pro-
ducing both motor-related and noise signals) across subjects can be shifted due to the individual brain fold-
ing.
Figure 37: The process of dipole manipulation. EEG signals are decomposed by SSD. For each component,
MUSIC is used to find a single dipole fit. The location of the dipole is modified in a four-compartment head
model. This yields a pattern that corresponds to the changed dipole, which can be used to generate new
signals, see Figure 38. Additional adjustments made for the altered pattern are described in Appendix A.1.
Dipole shift. The dipole position is shifted to another voxel of the head model in its neighborhood, and
the corresponding (augmented) pattern ˜
aiis determined. To generate data for other imaginary participants,
the dipole is shifted to different voxels.
52
Since the candidate positions for dipole shifts are restricted by the grid of the head model, we cannot shift
the dipole exactly to a predefined distance r. To shift the dipole Ntimes, we select Nclosest to the original
location ρvoxels, that have at least the predefined distance. This results in choosing the voxels that lie on
(or very close to) the sphere with center ρand radius r. After the dipole is shifted to a new voxel position ˜
ρ,
the augmented pattern ˜
aiis determined by the leadfield corresponding to the new voxel. We apply further
minor adjustments to the augmented pattern, including sign correction and normalization (for details, refer
to Appendix A.1).
Figure 38: Generating data with augmented patterns. The measured signals are decomposed with a back-
ward matrix W. One component (here with index i) is selected. As an alternative to the corresponding
pattern ai, an augmented pattern ˜
aiis determined by changing the fitted dipole. This process is detailed in
Figure 37. Mixing the projected signals of the augmented pattern with the projected signals of the remaining
components yields the generated data. The regeneration of the original signals (in the upper right) is just
included in the figure for illustration of the method. Note, that also dipoles of several components can be
altered, see text.
Data generation. In the final step, the augmented pattern is used to project the source component.
It is summed up with the rest (non-augmented) projected components to generate artificial EEG data (see
Figure 38):
˜
x(t)=s′
1(t)a1+s′
2(t)a2+···+yi(t)˜
ai+···+s′
p(t)ap,
while original data can be reconstructed with non-augmented patterns:
x(t)=s′
1(t)a1+s′
2(t)a2+···+s′
i(t)ai+···+s′
p(t)ap.
6.2.2 DA for within-subject classification
We also propose a data augmentation method that considers trial-to-trial variability within one-subject
data. Classification of the MI data relies on learning the differences in corresponding motor-related signals.
However, measured EEG signals also contain a lot of unrelated oscillations caused by noise or artifacts (see
Section 3.1.1). To enhance the robustness of a classifier against noise, we propose to recombine the motor-
related and noise-related components from different trials within each class. In addition, as one-subject
53
datasets usually contain fewer trials compared to multi-subject datasets, we expect that the classifier would
benefit from the increased amount of data after implementing the DA method.
To be more precise, we apply source decomposition to the original trials, localize the dipoles of the cur-
rent sources, identify motor-related components, and recombine them with rest components (presumed to
represent noise) from other trials.
Source decomposition and localization. Similar to the the previous method, we apply SSD to decom-
pose EEG signals into source components S′and obtain the corresponding pattern matrix A:
A=¡a1a2... ai... ap¢.
For each pattern ai(column of A) we fit a dipole with the MUSIC algorithm and obtain the corresponding
location ρi.
Identifying motor-related components. By specifying the parameters of the SSD algorithm, in partic-
ular, by setting the central band frequency to 8 −13 Hz, we expect that some of the strongest SSD compo-
nents will be associated with MI. In this case, the simplest strategy would be assigning the first component
as motor-related and the remaining components as noise-related. However, the analysis of the component
source locations (see Section 7.3) indicates that for certain subjects, the first several components identified
by the SSD algorithm exhibit visual-related characteristics (with a fitted dipole located in the visual cor-
tex) while the MI component appears later. This is common in MI EEG data because the frequency ranges
of visual and sensorimotor rhythms often overlap or completely coincide in many subjects [Blankertz et al.,
2008b]. As a consequence, we propose an alternative strategy that involves defining a region of interest (ROI)
in the motor cortex where the specific MI-related activity is anticipated. The mSSD components originated
in the predefined ROI are assigned as MI-related, while the remaining lcomponents, (m+l=p) are assigned
as noise-related, with the corresponding motor Amotor ∈Rc×mand noise Anoi se ∈Rc×lpatterns, respectively.
In order to determine the optimal strategy and number of components for our experiment, we per-
formed a preliminary study using a limited amount of data. The findings of this study, provided in Ap-
pendix A.2, revealed that selecting the three strongest SSD components (m=3) yielded the best results.
Consequently, all the results presented further in Section 7.2 were obtained using this particular strategy.
Recombination and data generation. Finally, we choose two original trials Aand Bfrom the training
dataset and recombine their mmotor-related and lnoise-related components (see Figure 39). The EEG
values for artificial trials ˜
x1(t) and ˜
x2(t) are calculated as follows:
˜
x1(t)=s′
A1(t)a1+···+s′
Am (t)am
|{z }
motor components A
+s′
B(m+1)(t)am+1+···+s′
B(m+l)(t)am+l
|{z }
noise components B
,
˜
x2(t)=s′
B1(t)a1+···+s′
B m (t)am
|{z }
motor components B
+s′
A(m+1)(t)am+1+···+s′
A(m+l)(t)am+l
|{z }
noise components A
.
For each original trial, motor and noise components can be recombined with multiple other original tri-
als (see details in Section 6.4.2).
6.3 Classification
As outlined in Section 5.3, the impact of data augmentation on different classification models can vary.
Therefore, in order to assess the efficacy of our method, we employ several classifiers to evaluate its per-
formance.
6.3.1 Preprocessing
We apply 19 Laplacian (spatial) filters (see Section 3.1.2) and a band-pass filter of 8 to 13 Hz to the EEG signals
to obtain more focal signals. In addition, we extract epochs in a time interval of 3.5 s duration starting 1 s
after the visual cue from each trial.
54
Figure 39: Generating data with recombination of motor- and noise-related components. Spatial filters W
and spatial patterns Acorresponding to the components that represent both noise and MI task are selected.
For the original trials Aand B, the motor and noise components are projected separately. New artificial trials
are obtained by the recombination of motor and noise signals from different trials.
6.3.2 Linear Discriminant Analysis (LDA)
For the classification with LDA (see Section 4.1.1), we extract band-power features by calculating the vari-
ance within each trial and applying the logarithm. This results in 19-dimensional features. As we described
in Section 3.2.2, logarithm transformation spreads the resulted features, shifts their distribution closer to
Gaussian, and makes covariance matrices of the classes more similar, which makes this feature extraction
method well-suited for LDA.
Laplacian spatial filters demonstrate strong discriminative power in motor imagery, although CSP typi-
cally outperforms them and is widely accepted as a standard approach [Blankertz et al., 2008b]. The reason
for utilizing Laplacian filters is due to the challenges of learning CSP for cross-subject classification, where
labeled data for the target subject is unavailable, requiring additional pooled or ensemble design proce-
dures [Lotte, 2015]. A detailed explanation of this issue is provided in Section 8.2.2.
Besides the plain LDA, we also use LDA with shrinkage (see Section 4.1.1) of the covariance matrix [Vi-
daurre et al., 2009, Blankertz et al., 2011]. We apply the Ledoit-Wolf estimator [Ledoit and Wolf, 2004] of the
shrinkage parameter using the method presented in [Schäfer and Strimmer, 2005]. We refer to LDA classifier
with shrinkage as sLDA.
6.3.3 Network architectures
We employ two convolutional neural networks, introduced in [Schirrmeister et al., 2017], to evaluate the
proposed method. We refer to them as DeepNN and ShallowNN. We apply both of them to the preprocessed
epochs as explained above, see Section 6.3.1.
DeepNN. The deep neural network contains four blocks of convolution and max pooling. The first
block is split into temporal and spatial convolutions, which was shown to better handle the 2D-array input
of EEG data, and lead to a better performance [Schirrmeister et al., 2017]. This block is followed by three
55
Figure 40: DeepNN architecture (adapted from [Schirrmeister et al., 2017]). The trial with dimensions 19 ×
351 is processed with temporal and spatial convolutions in block 1, followed by standard convolutions in
blocks 2 to 4. In all four blocks, max pooling and ELU activation functions are applied. Subsequently, 2
softmax units are used for the binary classification.
standard convolution-max-pooling blocks and a dense softmax classification layer. The exponential linear
units (ELUs) are used as activation functions.
The details of the architecture can be found in the original paper [Schirrmeister et al., 2017]. We had to
decrease the length of the fourth convolutional filter (from 10 to 3) and increase the stride of the last max
pooling layer (from 3 to 4) to adjust for the length of trials in our dataset and use only two softmax units for
the binary classification (see Figure 40).
ShallowNN. The first two layers of the shallow neural network (see Figure 41) are the temporal and spa-
tial convolutions, as in the DeepNN. After that, a squaring nonlinearity, a mean pooling, and a logarithmic
activation function are applied. These steps imitate the log-variance computation, which is used, for exam-
ple, in CSP-based classification [Blankertz et al., 2008b], and which we use in our LDA classification. Finally,
the dense softmax classification layer is applied.
56
Figure 41: ShallowNN architecture (adapted from [Schirrmeister et al., 2017]). After temporal and spatial
convolutions of the ShallowNN, a squaring nonlinearity, a mean pooling layer and a logarithmic activation
function follow. The output is classified using two softmax units.
The negative log-likelihood loss function was optimized with SGD optimizer to train the neural networks.
In Section 7.1, we report the results for cross-subject classification with the models with initially chosen hy-
perparameters, i.e., learning rate, batch size, and weight decay (we refer to these models as iDeepNN and
iShallowNN), as well as for the models with optimized values found with a grid search (oDeepNN and oShal-
lowNN). In particular, the learning rate, batch size, and weight decay parameters for the DeepNN are (0.005,
64, 0.1) (iDeepNN) and (0.005, 32, 0.1) (oDeepNN), and for the ShallowNN are (0.0006, 64, 0.1) (iShallowNN)
and (0.0015, 64, 0.1) (oShallowNN). In Section 7.2, for within-subject classification, we employ only optimal
variants, which we refer to as ShallowNN and DeepNN.
6.4 Evaluation
We evaluate the two proposed data augmentation methods on the motor imagery dataset described in Sec-
tion 6.1 for cross-subject and within-subject classification.
Our first method is applied to cross-subject classification. That means, for probing the classification
performance for a particular participant, we trained a classifier only on the data of the other participants. We
investigate the effect of data augmentation in three scenarios. In the first scenario, we consider a relatively
small number of trials per subject, denoted as k, while in the second scenario, the number of subjects,
denoted as n, is limited. In the third scenario, we consider limiting both kand n.
Our second data augmentation method is employed for within-subject classification. Here, the classifier
is trained to classify trials from a specific participant using their own previous trials as training data. We con-
sider a single scenario and vary the number of original trials, denoted as k, for classification and subsequent
data augmentation.
The performance of multiple classification models (see Section 6.3) is investigated. They are trained
on the original data before DA, and on the increased training dataset after the proposed DA methods are
applied. A paired t-test is applied to compare the average classification accuracy (percent correct) before
57
and after data augmentation. The normality of the measured differences is checked with a Shapiro-Wilk
test.
6.4.1 Cross-subject classification
In the cross-subject classification, we implement the leave-one-subject-out technique. The data of one sub-
ject is left out and used only to test the classification performance. From the remained data, koriginal trials
from ndifferent subjects are selected for training. Thus, the training dataset contains k·ntrials.
Then we consider the number of modifications N.N=1 means that only the original data is used. For
N>1, each component is modified in N−1 different ways, as described in section 6.2.1. Accordingly, for
each available participant, data of N−1 imaginary participants are generated. And the augmented dataset
has Ntimes the size of the original dataset. A small exploration suggested that N=10 is a good choice,
consequently, we adopted this value for subsequent experiments.
Few-trial scenario. In the first scenario, we limit the number of original trials kper subject and com-
pare two settings. First, we train the classifier on koriginal trials of 17 participants and test on 140 original
trials of the target participant. Second, for all 17 training participants, we add k(N−1) generated trials to
the training dataset, while still testing on 140 original trials of the target participant. Better accuracy in the
second setting indicates that adding generated data improves classification performance.
Few-subject scenario. In the second scenario, instead of limiting the number of trials per subject, we
reduce the number of participants nused for training. Initially, the classifier is trained on all 140 trials
from nsubjects (where n<17) and tested on 140 trials of the target subject. Following the augmentation,
additional 140(N−1) artificial trials are included for each of the nparticipants, resulting in an expanded
training dataset, which is used for the classification. The learned model is then tested on the original 140
trials of the target subject.
Mixed scenario. Finally, we test our DA method in the mixed scenario. In this case, both the number of
the participants nand the number of trials per participant kare limited. Similarly to the previous scenarios,
we conduct classification tasks on test trials using two datasets: the original dataset consisting of n·ktrials
and the augmented dataset containing n·k·Ntrials.
6.4.2 Within-subject classification
In within-subject classification, the classifier is trained individually for each participant using their own
previous trials as training data. To assess the impact of data augmentation, we compare the performance of
the original classification with the results obtained when augmented data is incorporated into the training
dataset.
To augment the data, we follow the approach described in Section 6.2.2, where for each measured EEG
trial, selected SSD components are shuffled with the corresponding components of N−1 other trials. This
process expands the training dataset by a factor of N. Through a preliminary study (refer to Section A.2), we
set the parameter Nto 10. Hence, we proceed with this value for Nin subsequent experiments.
We explore different numbers of original trials, denoted as k, ranging from 20 to 120. For each subject in
the dataset, we select ktrials to train the classifier. Subsequently, the classifier is evaluated on the remain-
ing trials from the same subject. Following this, we augment the same ktrials by incorporating k·(N−1)
augmented trials into the training dataset. The classifier is then retrained using this augmented dataset and
tested on the same set of test data.
6.4.3 Evaluation of generated trials
The primary objective of our data augmentation methods is to enhance the classification accuracy and re-
duce the required amount of measured data for training BCI systems. Hence, the main criterion for evalu-
ating the quality of augmented data is the improvement in classification results. However, we also conduct
additional analyses of the data augmentation methods to assess the quality of generated trials in terms of
how realistic and diverse they are.
Firstly, we investigate whether the localized and shifted dipoles in our first data augmentation method
align with motor-related brain regions associated with motor imagery tasks. Specifically, we visualize the
58
original and shifted dipoles for each subject and compare their positions with the brain area reported in [Hard-
wick et al., 2017] as being active during MI tasks.
Secondly, for both proposed methods, we employ PCA and t-SNE [Maaten and Hinton, 2008] techniques
to project the 19-dimensional feature vectors onto a 2Dspace. We compare the distribution of the original
and augmented data, expecting the augmented data to exhibit a similar distribution to the original data
while also containing diverse samples.
Lastly, we examine whether the augmented data retain the characteristic property of MI data, known
as event-related desynchronization (see Section 2.3.1). To accomplish this, we plot PSD of the original and
augmented data in channels C3 and C4.
59
7 Physiology-informed data augmentation. Results
In this chapter, we present results for cross-subject classification (few trials, few subjects, and mixed scenar-
ios) and within-subject classification. For each scenario, we first present original results, i.e., without aug-
mentation. After that, we report the performance after the augmented data is added to the training dataset.
Our analysis includes the results for LDA (with and without shrinkage), ShallowNN (with initial and optimal
hyperparameters), and DeepNN (with initial and optimal hyperparameters), see Section 6.3. Additionally,
we provide the results of visual evaluation of the augmented trials.
7.1 Cross-subject classification
For the training of a cross-subject classifier, koriginal trials from nsubjects were used. The results for N=1
correspond to baseline classification without data augmentation, while N=10 correspond to the increased
training dataset by 10 times. The results are presented for the few-trial scenario (limited number of trials
per subject k), few-subject scenario (limited number of subjects n), and mixed scenario (both nand kare
limited) as described in Section 6.4.1.
7.1.1 Few-trial scenario
Baseline classification. Figure 42 illustrates the performance of six examined classifiers on the original
data without augmentation. Performance of classifiers improved with an increasing amount of training data.
Among all classifiers, shrinkage LDA demonstrated the highest classification accuracy when the training
data was strongly limited (k<20), whereas deep learning models, particularly oShallowNN, exhibited better
performance when trained on larger datasets (k≥50).
Figure 42: Cross-subject classification results of the models trained on original trials depending on the num-
ber of available training trials per subject, denoted as k. Standard deviation bars are omitted for clarity. See
Table 5 for statistical analyses.
Classification with augmented data. In this section, we present the results of classification after aug-
mented data was added to the original training dataset in comparison to the baseline classification.
Results for the classification with LDA models for k=20 are presented in Figures 43a and 43b. The
average accuracy raised by 2.93% (from 75.74% to 78.67%) and by 2.07% (from 76.56% to 78.63%) for LDA
without and with shrinkage, respectively.
Figures 43c and 43d illustrate classification accuracy for ShallowNN models (k=20). ShallowNN with
initial parameters gained 6.55% average accuracy (from 69.93% to 76.48%), while the performance of the
model with optimal parameters improved by 2.23% (from 76.70% to 78.93%).
60
Finally, we present the outcomes for the DeepNN models for k=20 (see Figures 43e, 43f). The average
classification accuracy with iDeepNN increased by 2.22% (from 75.82% to 78.04%), while oDeepNN exhib-
ited an improvement of 0.96% (from 78.22% to 79.18%).
(a) LDA (b) sLDA
(c) iShallowNN (d) oShallowNN
(e) iDeepNN (f) oDeepNN
Figure 43: Classification accuracy (percent correct) with different classification models when 20 original tri-
als per subject were available (k=20). Results for N=1 refer to classification without data augmentation.
Classification was done with participant-independent classifiers using a leave-one-participant-out valida-
tion scheme. Participants on the x-axis are sorted separately for each classifier regarding to N=1 results.
In Figure 44, classification results before and after data augmentation depending on kare presented
separately for LDA, ShallowNN, and DeepNN models. LDA benefited from data augmentation when low
amount of data was available, while classification accuracy with ShallowNN and DeepNN also improved at
bigger values of k. In particular, LDA models gained accuracy when k<40, while no significant improve-
ment was observed at k≥40. For ShallowNN, data augmentation was beneficial not just at small values
of k, but also when a substantial amount of original data was available. However, the biggest boost was ob-
61
Table 5: Changes in classification accuracy after data augmentation in few-trial scenario
LDA sLDA iShallowNN oShallowNN iDeepNN oDeepNN
k=5
original 70.85 74.07 58.26 63.08 61.67 64.63
augmented 73.15 74.15 64.18 65.93 71.07 70.93
difference +2.30** +0.08 +5.92*** +2.85*** +9.40*** +6.30***
k=20
original 75.74 76.56 69.93 76.70 75.82 78.22
augmented 78.67 78.63 76.48 78.93 78.04 79.18
difference +2.93*** +2.07*** +6.55*** +2.23** +2.22 +0.96
k=90
original 78.89 78.85 80.26 81.59 76.74 79.11
augmented 78.59 78.63 82.11 82.78 79.85 81.15
difference −0.30 −0.22 +1.85*** +1.19** +3.11** +2.04*
k=120
original 79.04 79.11 80.30 82.59 78.93 80.07
augmented 79.22 79.22 82.22 83.07 80.04 81.44
difference +0.18 +0.11 +1.92*** +0.48 +1.11 +1.37
paired t-test: ***p<0.01, **p<0.05, *p<0.1
served in cases of limited amount of data. Similarly, DA enhanced the performance of DeepNN models at
low and high values of k(refer to Figure 44 and Table 5). In Table 5, the detailed results for selected values
of k=5,20,90,120 are presented. More plots and detailed results for all investigated values of kare reported
in Appendix B.1.
(a) LDA (b) ShallowNN (c) DeepNN
Figure 44: Classification results with different models trained on the original and augmented datasets, de-
pending on k. Standard deviation bars are omitted for clarity. See Table 5 and Appendix B.1 for statistical
analyses.
7.1.2 Few-subject scenario
Baseline classification. The results for baseline classification with six classifiers are presented in Fig-
ure 45. oDeepNN achieved the highest accuracy when low number of subjects were available (n=3 and n=
5), whereas oShallowNN model outperformed other classifiers at n=10 and n=17.
Classification with augmented data. We provide the results of classification following the addition of
augmented data to the original training dataset, in comparison to the baseline classification. Figure 46b
depicts the results for n=5. Detailed results for n=3,5,10,17 are presented in Table 6. Corresponding plots
for all investigated values of nare reported in Appendix B.2.
62
Figure 45: Cross-subject classification results of the models trained on original data depending on the num-
ber of available training subjects, denoted as n. Standard deviation bars are omitted for clarity. See Table 6
for statistical analyses.
Figures 46a and 46b present the results for the classification using LDA models. The average accuracy in-
creased by 2.81% (from 70.15% to 72.96%) for LDA without shrinkage, and by 1.67% (from 71.33% to 73.00%)
for LDA with shrinkage.
Figures 46c and 46d depict classification accuracy for ShallowNN models. The iShallowNN model demon-
strated a significant improvement in accuracy, gaining 4.00% (from 68.07% to 72.07%), while the perfor-
mance of the model with optimal parameters gained 1.70% (from 73.11% to 74.81%).
We also present the results obtained from the DeepNN models, as shown in Figures 46e, 46f. The av-
erage classification accuracy with iDeepNN increased by 4.78% (from 71.89% to 76.67%), while oDeepNN
exhibited an improvement of 3.04% (from 75.33% to 78.37%).
The aggregated results for all investigated values of nare also presented in Table 6 and Figure 47. LDA
models exhibited a significant gain in classification accuracy only at n=5. In contrast, ShallowNN and
DeepNN models benefited from augmented data in all investigated cases, although the observed improve-
ment decreased when the training dataset contained data from all 17 subjects.
7.1.3 Mixed scenario
Baseline classification. Results for the baseline classification in the mixed scenario (both parameters n
and kare limited) are presented in Figure 48. The classification accuracy for all classifiers increased with n
and k. Compared to the other approaches, LDA with shrinkage learned the best from small amount of
training samples, while deep learning models outperformed LDA classifiers when nand kwere bigger.
Classification with augmented data. Figure 49 illustrates how different classifiers benefited from data
augmentation. Accuracy gain values are plotted for various combinations of parameters nand k.
For extremely small datasets with limited observations (k=5 and n=3), none of the models demon-
strated significant improvements from data augmentation. However, as the values of kand nincreased, clas-
sifiers (each with varying amounts of original data) began to experience substantial enhancements, which
then gradually diminished as more measured data was utilized. Each classifier reached its peak improve-
ment at different points. For instance, LDA exhibited the most significant improvement when the data was
extremely limited (k=10 and n=5), while oShallowNN demonstrated the biggest enhancement with larger
amounts of available data (k=60 and n=5).
More complex DeepNN models gained up to 17.5% accuracy, while simpler LDA and ShallowNN mod-
els improved by 8.45% and 7.19% at most. In addition, suboptimal models (LDA, iShallowNN, iDeepNN)
demonstrated a bigger advantage from data augmentation compared to their optimal variants (sLDA, oS-
hallowNN, oDeepNN).
63
(a) LDA (b) sLDA
(c) iShallowNN (d) oShallowNN
(e) iDeepNN (f) oDeepNN
Figure 46: Classification accuracy (percent correct) with different classification models when all original tri-
als from 5 subjects were available (n=5). Results for N=1 refer to classification without data augmentation.
For each target subject, n=5 training subjects are selected randomly (same selection for each classifier).
Participants on the x-axis are sorted separately for each classifier regarding to N=1 results.
The baseline performance and the impact of data augmentation depended not only on the overall num-
ber of original trials n·k. Specifically, LDA models exhibited lower classification accuracy when a signifi-
cant portion of the available trials, even if numerous, originated from only a few participants (lower nwith
higher kcompared to higher nwith lower k). In these scenarios, it is noteworthy that LDA models exhib-
ited a greater advantage from the data augmentation, consequently yielding a more substantial increase in
accuracy. For more details, please refer to Appendix B.3.
64
(a) LDA (b) ShallowNN (c) DeepNN
Figure 47: Classification results with different models trained on the original and augmented datasets, de-
pending on n. Standard deviation bars are omitted for clarity. See Table 6 for statistical analyses.
Table 6: Changes in classification accuracy after data augmentation in few-subject scenario
LDA sLDA iShallowNN oShallowNN iDeepNN oDeepNN
n=3
original 70.19 70.96 64.33 67.67 69.59 74.74
augmented 68.59 68.85 67.59 69.93 72.89 76.81
difference −1.60 −2.11 +3.26** +2.26*+3.30*+2.07
n=5
original 70.15 71.33 68.07 73.11 71.89 75.33
augmented 72.96 73.00 72.07 74.81 76.67 78.37
difference +2.81*+1.67 +4.00*** +1.70*+4.78*** +3.04**
n=10
original 78.19 78.26 77.22 80.52 74.70 78.56
augmented 78.19 78.22 80.18 81.56 80.04 81.48
difference +0.00 −0.04 +2.96*** +1.04*+5.34*** +2.92
n=17
original 79.15 79.07 82.37 83.74 80.18 81.15
augmented 79.19 79.19 83.18 84.26 80.89 82.78
difference +0.04 +0.12 +0.81 +0.52 +0.71 +1.63
***p<0.01, **p<0.05, *p<0.1
65
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
54
60
66
72
78
84
classification accuracy, %
(a) LDA
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
54
60
66
72
78
84
classification accuracy, %
(b) sLDA
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
54
60
66
72
78
84
classification accuracy, %
(c) iShallowNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
54
60
66
72
78
84
classification accuracy, %
(d) oShallowNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
54
60
66
72
78
84
classification accuracy, %
(e) iDeepNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
54
60
66
72
78
84
classification accuracy, %
(f) oDeepNN
Figure 48: Baseline classification accuracy (percent correct) with different classification models in the mixed
scenario (when both parameters nand kwere limited).
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
* * * * *
****** *********
* * * *
* * * *
0.0
1.5
3.0
4.5
6.0
7.5
accuracy gain, %
(a) LDA
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
*******
*
0.0
1.5
3.0
4.5
6.0
7.5
accuracy gain, %
(b) sLDA
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
* ***** * * * *** *
*********** * ***
******** * * ****
********* ******
0.0
1.5
3.0
4.5
6.0
accuracy gain, %
(c) iShallowNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
* * ** * *
* * * * * * *
* * * * *
*** * * * *
0.0
1.5
3.0
4.5
6.0
accuracy gain, %
(d) oShallowNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
******* * *******
******* * **** ***
****** * * * **** ***
****** * * * * *
0.0
2.5
5.0
7.5
10.0
12.5
15.0
accuracy gain, %
(e) iDeepNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
** *** * * * *
*** * * * * * * * *
* * * * * * * *
* *
0.0
2.5
5.0
7.5
10.0
12.5
15.0
accuracy gain, %
(f) oDeepNN
Figure 49: Accuracy gain after adding augmented data to the training dataset in the mixed scenario (when
both parameters nand kwere limited). Note that colorbar limits are different for different models. Negative
gain values, i.e., when the accuracy decreased, are set to zero for clarity. The plots for accuracy losses can be
found in Appendix B.3. The results for combinations of parameters n=17, k=20, and n=5, k=140 have
been already presented in Figure 43 and Figure 46, respectively. *p<0.05
66
7.2 Within-subject classification
Our second data augmentation method, designed for within-subject classification (see Section 6.2.2), was
tested under a single scenario. Within each subject, we varied the number of original trials, denoted as k,
and utilized them for baseline classification. Subsequently, the same trials were augmented and employed
to retrain the classifier. We compared the classification accuracy before and after augmentation to evaluate
the effectiveness of the method.
Baseline classification. Figure 50 presents the baseline classification results for the four investigated
classifiers. When a low number kof measured trials was available, LDA with shrinkage outperformed the
other classifiers. However, at high values of k, both LDA and ShallowNN achieved similar accuracy levels as
shrinkage LDA. The DeepNN model demonstrated significantly worse performance.
Figure 50: Within-subject classification results of the models trained on original trials depending on the
number of available training trials, denoted as k. Standard deviation bars are omitted for clarity. See Table 7
for statistical analyses.
Classification with augmented data. We present the results for within-subject classification after data
augmentation in comparison to the baseline performance. LDA, ShallowNN, and DeepNN models showed
significant enhancements in classification accuracy, while LDA with shrinkage did not benefit from data
augmentation (see Figure 52).
For the three models that benefited from data augmentation, we provide subject-wise results for selected
values of kin Figure 51 and Table 7. Additional plots (Figure 75) and detailed results (Table 9 and Table 10)
for other values of kcan be found in Appendix C.
LDA exhibited its highest improvement in classification when k=20 trials were available, achieving an
accuracy gain of 11.02% (from 60.87% to 71.89%, see Figure 51a). The ShallowNN and DeepNN models
derived the most benefit from data augmentation at k=60 (with a 4.82% accuracy gain, from 75.07% to
79.89%, see Figure 51e) and at k=90 (with an 11.35% accuracy gain, from 72.06% to 83.41%, see Figure 51i),
respectively.
Figure 52 illustrates the change in classification accuracy for each classifier based on the number of avail-
able original trials. Data augmentation significantly improved the performance of the LDA classifier for k
values in the range of 20 to 85. In the case of deep learning models ShallowNN and DeepNN, considerable
gains were observed across all investigated values of k(from k=20 to k=120). While the increase in clas-
sification accuracy gradually diminished with increasing kfor LDA, neural networks exhibited substantial
improvements at larger values of kup to k=120 (with gains of 4.23% and 9.04%, respectively). For detailed
results, please refer to Tables 9 and 10.
Figure 53 provides a comparison of the results among the four classification models after data augmen-
tation. For lower values of k, the LDA model with shrinkage continued to exhibit superior performance.
However, as the value of kincreased, specifically within the range of 90 to 120, data augmentation facili-
tated the ShallowNN and DeepNN classifiers in surpassing the classification accuracy of the shrinkage LDA
model.
67
(a) LDA, k=20 (b) ShallowNN, k=20 (c) DeepNN, k=20
(d) LDA, k=60 (e) ShallowNN, k=60 (f) DeepNN, k=60
(g) LDA, k=90 (h) ShallowNN, k=90 (i) DeepNN, k=90
Figure 51: Classification accuracy (percent correct) with three different classification models for k=20,
k=60, and k=90. Results for N=1 refer to classification without data augmentation. Classification was
done within each participant. Participants on the x-axis are sorted separately for each classifier regarding
to N=1 results.
(a) LDA (b) ShallowNN (c) DeepNN
Figure 52: Classification results with different models trained on the original and augmented datasets, de-
pending on k. Standard deviation bars are omitted for clarity. See Table 7 and Appendix C for statistical
analyses.
68
Table 7: Changes in within-classification accuracy after data augmentation
LDA ShallowNN DeepNN
k=20
original 60.87 68.32 62.69
augmented 71.89 70.02 64.44
difference +11.02*** +1.70*** +1.75**
k=30
original 69.07 70.81 62.98
augmented 75.06 72.78 67.95
difference +5.99*** +1.97*** +4.97***
k=60
original 79.18 75.07 67.81
augmented 80.93 79.89 75.89
difference +1.75** +4.82*** +8.08***
k=90
original 81.28 80.01 72.06
augmented 81.82 83.95 83.41
difference +0.54 +3.94*** +11.35***
k=120
original 82.12 80.87 74.23
augmented 81.22 85.10 83.27
difference −0.90 +4.23*** +9.04***
***p<0.01, **p<0.05, *p<0.1
Figure 53: Within-subject classification results of the models trained on augmented trials depending on the
number of original training trials, denoted as k. Standard deviation bars are omitted for clarity. See Table 7
for statistical analyses.
69
7.3 Evaluation of generated data
In this section, we present the outcomes of the evaluation of augmented data. We begin by providing the
analysis of the source localization and dipole modification steps of our first data augmentation method (Sec-
tion 6.2.1). Following that, we present the 2Ddata visualization and the analysis of spectral characteristics
for both of our data augmentation methods.
Dipole visualization. We analyzed the source localization step of our method. Specifically, our objec-
tive was to examine cross-subject variability in source locations and verify whether shifted dipoles in our
methods were situated within brain areas related to MI, thereby confirming that we augmented data as in-
tended. We visually represented the positions of localized dipoles for different subjects and identified those
associated with motor imagery tasks. Additionally, we provided the locations of the modified dipoles used
for data augmentation.
Figure 54 showcases the positions of localized source dipoles for two selected subjects. For subject 7,
the first two sources were located in the motor area, suggesting their involvement in motor imagery of right
and left-hand movements respectively. The subsequent components exhibited visual characteristics as they
were located in the visual area. In subject 1, the motor-related components appeared as the third and fifth
components. These two subjects serve as representative examples from our dataset. Among the 18 subjects,
half of them (9 out of 18) exhibited motor-related components as the first ones, similar to subject 7, while
the remaining half showcased stronger visual components with motor imagery components appearing later
as in subject 1. In 16 out of 18 subjects, motor components emerged among the first 5. For a visualization of
localized dipoles for all subjects, please refer to Appendix D.1.
Figure 54: Localized dipoles for the first five SSD components in subject 7 and subject 1. In subject 7, the
first two components were linked to motor imagery. In contrast, subject 1 exhibited the first two localized
sources in the visual area, while motor-related components appeared as the third and fifth components.
We also present the source dipole locations corresponding to the strongest SSD components associated
with motor imagery activity for all subjects. These components were selected using manually defined re-
gions of interest, with the boundaries of these regions determined based on reported results [Hardwick et al.,
2017] investigating the activated brain areas during motor imagery (see Appendix D.1). Figure 55 portrays
the dipole locations for all 18 subjects. The dipole locations vary across subjects.
To investigate the step of dipole modification, we performed five shifts (as outlined in Section 6.2.1)
of the dipoles that correspond to identified motor-related components. Subsequently, we visualized the
locations of the modified dipoles, indicated in orange (see Figure 56). The majority of these augmented
70
(a) Right-hand MI (b) Left-hand MI
Figure 55: Localized dipoles corresponding to the motor imagery task of right and left-hand movements for
all 18 subjects.
dipoles aligned with the corresponding regions. However, in several rare cases, dipoles have shifted to the
opposite hemisphere.
(a) Right-hand MI (b) Left-hand MI
Figure 56: Each of the localized dipoles (in blue) was modified 5 times. The positions of the corresponding
modified dipoles (in orange) are presented.
Data visualization in 2D. To examine the distribution of augmented data in comparison to the original
data, we employed visualization techniques to represent EEG trials in a 2Dspace. We expected augmented
data to be distributed similarly to original data and to contain various trials. We utilized two dimension-
ality reduction methods, namely PCA and t-SNE [Maaten and Hinton, 2008], on the 19-dimensional band-
power feature vectors that were used for LDA classification (see Section 6.3.2). The results for three specific
subjects, chosen based on their exhibited visual separability between classes, are depicted in Figures 57
and 58 (DA for cross-subject classification) and in Figures 60 and 61 (DA for within-subject classification).
For our first DA method, the results of the PCA analysis are presented in Figure 57, demonstrating that
both the original and augmented data exhibited similar distributions. Additionally, the data augmentation
technique generated diverse trials. Similar observations can be made when examining the t-SNE mapping,
as depicted in Figure 58.
Figure 57: Original and augmented feature vectors projected onto the first two PCA components for three
selected subjects (results for DA method for cross-subject classification).
The distribution of data for all 18 subjects is demonstrated in Figure 59. When visualized using two
PCA components (see Figure 59a), the augmented trials aligned closely with the original data. Although
not distinctly separated visually, the cluster of trials associated with right-hand MI appeared on the plot
71
Figure 58: t-SNE embedding of original and augmented feature vectors for three selected subjects (results
for DA method for cross-subject classification).
higher than the cluster for left-hand trials. In contrast, t-SNE separated the trials corresponding to different
subjects, resulting in 18 clusters (see Figure 59b).
(a) PCA (b) t-SNE
Figure 59: PCA and t-SNE applied to original and augmented data of all subjects (results for DA method for
cross-subject classification).
We conducted a separate analysis for each subject’s data using our second data augmentation method,
which was designed for within-subject classification. The results of the PCA can be seen in Figure 60, while
Figure 61 illustrates the outcomes for t-SNE. Similar to our first augmentation method, the modified trials
exhibited diversity and closely resembled the original distribution. It is worth noting that within-subject
classification involved a different group of subjects (refer to Section 6.1), meaning that the same subject
indices may correspond to different data compared to cross-subject classification.
Figure 60: Original and augmented feature vectors projected onto the first two PCA components for three
selected subjects (results for DA method for within-subject classification).
72
Figure 61: t-SNE embedding of original and augmented feature vectors for three selected subjects (results
for DA method for within-subject classification).
PSD for original and augmented trials. To assess whether data augmentation maintains event-related
desynchronization (ERD), which is a common characteristic of the MI paradigm (see Section 2.3.1), we cal-
culated the power spectral density (PSD) of the data. We generated PSD plots for two Laplacian channels (C3
and C4) using both the original and augmented data. The data was filtered within the frequency range of 8Hz
to 13Hz. For our first data augmentation method, the results for three selected subjects are shown in Fig-
ure 62. In all three subjects, ERD was observed, which was consistent with the majority of subjects in our
dataset (refer to Appendix D.2). Furthermore, we found that ERD was preserved in the augmented data. In-
terestingly, some subjects, such as subject 9, exhibited noticeable differences in PSD between the original
and augmented data, while for other subjects, like subject 8, the difference was minimal.
Figure 62: PSD of original and augmented data in Laplacian channels C3 and C4 for three selected sub-
jects (results for DA method for cross-subject classification).
The analysis was conducted again using our second data augmentation technique. The results are pre-
sented in Figure 63. ERD was observed in the original data, and it was also preserved in the augmented data.
Notably, the observed differences in PSD between the original and augmented data are visually smaller than
those observed in the analysis of the first data augmentation method (Figure 62).
Figure 63: PSD of original and augmented data in Laplacian channels C3 and C4 for three selected sub-
jects (results for DA method for within-subject classification).
73
8 Discussion
The objective of our study was to enhance motor imagery EEG classification in brain-computer interface
systems, characterized by significant cross-subject and trial-to-trial variability and a limited number of
available measured trials. To address this challenge, we proposed two novel data augmentation methods
that leverage physiological information about the brain, which is novel compared to previous (traditional
and DL-based) approaches. We aimed to investigate the potential of physiology-informed data augmenta-
tion in improving the performance of BCI systems when dealing with challenging EEG data classification.
Our methods were proposed for cross-subject (few trials, few subjects, and mixed scenarios) and for
within-subject classification. The performance was tested on MI EEG dataset containing the data measured
from 18 subjects (cross-subject classification) and 37 subjects (within-subject classification) performing left-
hand and right-hand motor imagery tasks. We investigated the effect of data augmentation on cross-subject
classification accuracy with LDA, iShallowNN, and iDeepNN models, and their corresponding optimal vari-
ants: sLDA, oShallowNN, and oDeepNN, while LDA, sLDA, oShallowNN, and oDeepNN were employed for
within-subject classification (refer to Section 6.3).
The results of cross-subject classification before and after augmentation indicate that the proposed
physiology-informed DA method effectively enhances EEG classification. The performance improves sig-
nificantly across different classifiers and different scenarios, gaining up to 17.48% accuracy. Moreover, the
results suggest that the method can be successfully applied to extremely small, but also to relatively large
training datasets. For most of the classifiers, a significant improvement is achieved across a wide range of
parameters n(number of original subjects) and k(number of original trials per subject) of a training dataset.
Our method for within-subject classification also demonstrates a notable enhancement in classification
accuracy across LDA, ShallowNN, and DeepNN models. Particularly, the method exhibits significant bene-
fits for the LDA model when the number of original trials is limited. Additionally, it proves highly effective
for neural networks across a wide range of values for k, resulting in an accuracy increase of up to 11.35%.
In addition, the analysis of both methods indicates that augmented EEG trials are realistic and diverse.
Modified data exhibits distribution similar to the original data and preserves the expected ERD pattern com-
monly observed in MI EEG. Furthermore, in the first method, a predominant majority of localized and
shifted dipoles are found in brain regions associated with motor imagery, suggesting that source dipoles
are modified as intended according to physiological expectations.
In this chapter, we discuss the obtained results (Section 8.1), as well as limitations of our study associ-
ated with the used dataset, classification models, and particular algorithms employed in our methods (Sec-
tion 8.2). In addition, we outline future research perspectives and make conclusions in Section 8.3.
8.1 Results interpretation
8.1.1 Cross-subject classification
Baseline classification. The results of the baseline classification support the commonly held belief that
LDA, particularly sLDA, is highly effective in EEG classification when the amount of available training data
is extremely limited. However, as the training dataset increases in size, deep learning models achieve com-
parable accuracy and eventually outperform sLDA (refer to Figures 42 and 48).
For instance, comparing sLDA with oDeepNN, when the number of training trials n·kis below 100,
oDeepNN performs worse than sLDA. Between 100 and 250 trials, both sLDA and oDeepNN exhibit similar
performance, with each outperforming the other depending on the specific values of nand k. However,
when more than 250 trials are available, oDeepNN consistently demonstrates superior results. In the few-
subject scenario, oDeepNN surpasses LDA already at a minimum subject count of n=3 (see Figure 45). This
superiority is due to the relatively large training dataset consisting of 140 trials per subject, resulting in a
total of n·k=520 trials.
When a considerable amount of training trials is available (approx. 600 trials or more), oShallowNN con-
sistently outperforms oDeepNN, aligning with the findings reported in [Schirrmeister et al., 2017].
Boost in cross-subject classification after adding augmented trials. The proposed data augmenta-
tion method proves to be a highly effective tool for enhancing cross-subject EEG classification. Improved
performance is observed across all three investigated scenarios for all classification models. Significant im-
provement is achieved for numerous settings from very small to relatively large training datasets for most
74
of the classification models (Figure 49). In particular, results indicate that significant gain can be obtained
when the total number of training trials ranges from 21 (in case of LDA) up to 2040 (in case of iShallowNN).
Effect on simple and complex models. Deep learning methods are widely recognized for their high
data requirements during the training process. Accordingly, we anticipated that more complex models
would derive greater benefits from data augmentation compared to simpler models. Overall, our findings
align with this expectation. Specifically, our results demonstrate that iDeepNN and oDeepNN models ex-
hibit notable improvements of up to 17.48% and 10.0%, respectively. In contrast, the enhancements ob-
served in LDA and ShallowNN models do not exceed 8.5%.
Meanwhile, it is important to note that the extent of improvement is greatly influenced by the specific
scenario, particularly the amount of measured data available. The effect of data augmentation exhibits vari-
ations depending on the number of participants nand the number of trials per participant k.
Generally, classifiers benefit from data augmentation within a specific range of amounts of available
training trials (refer to Figure 49). The upper bound of this range can be explained by the fact that the clas-
sifier already employs sufficient data to learn, and additional augmented trials do not provide new infor-
mation, thus not improving its performance. Regarding the lower bound, we attribute it to the inherent
operation of the data augmentation method itself. Specifically, we assume that steps such as source de-
composition and source localization may produce inaccurate results when the data is extremely limited,
restricting the effectiveness of the data augmentation.
Simple LDA models do not require a lot of training data, in contrast to more complex deep learning clas-
sifiers. Accordingly, LDA models demonstrate significant improvement up to a maximum training dataset
size of 600 trials, whereas deep learning models continue to show improvement as the number of trials in-
creases, reaching up to 2040 trials.
Effect on the optimal and suboptimal models. The results indicate that data augmentation provides
a bigger advantage for suboptimal models, namely LDA, iShallowNN, and iDeepNN, compared to their op-
timal counterparts, including sLDA, oShallowNN, and oDeepNN. Importantly, even when the best model is
selected for a specific scenario, the application of data augmentation techniques enables further enhance-
ments in performance. However, the fact that suboptimal models improve even more has two important
implications. Firstly, it indicates that data augmentation compensates suboptimality of classifiers. Secondly,
this finding highlights an additional challenge in comparing different data augmentation methods.
Data augmentation has the potential to bridge the classification accuracy gap between suboptimal and
optimal models. For instance, in the case of LDA, the omission of shrinkage leads to significantly infe-
rior performance on very small datasets. However, after augmenting the data, both LDA with and without
shrinkage achieve similar performance levels (see Table 5). This compensatory effect of data augmentation
is also observed in deep learning models. This finding demonstrates the ability of data augmentation to ef-
fectively compensate for suboptimal training, enabling models to achieve performance comparable to that
of optimal models.
Bigger gain in suboptimal models also suggests that comparing different data augmentation methods
reported in various articles can be challenging. The effectiveness of these methods depends not only on
factors such as the dataset and classification model, but also on the optimality of the classifier.
Augment or not augment? The observation that optimal models benefit less from data augmentation
raises the question whether data augmentation is necessary. Perhaps we can train the classification model
in an even better way to eliminate the need for data augmentation. Moreover, we know that neural net-
works are able to learn sophisticated functions, so transformations for data augmentation can be potentially
learned by DL classifiers instead of having additional pre-processing step.
However, while neural networks are powerful models capable of learning complex patterns and repre-
sentations from data, they are not inherently aware of the concept of data augmentation. They learn by
adjusting weights and biases based on input data and target outputs in order to minimize a loss function.
Augmented samples commonly incorporate external knowledge about transformations that will increase
the diversity of the dataset. Neural networks do not have built-in mechanisms to automatically generate
such augmented samples or learn the transformations themselves. Thus, augmented data can bring new
information and improve classification accuracy even if the model is optimally trained on the original data.
To automate and integrate data augmentation into deep learning frameworks, researchers have explored
meta learning approaches. Meta learning data augmentation involves training a separate neural network to
75
generate augmented data specific to a given task. For instance, approaches like neural augmentation [Perez
and Wang, 2017] and smart augmentation [Lemley et al., 2017] train a second neural network to combine
two random samples from the dataset and create a new augmented sample. Other techniques, such as Au-
toAugment [Cubuk et al., 2019] and automated image preprocessing [Minh et al., 2021], utilize a reinforce-
ment learning algorithm to search for an optimal augmentation policy from a predefined set of geometric
transformations with varying degrees of distortion. All these approaches, however, involve training an ad-
ditional neural network or using a separate algorithm for data augmentation, and cannot be considered as
incorporated into one NN classification model.
One alternative to data augmentation is incorporating expert knowledge directly into the network archi-
tecture. This can be achieved by imposing structural constraints. For instance, in computer vision, domain
experts can identify relevant visual features for the task, which can then be used to design neural network
architectures that effectively capture those features. Examples include the utilization of convolutional layers
to leverage spatial locality and translation invariance in images, or the incorporation of skip connections or
attention mechanisms in medical image segmentation to ensure alignment with known anatomical struc-
tures.
Furthermore, expert knowledge can be incorporated through additional loss terms or regularization
penalties. For example, a regularization term can enforce solution smoothness in accordance with domain
knowledge. However, it can be challenging to identify appropriate structural constraints or additional loss
terms to provide the model with particular information. In such cases, data augmentation remains the sole
solution.
In conclusion, neural networks lack inherent capacity for learning data augmentation. The integration
of extra knowledge into deep learning frameworks is a challenging task and can be applied only in limited
cases. Therefore, data augmentation remains a powerful tool for improving the performance of classification
models.
8.1.2 Within-subject classification
Baseline classified’s cation. The evaluation results for the baseline classification show that sLDA out-
performs the other classifiers examined. Particularly, when the number of recorded trials is low, sLDA per-
forms exceptionally well, surpassing the second-best classifier by more than 8% at k=20 (refer to Figure 50).
In contrast to the cross-subject classification results, neural networks do not surpass the performance of
the shrinkage LDA model, even at high values of k. This can be attributed to the fact that within-subject clas-
sification has a smaller training dataset due to using only kmeasured trials from a single participant. Addi-
tionally, similar to cross-subject classification, the ShallowNN model shows better results than the DeepNN
model.
Boost in within-subject classification after adding augmented trials. Our proposed data augmenta-
tion method demonstrates a significant enhancement in the classification performance of LDA, ShallowNN,
and DeepNN classifiers. Specifically, the neural network models exhibit a significant increase in classifica-
tion accuracy across all examined values of k. On the other hand, LDA benefits from data augmentation
when the available number of recorded EEG trials is less than 85 (Figure 52).
It is worth noting that LDA with shrinkage does not exhibit notable improvements through data augmen-
tation (Figure 52). This can be attributed to the nature of within-subject classification, which is generally
considered an easier task compared to cross-subject classification due to the reduced variability in the data.
As sLDA is highly effective in learning from limited data, it is likely that it performs well with the original
trials, resulting in no significant improvement after data augmentation.
Importantly, our data augmentation technique enables deep learning models to surpass the perfor-
mance of sLDA when the value of kis equal to or greater than 90 (refer to Figure 53). This implies that
in practical applications, the combination of neural networks with our data augmentation method is more
advantageous in terms of classification accuracy compared to using shrinkage LDA.
Effect on simple and complex models. Our findings imply that data augmentation has a greater impact
on the performance of complex neural network models compared to simpler LDA classifiers. Our experi-
ments indicate that data augmentation consistently improves the performance of neural networks across
different values of k, whereas its benefits are limited for LDA and only observed within a specific range of
data. This outcome aligns with the expectation that complex models generally require more training data,
as discussed in Section 8.1.1.
76
Specifically, LDA demonstrates a significant increase in accuracy when the training data is extremely
limited, such as a notable improvement of 11.02% at k=20. However, the influence of data augmentation
on LDA diminishes as kincreases. At k=85, the improvement is merely 0.75%, and for larger numbers of
available training trials, there is no significant enhancement.
In contrast, deep learning classifiers benefit from data augmentation across all investigated regimes.
Notably, DeepNN exhibits a substantial boost in accuracy, reaching a maximum improvement of 11.35% at
k=90, while ShallowNN achieves its highest improvement of 4.82% at k=60.
8.1.3 Comparison to other methods
Our data augmentation methods improve MI EEG classification results by up to 4.30% −17.48% (cross-
subject classification) and by up to 4.82% −11.35% (within-subject classification) depending on the used
classification model (see Figures 49 and 51). These results outperform reviewed (see Table 3) traditional
methods designed for MI EEG, which allow 1.9% −5.5% improvement. The only reviewed traditional ap-
proach that demonstrates similar results (allows up to 15% improvement) is the segmentation-recombination
method [Lotte, 2015] employed for MI and workload data. Furthermore, our methods demonstrate compa-
rable performance to DL-based MI-EEG data augmentation methods that enhance classification accuracy
by 3.5%−21%. Importantly, unlike deep learning methods, our approaches remain applicable even to very
small training datasets. Moreover, the steps of our methods are understandable from a physiological per-
spective (see Section 8.1.4), which facilitates modifications in contrast to “black-box” deep learning tech-
niques.
Deep generative models, such as GAN-based methods, require a substantial amount of measured data
for effective training. For instance, in the study by [Fahimi et al., 2021], a GAN is trained on 1040 trials from
various subjects within a dataset, supplemented by additional 40 trials for a conditional vector specific to
the target subject. Another study [Panwar et al., 2020] does not explicitly state the number of original trials
used for training, but utilizes 5000 artificial sinusoids for a simulated task, suggesting that a similar amount
of recorded data would be needed for real EEG data. Another approach uses [Luo et al., 2020] at least 1920
original samples for training a deep generative model. Finally, 180 and 648 measured trials per subject for
two different datasets are employed [Zhang et al., 2020] in another study.
These examples highlight the significant data requirements for training deep generative models, which
can pose limitations in practical applications. In contrast, our methods demonstrate significant classifica-
tion improvements even with a minimal number of measured trials, such as 21 (n=3, k=7) and 25 (n=5,
k=5) for cross-subject classification (refer to Figure 49), and k=20 for within-subject classification (refer
to Table 7).
As highlighted in Section 5.3, it is challenging to compare different DA methods since they are tested
on different datasets containing different amounts of measured data, and with different classification mod-
els. Additionally, as discussed earlier in Section 8.1.1, the impact of data augmentation on a specific clas-
sifier is influenced by its optimality, including factors such as the choice of optimal regularization method
or set of hyperparameters. As a result, reported improvements in different articles can vary significantly,
ranging from a few percent to as high as 40% (emotion recognition data) (see Table 3). These variations in
performance are likely influenced by various factors beyond the data augmentation technique itself. This
highlights the importance of standardization when assessing and comparing the performance of data aug-
mentation techniques in future studies.
8.1.4 Realistic augmented data
In addition to enhancing the accuracy of MI EEG classification, our DA methods generate realistic EEG trials
in a biologically plausible manner. The findings presented in Section 7.3 demonstrate that our first method
successfully identifies MI-related sources, localizes corresponding dipoles, and modifies them reasonably.
Moreover, the obtained by both of our methods augmented trials exhibit diversity while maintaining a real-
istic distribution and preserving important MI-related characteristics such as event-related desynchroniza-
tion.
Our first method aims to identify motor-related components in EEG data, localize corresponding dipoles,
and shift them to new locations that may represent sources in other unmeasured subjects. The analysis (see
Figure 54) reveals that MI-related components rank among the strongest SSD components across nearly all
subjects, indicating the effectiveness of the chosen SSD method. Visual-related components also appear
77
among the strongest, which is common in MI EEG data because the frequency ranges of visual and sensori-
motor rhythms often overlap or completely coincide in many subjects [Blankertz et al., 2008b].
The fact that positions of localized dipoles corresponding to MI components differ between subjects (re-
fer to Figure 77), supports our assumption that varying source locations contribute to the substantial cross-
subject variability in the data. After corresponding modifications, the predominant majority of shifted
dipoles are found (Figure 56) in brain regions associated with motor imagery, suggesting the generation
of realistic and diverse EEG data. However, a few modified dipoles are found in the opposite hemisphere.
These instances of shifted sources to the opposite hemisphere may result in data with the opposite label,
such as left hand instead of right hand. Although these occurrences are rare, they could potentially con-
tribute to a decrease in classification accuracy in some subjects.
Visualization of augmented trials in 2Dwith PCA and t-SNE indicates generation of realistic and diverse
samples. Both PCA and t-SNE reveal that the distribution of augmented trials is similar to the original. In
some cases, visual separation of two classes is more prominent in t-SNE than in PCA (compare Figures 57
and 58). However, interpreting the results of the non-linear iterative t-SNE method can be challenging, as
direct inferences from its output are not possible. When visualizing all subjects together, t-SNE clusters the
data based on subjects, while specific PCA components separate the classes (Figure 59). Similar findings
from our second data augmentation (see Figures 60 and 61) method support the conclusion that it also
effectively produces diverse and realistic EEG trials.
The preservation of ERD in the augmented data further supports the generation of natural MI EEG sam-
ples. In all subjects where this typical for MI phenomenon is observed, it persists in the modified trials. For
the first data augmentation method, the power spectral density of the augmented data differs significantly
from the original data in some subjects, while in other subjects this difference is minimal. It can be ex-
plained by the fact that although the dipoles for all subjects are shifted by the same distance, they are shifted
in different directions, potentially affecting the characteristics of the resulting trials differently. In contrast,
these differences are minimal for our second augmentation method. This outcome was anticipated, as this
method involves reshuffling components within a single subject, without introducing additional variability
through the shifting of a source dipole.
These observations collectively indicate that our methods allow producing realistic and various EEG
trials, which can be utilized not only for direct data augmentation but also for other purposes. For instance,
it can be applied to address class-imbalance issues, reconstruct corrupted channels, or synthesize diverse
data for research purposes. In addition, understanding the steps of the method enables its modifications
and improvements, which is not possible in the end-to-end deep learning methods.
8.2 Limitations
8.2.1 Data
The study is subject to various limitations concerning the used dataset. These limitations include a restricted
number of subjects, a focus on motor-imagery classes limited only to left and right-hand movements, and
the specifically chosen motor imagery paradigm.
The restricted number of subjects in the dataset raises concerns regarding potential inherent bias within
the data. Multiple studies [Javaid et al., 2022, Al Zoubi et al., 2018, Cantillo-Negrete et al., 2016, Ujma et al.,
2019, Subirats et al., 2018] investigated the influence of age, gender, intelligence (IQ or education level), and
health status on EEG recordings. These studies consistently show that age has a significant influence on the
measurements [Al Zoubi et al., 2018], while the impact of gender is moderate [Cantillo-Negrete et al., 2016].
Additionally, certain EEG characteristics may vary depending on general intelligence [Luo et al., 2021], while
others remain unaffected [Ujma et al., 2019]. These tendencies have also been observed [Subirats et al.,
2018, Pillette et al., 2021] particularly in motor imagery EEG data. Although EEG is known to be highly
variable due to numerous reasons, these specific factors should be considered by selecting/recording corre-
sponding datasets. The original dataset in this study included participants representing a wide range of ages
and balanced gender representation. However, after limiting the dataset to only 18 subjects (cross-subject
classification) and to 37 subjects (within-subject classification) (see Section 6.1), different groups might not
be well represented. Therefore, future research should test DA methods on a larger and more representative
group.
The scope of our study is limited to left and right-hand MI, while BCI systems utilize more tasks, includ-
ing motor imagery of feet or tongue movements. As discussed in Section 6.1, the MI of a foot can be different
between participants, making the development of participant-independent models more complex. More-
78
over, incorporating additional motor imagery tasks would require use of more regions of interest for source
localization and more careful shifting of the dipoles to prevent unwanted label alterations. Further investi-
gation of the applicability of our methods to other MI tasks is needed.
In addition, our study specifically focuses on the motor imagery paradigm, which is currently the most
investigated [Aggarwal and Chugh, 2022] paradigm in BCIs (see Section 2.3). Our methods are based on
physiological principles that have the potential to be applied to other paradigms as well. The process of
identifying underlying sources and introducing variability to their locations, or combining task-related and
noise-related components, can be adjusted to correspond to different paradigms. However, such adap-
tations would require significant modifications to the algorithmic steps. The conclusion about the appli-
cability of proposed methods to other BCI paradigms can only be made following the conduction of the
corresponding investigation.
8.2.2 Classification models
There are several limitations associated with the classification models used in our study. Our results demon-
strate that the choice of classification model, as well as the quality of its training, significantly impacts the
effectiveness of data augmentation. Therefore, careful consideration of the classification models and their
training details is important.
While our proposed DA methods were applied to LDA and NN classifiers, it is worth noting that there
are other effective classifiers for MI EEG classification (refer to Section 6.3). Ideally, the impact of data aug-
mentation should be investigated using multiple classification models. Specifically, future research should
include the results obtained from Riemannian classifiers, which are also considered as state-of-the-art.
For LDA classification, we extracted features from EEG trials using Laplacian filters. However, the com-
monly accepted standard involves using CSP filters in combination with LDA. Both Laplacian and CSP fil-
ters exhibit strong discriminative power in motor imagery data, but CSP typically outperforms Laplacian
filters [Blankertz et al., 2008b]. Due to the substantial variability across subjects, CSP filters trained for one
subject are usually suboptimal for another subject. Consequently, learning CSP filters in cross-subject clas-
sification, where labeled data for the target subject is unavailable, becomes problematic.
To address this issue, two approaches, namely pooled and ensemble design [Lotte, 2015], have been pro-
posed. In the pooled design, CSP spatial filters are optimized using combined data from multiple users [Lotte
et al., 2009]. In this case, it may be necessary to utilize a frequency band that is broad enough to be relevant
for multiple subjects [Lotte et al., 2009]. In the more advanced and efficient ensemble design, CSP filters
are learned for each available subject individually and then combined to be efficient across subjects. For
instance, it has been proposed [Fazli et al., 2009] to learn a CSP-LDA pair for each subject and to consider
them as potential basis functions. Optimal weighting for the classifier outputs of these basis functions was
determined using quadratic regression with l1regularization, enabling reliable generalization to subjects
that are not included in the ensemble. In our study, we use Laplacian spatial filters, which can be directly
applied to the data without relying on pooled or ensemble design. However, future studies should consider
utilizing CSP filters, which could potentially result in improved classification performance.
We evaluate the effects of the proposed data augmentation methods on both shallow and deep con-
volutional networks. Since the introduction of these methods [Schirrmeister et al., 2017], there have been
advancements in deep learning classifiers for motor imagery EEG classification. Several examples of broadly
used models for MI EEG classification are EEGNet [Lawhern et al., 2018], EEG-TCNet [Ingolfsson et al., 2020],
EEGNeX [Chen et al., 2022], and EEG-ITNet [Salami et al., 2022]. Researchers should consider incorporat-
ing some of these models into their future investigations. Additionally, while we use filtered EEG signals to
compare the results between neural networks and LDA, future studies on data augmentation could explore
the use of raw data for NN models.
8.2.3 Methods
Another set of limitations is connected to the techniques applied in each step of the proposed data augmen-
tation methods. DA methods inherit limitations of these particular techniques for source separation, source
localization, and dipole modification steps, presenting a room for further investigation and improvement in
future research.
The SSD algorithm, which we use for source decomposition, relies on manually specified frequency
bands associated with neuronal activity in the brain. The performance of SSD depends on capturing pro-
nounced peaks of EEG within these specified bands, and can be influenced by the presence of other oscilla-
tions [Nikulin et al., 2011]. Future studies can employ one of the proposed in [Nikulin et al., 2011] strategies
79
for identifying frequencies of interest automatically, or consider alternative source separation methods.
Similarly, the MUSIC algorithm used for source localization has its limitations. It may yield poor accu-
racy when dealing with correlated sources [Mosher and Leahy, 1999]. As discussed in Section 3.3.3, several
variations of MUSIC, including the recursively applied and projected MUSIC (RAP-MUSIC) [Mosher and
Leahy, 1999] and the first principle vector (FINE) localization method [Xu et al., 2004] partially address this
limitation. In the future, incorporating improved versions of the MUSIC algorithm or alternative source
localization methods such as based on minimum norm or beamforming (see Section 3.3.3) might be bene-
ficial.
In our first method, we modify the localized dipoles by shifting their positions by a predefined distance.
However, it is important to note that the orientation of these dipoles also significantly impacts the projected
EEG signals. Consequently, altering dipole orientations could introduce additional variation to the training
dataset and should be considered in future research.
Our second method has additional limitation on the amount of augmented data that can be generated.
Due to the reshuffling of motor-related and noise-related components within the original trials, the number
of possible combinations is restricted. For instance, when k=10, the original dataset consists of only 5 trials
per class, allowing for a maximum increase in data of ⌊k
2⌋−1=4 times. Increasing datasets by 10 times is,
therefore, not possible for k<20. In future research, dividing the noise components into subgroups could
enable a broader range of shuffle combinations with only a limited number of trials.
8.3 Conclusions
This thesis focused on addressing the challenge of significant cross-subject and trial-to-trail variability in
motor imagery data, which often leads to poor performance of brain-computer interface systems. To tackle
this issue, we proposed two novel data augmentation methods based on physiological information about
the brain. These methods aimed to introduce variability into the training dataset and enhance classification
accuracy in motor imagery EEG data using LDA and neural network classifiers.
The results demonstrated the effectiveness of our proposed methods, surpassing traditional data aug-
mentation techniques and showing comparable performance to deep learning approaches. Notably, our
methods led to significant improvements in classification accuracy across various scenarios, ranging from
extremely limited to relatively large amounts of training data. Apart from enhancing classification accuracy
in BCI applications, these data augmentation methods also have potential for suppressing calibration time.
The significant performance enhancement in cross-subject classification (where no trials from the target
subject are used) suggests the feasibility of calibration-free BCI systems, enabling faster and more conve-
nient use of BCIs. Additionally, clear physiology-informed steps of our methods provide opportunities for
further modifications and improvements based on additional expert knowledge.
Furthermore, the generation of realistic and diverse EEG trials through our methods opens possibilities
for addressing EEG channel reconstruction, class imbalance problem, and simulation of artificial data for
research purposes. For example, our first approach can be adapted to recover corrupted or missing EEG
channels. This can be achieved by identifying source signals from the measured channels and projecting
them onto new channels using a head model. Both methods can also help tackle EEG class imbalance prob-
lem. To increase the available data for the class with fewer recorded trials, data augmentation can be applied
specifically to that class.
We investigated how different classifiers benefit from our methods based on their complexity and op-
timality. We found that more complex models exhibited a higher degree of performance enhancement,
thereby presenting further potential of deep neural networks to improve classification accuracy in EEG clas-
sification. Moreover, suboptimal methods showed bigger improvements, indicating the ability of DA meth-
ods to narrow the gap between optimal and suboptimal classifiers. This also highlighted the challenge of
comparing different data augmentation methods, which should be addressed in future research. Optimal
classifiers gained smaller yet significant improvement, demonstrating that our methods provide valuable
additional information and enhance the performance of well-trained models.
While our methods showed promising results, there is room for improvement by employing more so-
phisticated techniques for source decomposition and source localization, or introducing further variation
into the dataset through dipole rotations. Future research should explore additional possibilities for incor-
porating physiological knowledge into data augmentation methods. Furthermore, the applicability of our
methods should be investigated for different motor imagery tasks and BCI paradigms. Finally, the impact of
data augmentation on a broader range of state-of-the-art traditional classifiers, Riemannian classifiers, and
neural network models should be investigated.
80
Appendices
A Methods implementation
A.1 Dipole shift
As we describe in Section 6.2.1, after the dipole is shifted to a new voxel position, the augmented pattern ˜
ai
is determined by the leadfield corresponding to the new voxel. After that, we apply further adjustments to
this pattern.
Since MUSIC does not consider sign of patterns, we change (if needed) the sign of the augmented pat-
tern, so that aiand ˜
aihave the same sign. We also normalize the augmented pattern, so its norm is equal to
the norm of the original pattern: ∥ai∥=∥˜
ai∥.
However, we observed that despite having the same norm, sign, and highly correlated values, certain
channels in the original and augmented patterns exhibit significant differences. This discrepancy arises
not from the dipole shift itself, but from errors introduced by the head model and the source localization
algorithm.
To address this issue, we introduce a channel-wise adjustment for the augmented pattern (see Figure 64).
For a specific channel, we have the value of the original SSD pattern ai, value for the pattern ai
l oc ali zed
corresponding to the localized dipole, and value for the pattern ai
shi f t ed corresponding to the shifted dipole.
Instead of using the values associated with the shifted dipole for the augmented pattern, we employ adjusted
values for each channel, given by ˜
ai=aiai
shi f t ed
ai
l oc al i zed
. This adjustment allows us to apply the shift to the original
pattern in the form of the ratio ai
shi f t ed
ai
l oc al i zed
and eliminate source localization error.
aiai
l oc ali zed ai
shi f t ed ˜
ai
Figure 64: Instead of using the shifted pattern ai
shi f t ed for data augmentation, further adjustments are em-
ployed in order to eliminate source localization error (see text), which results in the pattern ˜
ai. The patterns
are plotted for the strongest SSD component of subject 7. The position of the localized dipole for this com-
ponent is presented in Figure 54. All four patterns have the same norm (equal to 20.7) and sign. However, the
localized and shifted patterns differ from the original SSD pattern much more than the resulted augmented
pattern: ∥ai−ai
l oc ali zed ∥ = 203.4, ∥ai−ai
shi f ted ∥ = 198.6, ∥ai−˜
ai∥ = 5.3.
A.2 Choosing a particular strategy to shuffle components
We conducted a preliminary study with a limited amount of data to determine the optimal strategy and
number of components for our experiment. Specifically, we compared the within-subject classification ac-
curacy before and after data augmentation, as explained in Section 6.4.2, for various numbers of original
trials k. We tested the following values of k: 20, 40, 60, 80, 100, and 120. The classifiers used in the study
were LDA, ShallowNN, and DeepNN. We calculated the average difference between augmented and original
classification results for two strategies: shuffling the strongest SSD components and shuffling components
selected based on ROI (see Section 6.2.2). Additionally, we considered shuffling a different number of com-
ponents: m=1, m=2, and m=3.
For the strategy of selecting the strongest components, the average difference between augmented and
original results was found to be 1.80±0.36 (mean ±standard error of the mean (SEM)). On the other hand,
for selecting components based on ROI, the average difference was 1.27±0.36. The results for m=1, m=2,
81
and m=3 components were 1.09 ±0.46, 1.68 ±0.44, and 1.84 ±0.43, respectively. Based on these findings,
we chose the strategy of shuffling the three strongest SSD components.
Furthermore, we investigated the effect of adding different amounts of augmented trials. Doubling the
dataset size (N=2) resulted in an accuracy increase of 0.39±0.23. Increasing the dataset size by N=5 times
yielded an accuracy increase of 1.84 ±0.40, and for N=10, the increase was 2.38 ±0.52. Consequently, we
selected N=10 for further studies.
82
B Cross-subject classification
In this section, we present detailed results for cross-subject classification before and after data augmenta-
tion. Classification models are trained on koriginal trials per subject from nsubjects and tested on one
target subject. In few-trial scenario, data from all subjects (n=17) is used, but kis limited. In contrast,
few-subject scenario employs all available trials (k=140) from a limited number of subjects n. The results
on the N-time-bigger augmented dataset are compared to the results on the original (N=1) data.
B.1 Few-trial scenario
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 65: Effect of data augmentation on different classification models for k=5.
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 66: Effect of data augmentation on different classification models for k=20.
Figures 65 to 68 illustrate classification results before and after data augmentation in few-trial scenario
for k=5, k=20, k=90, and k=120. Table 8 contains detailed results for all investigated values of k.
83
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 67: Effect of data augmentation on different classification models for k=90.
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 68: Effect of data augmentation on different classification models for k=120.
84
Table 8: Effect of data augmentation on classification accuracy
LDA sLDA iShallowNN oShallowNN iDeepNN oDeepNN
k=5
original 70.85 74.07 58.26 63.08 61.67 64.63
augmented 73.15 74.15 64.18 65.93 71.07 70.93
difference +2.30** +0.08 +5.92*** +2.85*** +9.40*** +6.30***
k=7
original 72.67 74.78 61.11 65.93 55.93 69.04
augmented 74.89 75.07 68.30 69.81 73.41 74.93
difference +2.22 +0.29 +7.19*** +3.88*** +17.48*** +5.89***
k=10
original 75.56 77.22 63.30 68.26 64.15 73.07
augmented 76.59 76.81 67.93 70.63 73.74 74.93
difference +1.03 −0.41 +4.63*** +2.37** +9.59*** +1.86
k=12
original 76.52 77.67 66.74 72.04 68.15 73.70
augmented 78.26 78.67 69.22 71.00 74.67 76.07
difference +1.74** +1.00 +2.48** −1.04 +6.52*** +2.37
k=15
original 75.37 76.78 66.00 71.85 62.78 75.85
augmented 77.44 77.48 72.18 73.85 75.48 76.59
difference +2.07** +0.70 +6.18*** +2.00*+12.70*** +0.74
k=20
original 75.74 76.56 69.93 76.70 75.82 78.22
augmented 78.67 78.63 76.48 78.93 78.04 79.18
difference +2.93*** +2.07*** +6.55*** +2.23** +2.22 +0.96
k=25
original 76.67 77.19 68.74 74.33 74.30 78.37
augmented 76.74 76.81 71.82 76.33 78.89 78.89
difference +0.07 −0.38 +3.08** +2.00** +4.59*** +0.52
k=30
original 77.00 77.81 70.52 76.30 76.59 79.33
augmented 77.37 77.41 74.26 76.78 78.67 79.89
difference +0.37 −0.40 +3.74*** +0.48 +2.08 +0.56
k=40
original 79.33 79.56 74.00 77.44 76.67 80.48
augmented 79.74 79.67 76.52 78.93 79.67 81.00
difference +0.41 +0.11 +2.52** +1.49 +3.00** +0.52
k=50
original 78.81 78.96 76.30 79.85 78.63 79.07
augmented 78.19 78.15 77.67 80.11 79.74 81.22
difference −0.62 −0.81 +1.37*+0.26 +1.11 +2.15
k=60
original 78.93 78.93 76.96 79.67 74.04 79.18
augmented 78.67 78.70 79.26 80.59 78.56 80.15
difference −0.26 −0.23 +2.30*** +0.92 +4.52** +0.97
k=70
original 78.56 78.74 78.26 80.78 75.67 79.33
augmented 79.33 79.37 80.48 82.18 78.70 79.52
difference +0.77 +0.63 +2.22** +1.40** +3.03 +0.19
k=80
original 78.67 78.67 79.70 81.89 78.89 79.30
augmented 77.44 77.48 82.44 82.52 81.22 80.70
difference −1.23 −1.19 +2.74*** +0.63 +2.33*+1.40
k=90
original 78.89 78.85 80.26 81.59 76.74 79.11
augmented 78.59 78.63 82.11 82.78 79.85 81.15
difference −0.30 −0.22 +1.85*** +1.19** +3.11** +2.04*
k=100
original 78.89 78.93 80.78 82.52 79.00 81.18
augmented 78.78 78.85 82.33 83.00 81.33 82.85
difference −0.11 −0.08 +1.55** +0.48 +2.33 +1.67
k=120
original 79.04 79.11 80.30 82.59 78.93 80.07
augmented 79.22 79.22 82.22 83.07 80.04 81.44
difference +0.18 +0.11 +1.92*** +0.48 +1.11 +1.37
k=140
original 79.15 79.07 82.37 83.74 80.18 81.15
augmented 79.19 79.19 83.18 84.26 80.89 82.78
difference +0.04 +0.12 +0.81 +0.52 +0.71 +1.63
***p<0.01, **p<0.05, *p<0.1 85
B.2 Few-subject scenario
Figures 69 to 72 illustrate classification results before and after data augmentation in few-subject scenario
for n=3, n=5, n=10, and n=17. Detailed results are presented in Table 6.
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 69: Effect of data augmentation on different classification models for n=3.
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 70: Effect of data augmentation on different classification models for n=5.
86
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 71: Effect of data augmentation on different classification models for n=10.
(a) LDA (b) iShallowNN (c) iDeepNN
(d) sLDA (e) oShallowNN (f) oDeepNN
Figure 72: Effect of data augmentation on different classification models for n=17.
87
B.3 Mixed scenario
Figure 73 represents instances where the accuracy of classification declined following the implementation of
data augmentation. The majority of these instances took place during classification using sLDA, particularly
in situations with a limited amount of training data. We believe that it happened due to the fact that sLDA
is recognized for its high efficiency in learning from very small data. Consequently, the introduction of data
augmentation does not provide advantages in these particular scenarios.
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
0
2
4
6
8
accuracy loss, %
(a) LDA
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
0
2
4
6
8
accuracy loss, %
(b) sLDA
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
0
2
4
6
8
accuracy loss, %
(c) iShallowNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
0
2
4
6
8
accuracy loss, %
(d) oShallowNN
(e) iDeepNN
5 7 10 12 15 20 25 30 40 50 60 70 80 90 100 120 140
trials per subject, k
351017
subjects, n
0
2
4
6
8
accuracy loss, %
(f) oDeepNN
Figure 73: Accuracy loss after adding augmented data to the training dataset in the mixed scenario (when
both parameters nand kwere limited). Negative loss values, i.e., when the accuracy increased, are set to
zero for clarity.
Figure 74 illustrates classification accuracy on the original (left) and augmented (middle) datasets, de-
pending on the total number of original training trials, k×n. Different colors indicate the number of sub-
jects n. Original classification plots (on the left) indicate that classifiers, especially LDA models, learn more
when n=17 (orange dots) compared to n=3 (blue dots) even though the total number of trials k×nis the
same. The difference between augmented and original classification accuracy (plotted on the right) demon-
strates how classifiers benefit from DA. Generally, all classifiers benefit from DA more when a relatively small
amount of original data is available k×n<700, while limited (NN models) to no (LDA models) significant
improvement is observed otherwise. In addition, the plots demonstrate that in the case of a small num-
ber of trials k×n, LDA models gain more when data comes from few subjects, while iShallowNN accuracy
improves more for n=17.
88
(a) LDA orig (b) LDA aug (c) LDA diff
(d) sLDA orig (e) sLDA aug (f) sLDA diff
(g) iShallowNN orig (h) iShallowNN aug (i) iShallowNN diff
(j) oShallowNN orig (k) oShallowNN aug (l) oShallowNN diff
(m) iDeepNN orig (n) iDeepNN aug (o) iDeepNN diff
(p) oDeepNN orig (q) oDeepNN aug (r) oDeepNN diff
Figure 74: Classification accuracy on the original (left) and augmented (middle) datasets, depending on
the total number of original training trials, k×n. Different colors indicate the number of subjects n. The
difference between augmented and original classification accuracy for each classifier is plotted in the right
column.
89
C Within-subject classification
Tables 9 and 10 contain detailed classification results for all investigated values of k. Figure 75 illustrates
subject-wise results for k=30 and k=120.
Table 9: Changes in within-classification accuracy after data augmentation (kfrom 20 to 70)
LDA ShallowNN DeepNN
k=20
original 60.87 68.32 62.69
augmented 71.89 70.02 64.44
difference +11.02*** +1.70*** +1.75**
k=25
original 65.03 69.59 62.57
augmented 73.82 71.05 64.8
difference +8.79*** +1.46*** +2.23***
k=30
original 69.07 70.81 62.98
augmented 75.06 72.78 67.95
difference +5.99*** +1.97*** +4.97***
k=35
original 72.3 71.83 65.87
augmented 76.44 74.65 68.92
difference +4.14*** +2.82*** +3.05***
k=40
original 74.26 72.84 67.54
augmented 77.57 76.03 71.82
difference +3.31*** +3.19*** +4.28***
k=45
original 75.70 73.40 67.86
augmented 78.52 77.38 76.55
difference +2.82*** +3.98*** +8.69***
k=50
original 76.98 73.98 67.54
augmented 79.25 77.43 73.20
difference +2.27*** +3.45*** +5.66***
k=55
original 78.43 74.49 67.92
augmented 80.01 79.10 77.75
difference +1.67** +4.61*** +9.83***
k=60
original 79.18 75.07 67.81
augmented 80.93 79.89 75.89
difference +1.75** +4.82*** +8.08***
k=70
original 80.45 78.34 71.58
augmented 81.73 80.95 77.44
difference +1.28** +2.61*** +5.86***
***p<0.01, **p<0.05, *p<0.1
90
Table 10: Changes in within-classification accuracy after data augmentation (kfrom 75 to 120)
LDA ShallowNN DeepNN
k=75
original 80.82 78.99 71.14
augmented 81.75 81.77 80.08
difference +0.93*+2.78*** +8.94***
k=80
original 81.11 79.48 70.99
augmented 81.75 82.4 78.92
difference +0.64 +2.92*** +7.93***
k=85
original 81.14 80.03 71.36
augmented 81.89 83.03 80.62
difference +0.75*+3.00*** +9.26***
k=90
original 81.28 80.01 72.06
augmented 81.82 83.59 83.41
difference +0.54 +3.94*** +11.35***
k=95
original 81.53 80.61 72.87
augmented 81.82 83.60 81.68
difference +0.29 +2.99*** +8.81***
k=100
original 81.58 80.86 74.87
augmented 81.67 83.77 83.58
difference +0.09 +2.91*** +8.71***
k=105
original 81.57 80.58 75.41
augmented 81.30 84.24 82.12
difference −0.27 +3.66*** +6.71***
k=110
original 81.53 80.57 75.19
augmented 81.37 84.73 83.27
difference −0.16 +4.16*** +8.08***
k=115
original 81.7 80.74 75.04
augmented 80.86 84.63 82.23
difference −0.84 +3.89*** +7.19***
k=120
original 82.12 80.87 74.23
augmented 81.22 85.10 83.27
difference −0.90 +4.23*** +9.04***
***p<0.01, **p<0.05, *p<0.1
91
(a) LDA, k=30 (b) ShallowNN, k=30 (c) DeepNN, k=30
(d) LDA, k=120 (e) ShallowNN, k=120 (f) DeepNN, k=120
Figure 75: Classification accuracy (percent correct) with three different classification models for k=30
and k=120. Results for N=1 refer to classification without data augmentation. Classification was done
within each participant. Participants on the x-axis are sorted separately for each classifier regarding to N=1
results.
D Evaluating generated EEG trials
D.1 Dipole visualization
Brain areas activated during motor imagery are similar to the ones activated during motor execution. The
differences in active brain regions during motor imagery, observation, and execution were investigated and
compared [Hardwick et al., 2017] (see Figure 76). Based on these results, we defined the borders for the
regions of interest related to MI tasks. Figure 77 illustrates localized dipoles for the first five SSD components
for each subject in our dataset.
Figure 76: Active brain areas during motor imagery, motor observation, and motor execution (from [Hard-
wick et al., 2017]).
92
Figure 77: Localized dipoles for the first five SSD components for each subject. Most of the subjects (16 out
of 18) have a motor-related component, among the first 5.
93
D.2 PSD of original and augmented data
Figure 78: PSD of original and augmented data for all subjects. Data augmentation effectively preserves ERD
in subjects who exhibit it in the original data.
94
References
[Abdelfattah et al., 2018] Abdelfattah, S. M., Abdelrahman, G. M., and Wang, M. (2018). Augmenting The
Size of EEG datasets Using Generative Adversarial Networks. In 2018 International Joint Conference on
Neural Networks (IJCNN), pages 1–6. ISSN: 2161-4407.
[Abiri et al., 2019] Abiri, R., Borhani, S., Sellers, E. W., Jiang, Y., and Zhao, X. (2019). A comprehensive review
of EEG-based brain–computer interface paradigms. Journal of Neural Engineering, 16(1):011001.
[Adeli et al., 2003] Adeli, H., Zhou, Z., and Dadmehr, N. (2003). Analysis of EEG records in an epileptic pa-
tient using wavelet transform. Journal of Neuroscience Methods, 123(1):69–87.
[Aggarwal and Chugh, 2022] Aggarwal, S. and Chugh, N. (2022). Review of Machine Learning Techniques
for EEG Based Brain Computer Interface. Archives of Computational Methods in Engineering, 29(5):3001–
3020.
[Al-Nafjan, 2022] Al-Nafjan, A. (2022). Feature selection of EEG signals in neuromarketing. PeerJ Computer
Science, 8:e944.
[Al-Saegh et al., 2021] Al-Saegh, A., Dawwd, S. A., and Abdul-Jabbar, J. M. (2021). Deep learning for motor
imagery EEG-based classification: A review. Biomedical Signal Processing and Control, 63:102172.
[Al Zoubi et al., 2018] Al Zoubi, O., Ki Wong, C., Kuplicki, R. T., Yeh, H.-w., Mayeli, A., Refai, H., Paulus, M.,
and Bodurka, J. (2018). Predicting Age From Brain EEG Signals—A Machine Learning Approach. Frontiers
in Aging Neuroscience, 10.
[AlSaleh et al., 2016] AlSaleh, M. M., Arvaneh, M., Christensen, H., and Moore, R. K. (2016). Brain-computer
interface technology for speech recognition: A review. In 2016 Asia-Pacific Signal and Information Pro-
cessing Association Annual Summit and Conference (APSIPA), pages 1–5, Jeju, South Korea. IEEE.
[Altaheri et al., 2021] Altaheri, H., Muhammad, G., Alsulaiman, M., Amin, S. U., Altuwaijri, G. A., Abdul, W.,
Bencherif, M. A., and Faisal, M. (2021). Deep learning techniques for classification of electroencephalo-
gram (EEG) motor imagery (MI) signals: a review. Neural Computing and Applications.
[Amin et al., 2019] Amin, S. U., Alsulaiman, M., Muhammad, G., Mekhtiche, M. A., and Shamim Hossain, M.
(2019). Deep Learning for EEG motor imagery classification based on multi-layer CNNs feature fusion.
Future Generation Computer Systems, 101:542–554.
[Ang et al., 2008] Ang, K. K., Chin, Z. Y., Zhang, H., and Guan, C. (2008). Filter Bank Common Spatial Pattern
(FBCSP) in Brain-Computer Interface. In 2008 IEEE International Joint Conference on Neural Networks
(IEEE World Congress on Computational Intelligence), pages 2390–2397. ISSN: 2161-4407.
[Arjovsky et al., 2017] Arjovsky, M., Chintala, S., and Bottou, L. (2017). Wasserstein GAN. arXiv:1701.07875
[cs, stat].
[Asadzadeh et al., 2020] Asadzadeh, S., Yousefi Rezaii, T., Beheshti, S., Delpak, A., and Meshgini, S. (2020).
A systematic review of EEG source localization techniques and their applications on diagnosis of brain
abnormalities. Journal of Neuroscience Methods, 339:108740.
[Atyabi et al., 2012] Atyabi, A., Luerssen, M., Fitzgibbon, S., and Powers, D. M. W. (2012). Evolutionary fea-
ture selection and electrode reduction for EEG classification. In 2012 IEEE Congress on Evolutionary Com-
putation, pages 1–8. ISSN: 1941-0026.
[Aydın et al., 2016] Aydın, E. A., Bay, O. F., and Gueler, I. (2016). Implementation of an Embedded Web Server
Application for Wireless Control of Brain Computer Interface Based Home Environments. Journal of Med-
ical Systems, 40(1):1–10.
[Azevedo et al., 2009] Azevedo, F. A., Carvalho, L. R., Grinberg, L. T., Farfel, J. M., Ferretti, R. E., Leite, R. E.,
Filho, W. J., Lent, R., and Herculano-Houzel, S. (2009). Equal numbers of neuronal and nonneuronal
cells make the human brain an isometrically scaled-up primate brain. Journal of Comparative Neurology,
513(5):532–541. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/cne.21974.
95
[Aznan et al., 2019] Aznan, N. K. N., Atapour-Abarghouei, A., Bonner, S., Connolly, J. D., Al Moubayed, N.,
and Breckon, T. P. (2019). Simulating Brain Signals: Creating Synthetic EEG Data via Neural-Based Genera-
tive Models for Improved SSVEP Classification. In 2019 International Joint Conference on Neural Networks
(IJCNN), pages 1–8. ISSN: 2161-4407.
[Baig et al., 2017] Baig, M. Z., Aslam, N., Shum, H. P. H., and Zhang, L. (2017). Differential evolution algo-
rithm as a tool for optimal feature subset selection in motor imagery EEG. Expert Systems with Applica-
tions, 90:184–195.
[Balakrishnan and Puthusserypady, 2005] Balakrishnan, D. and Puthusserypady, S. (2005). Multilayer per-
ceptrons for the classification of brain computer interface data. In Proceedings of the IEEE 31st Annual
Northeast Bioengineering Conference, 2005., pages 118–119, Hoboken, New Jersey, USA. IEEE.
[Bandara et al., 2018] Bandara, D. S. V., Arata, J., and Kiguchi, K. (2018). Towards Control of a Transhumeral
Prosthesis with EEG Signals. Bioengineering, 5(2):26. Number: 2 Publisher: Multidisciplinary Digital
Publishing Institute.
[Barachant et al., 2010] Barachant, A., Bonnet, S., Congedo, M., and Jutten, C. (2010). Riemannian geom-
etry applied to BCI classification. In Vigneron, Zarzoso, V. ., Moreau, V. ., Gribonval, E. ., Vincent, R. .,
and (Eds.), E., editors, LVA/ICA 2010 - 9th International Conference on Latent Variable Analysis and Signal
Separation, volume 6365 of Series: Lecture Notes in Computer Science. Subseries: Theoretical Computer
Science and General Issues, pages 629–636, Saint-Malo, France. Springer.
[Barachant et al., 2012] Barachant, A., Bonnet, S., Congedo, M., and Jutten, C. (2012). Multiclass
Brain–Computer Interface Classification by Riemannian Geometry. IEEE Transactions on Biomedical En-
gineering, 59(4):920–928. Conference Name: IEEE Transactions on Biomedical Engineering.
[Barachant et al., 2013] Barachant, A., Bonnet, S., Congedo, M., and Jutten, C. (2013). Classification of co-
variance matrices using a Riemannian-based kernel for BCI applications. Neurocomputing, 112:172–178.
Publisher: Elsevier.
[Bengio et al., 1994] Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with
gradient descent is difficult. IEEE transactions on neural networks / a publication of the IEEE Neural Net-
works Council, 5:157–66.
[Berger, 1929] Berger, H. (1929). Über das Elektrenkephalogramm des Menschen. Archiv für Psychiatrie und
Nervenkrankheiten, 87(1):527–570.
[Bhatia, 2007] Bhatia, R. (2007). Positive definite matrices. Princeton series in applied mathematics. Prince-
ton University Press, Princeton, N.J. OCLC: ocm70668921.
[Blankertz et al., 2016] Blankertz, B., Acqualagna, L., Dähne, S., Haufe, S., Schultze-Kraft, M., Sturm, I.,
Uš´cumlic, M., Wenzel, M. A., Curio, G., and Müller, K.-R. (2016). The Berlin Brain-Computer Interface:
Progress Beyond Communication and Control. Frontiers in Neuroscience, 10.
[Blankertz et al., 2007] Blankertz, B., Kawanabe, M., Tomioka, R., Hohlefeld, F., Müller, K.-R., and Nikulin, V.
(2007). Invariant Common Spatial Patterns: Alleviating Nonstationarities in Brain-Computer Interfacing.
In Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc.
[Blankertz et al., 2011] Blankertz, B., Lemm, S., Treder, M., Haufe, S., and Müller, K.-R. (2011). Single-trial
analysis and classification of ERP components–a tutorial. NeuroImage, 56(2):814–825.
[Blankertz et al., 2008a] Blankertz, B., Losch, F., Krauledat, M., Dornhege, G., Curio, G., and Müller, K.-R.
(2008a). The Berlin Brain-Computer Interface: Accurate performance from first-session in BCI-naive
subjects. IEEE Transactions on Biomedical Engineering, 55(10):2452–2462. Conference Name: IEEE Trans-
actions on Biomedical Engineering.
[Blankertz et al., 2010] Blankertz, B., Sannelli, C., Halder, S., Hammer, E. M., Kübler, A., Müller, K.-R., Curio,
G., and Dickhaus, T. (2010). Neurophysiological predictor of SMR-based BCI performance. NeuroImage,
51(4):1303–1309.
96
[Blankertz et al., 2008b] Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., and Muller, K.-r. (2008b). Op-
timizing Spatial filters for Robust EEG Single-Trial Analysis. IEEE Signal Processing Magazine, 25(1):41–56.
Conference Name: IEEE Signal Processing Magazine.
[Borisoff et al., 2004] Borisoff, J., Mason, S., Bashashati, A., and Birch, G. (2004). Brain-computer interface
design for asynchronous control applications: improvements to the LF-ASD asynchronous brain switch.
IEEE Transactions on Biomedical Engineering, 51(6):985–992. Conference Name: IEEE Transactions on
Biomedical Engineering.
[Boto et al., 2018] Boto, E., Holmes, N., Leggett, J., Roberts, G., Shah, V., Meyer, S. S., Muñoz, L. D., Mullinger,
K. J., Tierney, T. M., Bestmann, S., Barnes, G. R., Bowtell, R., and Brookes, M. J. (2018). Moving magne-
toencephalography towards real-world applications with a wearable system. Nature, 555(7698):657–661.
[Bousseta et al., 2018] Bousseta, R., El Ouakouak, I., Gharbi, M., and Regragui, F. (2018). EEG Based Brain
Computer Interface for Controlling a Robot Arm Movement Through Thought. IRBM, 39(2):129–135.
[Brette and Destexhe, 2012] Brette, R. and Destexhe, A., editors (2012). Handbook of Neural Activity Mea-
surement. Cambridge University Press, Cambridge.
[Britton et al., 2016] Britton, J. W., Frey, L. C., Hopp, J. L., Korb, P., Koubeissi, M. Z., Lievens, W. E., Pestana-
Knight, E. M., and St. Louis, E. K. (2016). Electroencephalography (EEG): An Introductory Text and Atlas of
Normal and Abnormal Findings in Adults, Children, and Infants. American Epilepsy Society, Chicago.
[Buzsáki and Draguhn, 2004] Buzsáki, G. and Draguhn, A. (2004). Neuronal oscillations in cortical networks.
Science (New York, N.Y.), 304(5679):1926–1929.
[Campbell et al., 1991] Campbell, P., Jones, K., Huber, R., Horch, K., and Normann, R. (1991). A silicon-
based, three-dimensional neural interface: manufacturing processes for an intracortical electrode array.
IEEE Transactions on Biomedical Engineering, 38(8):758–768. Conference Name: IEEE Transactions on
Biomedical Engineering.
[Cantillo-Negrete et al., 2016] Cantillo-Negrete, J., Carino-Escobar, R. I., Carrillo-Mora, P., Flores-Rodríguez,
T. B., Elías-Vinas, D., and Gutiérrez-Martínez, J. (2016). Gender Differences in Quantitative Electroen-
cephalogram During a Simple Hand Movement Task in Young Adults. Rev Inves Clin.
[Cecotti et al., 2015] Cecotti, H., Marathe, A. R., and Ries, A. J. (2015). Optimization of Single-Trial Detec-
tion of Event-Related Potentials Through Artificial Trials. IEEE transactions on bio-medical engineering,
62(9):2170–2176.
[Chai et al., 2017] Chai, R., Naik, G. R., Nguyen, T. N., Ling, S. H., Tran, Y., Craig, A., and Nguyen, H. T. (2017).
Driver Fatigue Classification With Independent Component by Entropy Rate Bound Minimization Analy-
sis in an EEG-Based System. IEEE Journal of Biomedical and Health Informatics, 21(3):715–724. Confer-
ence Name: IEEE Journal of Biomedical and Health Informatics.
[Chakladar and Chakraborty, 2018] Chakladar, D. D. and Chakraborty, S. (2018). Multi-target way of cur-
sor movement in brain computer interface using unsupervised learning. Biologically Inspired Cognitive
Architectures, 25:88–100.
[Chambon et al., 2018] Chambon, S., Galtier, M. N., Arnal, P. J., Wainrib, G., and Gramfort, A. (2018). A Deep
Learning Architecture for Temporal Sleep Stage Classification Using Multivariate and Multimodal Time
Series. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 26(4):758–769. Conference
Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering.
[Chang et al., 2014a] Chang, M., Mori, K., Makino, S., and Rutkowski, T. M. (2014a). Spatial Auditory Two-
step Input Japanese Syllabary Brain-computer Interface Speller. Procedia Technology, 18:25–31.
[Chang et al., 2014b] Chang, M. H., Baek, H. J., Lee, S. M., and Park, K. S. (2014b). An amplitude-modulated
visual stimulation for reducing eye fatigue in SSVEP-based brain–computer interfaces. Clinical Neuro-
physiology, 125(7):1380–1391.
97
[Chatterjee and Sanyal, 2020] Chatterjee, R. and Sanyal, D. K. (2020). Study of Different Filter Bank Ap-
proaches in Motor-Imagery EEG Signal Classification. In Pattnaik, P. K., Mohanty, S., and Mohanty, S.,
editors, Smart Healthcare Analytics in IoT Enabled Environment, Intelligent Systems Reference Library,
pages 173–190. Springer International Publishing, Cham.
[Chavarriaga et al., 2014] Chavarriaga, R., Sobolewski, A., and Millán, J. d. R. (2014). Errare machinale est:
the use of error-related potentials in brain-machine interfaces. Frontiers in Neuroscience, 8.
[Chen et al., 2017] Chen, J., Zhang, D., Engel, A. K., Gong, Q., and Maye, A. (2017). Application of a single-
flicker online SSVEP BCI for spatial navigation. PLOS ONE, 12(5):e0178385. Publisher: Public Library of
Science.
[Chen et al., 2022] Chen, X., Teng, X., Chen, H., Pan, Y., and Geyer, P. (2022). Toward reliable signals decoding
for electroencephalogram: A benchmark study to EEGNeX. arXiv:2207.12369 [cs, eess, q-bio].
[Chen et al., 2015] Chen, X., Wang, Y., Nakanishi, M., Gao, X., Jung, T.-P., and Gao, S. (2015). High-speed
spelling with a noninvasive brain–computer interface. Proceedings of the National Academy of Sciences,
112(44):E6058–E6067. Publisher: Proceedings of the National Academy of Sciences.
[Cheng et al., 2018] Cheng, D., Liu, Y., and Zhang, L. (2018). Exploring Motor Imagery Eeg Patterns for Stroke
Patients with Deep Neural Networks. In 2018 IEEE International Conference on Acoustics, Speech and
Signal Processing (ICASSP), pages 2561–2565. ISSN: 2379-190X.
[Cheng et al., 2020] Cheng, J. Y., Goh, H., Dogrusoz, K., Tuzel, O., and Azemi, E. (2020). Subject-Aware Con-
trastive Learning for Biosignals. arXiv:2007.04871 [cs, eess, stat].
[Cheng et al., 2017] Cheng, M., Lu, Z., and Wang, H. (2017). Regularized common spatial patterns with
subject-to-subject transfer of EEG signals. Cognitive neurodynamics, 11(2):173–181.
[Cho et al., 2014] Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and
Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine
Translation. arXiv:1406.1078 [cs, stat].
[Chu et al., 2018] Chu, Y., Zhao, X., Zou, Y., Xu, W., Han, J., and Zhao, Y. (2018). A Decoding Scheme for
Incomplete Motor Imagery EEG With Deep Belief Network. Frontiers in Neuroscience, 12.
[Citi et al., 2008] Citi, L., Poli, R., Cinel, C., and Sepulveda, F. (2008). P300-Based BCI Mouse With
Genetically-Optimized Analogue Control. IEEE transactions on neural systems and rehabilitation engi-
neering : a publication of the IEEE Engineering in Medicine and Biology Society, 16:51–61.
[Cohen, 2014] Cohen, M. X. (2014). Analyzing Neural Time Series Data: Theory and Practice. The MIT Press.
[Collewijn et al., 1985] Collewijn, H., van der Steen, J., and Steinman, R. M. (1985). Human eye movements
associated with blinks and prolonged eyelid closure. Journal of Neurophysiology, 54(1):11–27. Publisher:
American Physiological Society.
[Congedo et al., 2017] Congedo, M., Barachant, A., and Bhatia, R. (2017). Riemannian geometry for EEG-
based brain-computer interfaces; a primer and a review. Brain-Computer Interfaces, 4(3):155–174. Pub-
lisher: Taylor & Francis.
[Croft and Barry, 2000] Croft, R. and Barry, R. (2000). Removal of ocular artifact from the EEG: a review.
Neurophysiologie Clinique/Clinical Neurophysiology, 30(1):5–19.
[Cubuk et al., 2019] Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V. (2019). AutoAugment:
Learning Augmentation Policies from Data. arXiv:1805.09501 [cs, stat].
[Dai et al., 2020] Dai, G., Zhou, J., Huang, J., and Wang, N. (2020). HS-CNN: a CNN with hybrid convolution
scale for EEG motor imagery classification. Journal of Neural Engineering, 17(1):016025.
[Dai et al., 2019] Dai, M., Zheng, D., Na, R., Wang, S., and Zhang, S. (2019). EEG Classification of Motor
Imagery Using a Novel Deep Learning Framework. Sensors, 19(3):551.
98
[Dalal et al., 2008] Dalal, S. S., Guggisberg, A. G., Edwards, E., Sekihara, K., Findlay, A. M., Canolty, R. T.,
Berger, M. S., Knight, R. T., Barbaro, N. M., Kirsch, H. E., and Nagarajan, S. S. (2008). Five-dimensional
neuroimaging: Localization of the time-frequency dynamics of cortical activity. NeuroImage, 40(4):1686–
1700.
[Dhariwal and Nichol, 2021] Dhariwal, P. and Nichol, A. (2021). Diffusion Models Beat GANs on Image Syn-
thesis. In Advances in Neural Information Processing Systems, volume 34, pages 8780–8794. Curran Asso-
ciates, Inc.
[Dickhaus et al., 2009] Dickhaus, T., Sannelli, C., Müller, K.-R., Curio, G., and Blankertz, B. (2009). Predicting
BCI performance to study BCI illiteracy. BMC Neuroscience, 10(S1):P84, 1471–2202–10–S1–P84.
[Dinarès-Ferran et al., 2018] Dinarès-Ferran, J., Ortner, R., Guger, C., and Solé-Casals, J. (2018). A New
Method to Generate Artificial Frames Using the Empirical Mode Decomposition for an EEG-Based Motor
Imagery BCI. Frontiers in Neuroscience, 12.
[Ding and He, 2008] Ding, L. and He, B. (2008). Sparse source imaging in electroencephalography with ac-
curate field modeling. Human Brain Mapping, 29(9):1053–1067.
[Downey et al., 2017] Downey, J. E., Brane, L., Gaunt, R. A., Tyler-Kabara, E. C., Boninger, M. L., and
Collinger, J. L. (2017). Motor cortical activity changes during neuroprosthetic-controlled object inter-
action. Scientific Reports, 7(1):16947. Number: 1 Publisher: Nature Publishing Group.
[Easttom et al., 2021] Easttom, C., Bianchi, L., Valeriani, D., Nam, C. S., Hossaini, A., Zapała, D., Roman-
Gonzalez, A., Singh, A. K., Antonietti, A., Sahonero-Alvarez, G., and Balachandran, P. (2021). A Functional
Model for Unifying Brain Computer Interface Terminology. IEEE Open Journal of Engineering in Medicine
and Biology, 2:91–96. Conference Name: IEEE Open Journal of Engineering in Medicine and Biology.
[Edelman et al., 2016] Edelman, B. J., Baxter, B., and He, B. (2016). EEG Source Imaging Enhances the De-
coding of Complex Right-Hand Motor Imagery Tasks. IEEE transactions on bio-medical engineering,
63(1):4–14.
[Edlinger et al., 2009] Edlinger, G., Holzner, C., Groenegress, C., Guger, C., and Slater, M. (2009). Goal-
Oriented Control with Brain-Computer Interface. In Schmorrow, D. D., Estabrooke, I. V., and Grootjen,
M., editors, Foundations of Augmented Cognition. Neuroergonomics and Operational Neuroscience, Lec-
ture Notes in Computer Science, pages 732–740, Berlin, Heidelberg. Springer.
[Elstob and Secco, 2016] Elstob, D. and Secco, E. L. (2016). A Low Cost Eeg Based Bci Prosthetic Using Motor
Imagery. arXiv:1603.02869 [cs].
[Engel et al., 2023] Engel, J., Agrawal, K. K., Chen, S., Gulrajani, I., Donahue, C., and Roberts, A. (2023). GAN-
Synth: Adversarial Neural Audio Synthesis.
[Fahimi et al., 2021] Fahimi, F., Dosen, S., Ang, K. K., Mrachacz-Kersting, N., and Guan, C. (2021). Genera-
tive Adversarial Networks-Based Data Augmentation for Brain–Computer Interface. IEEE Transactions on
Neural Networks and Learning Systems, 32(9):4039–4051. Conference Name: IEEE Transactions on Neural
Networks and Learning Systems.
[Fahimi et al., 2019] Fahimi, F., Zhang, Z., Goh, W. B., Lee, T.-S., Ang, K. K., and Guan, C. (2019). Inter-subject
transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI. Journal of
Neural Engineering, 16(2):026007. Publisher: IOP Publishing.
[Farwell and Donchin, 1988] Farwell, L. A. and Donchin, E. (1988). Talking off the top of your head: toward
a mental prosthesis utilizing event-related brain potentials. Electroencephalography and Clinical Neuro-
physiology, 70(6):510–523.
[Fazli et al., 2009] Fazli, S., Popescu, F., Danóczy, M., Blankertz, B., Müller, K.-R., and Grozea, C. (2009).
Subject-independent mental state classification in single trials. Neural Networks, 22(9):1305–1312.
[Ferrez et al., 2023] Ferrez, P., Millan, J. d. R., Dornhege, G., Hinterberger, T., McFarland, D., and Müller, K.-R.
(2023). Error-Related EEG Potentials in Brain-Computer Interfaces.
99
[Freer and Yang, 2020] Freer, D. and Yang, G.-Z. (2020). Data augmentation for self-paced motor imagery
classification with C-LSTM. Journal of Neural Engineering, 17(1):016041. Publisher: IOP Publishing.
[Friston et al., 2008] Friston, K., Harrison, L., Daunizeau, J., Kiebel, S., Phillips, C., Trujillo-Barreto, N., Hen-
son, R., Flandin, G., and Mattout, J. (2008). Multiple sparse priors for the M/EEG inverse problem. Neu-
roImage, 39(3):1104–1120.
[Fuchs et al., 2007] Fuchs, M., Wagner, M., and Kastner, J. (2007). Development of Volume Conductor and
Source Models to Localize Epileptic Foci. Journal of Clinical Neurophysiology, 24(2):101.
[Fukunaga, 1990] Fukunaga, K. (1990). Introduction to Statistical Pattern Recognition Second Edition.
[Fukushima, 1980] Fukushima, K. (1980). Neocognitron: A self-organizing neural network model for a
mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193–202.
[Goldberger et al., 2000] Goldberger, A. L., Amaral, L. A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark,
R. G., Mietus, J. E., Moody, G. B., Peng, C. K., and Stanley, H. E. (2000). PhysioBank, PhysioToolkit,
and PhysioNet: components of a new research resource for complex physiologic signals. Circulation,
101(23):E215–220.
[Goldenholz et al., 2008] Goldenholz, D. M., Ahlfors, S. P., Hämäläinen, M. S., Sharon, D., Ishitobi, M., Vaina,
L. M., and Stufflebeam, S. M. (2008). Mapping the signal-to-noise-ratios of cortical sources in magnetoen-
cephalography and electroencephalography. Human Brain Mapping, 30(4):1077–1086.
[Gomarus et al., 2009] Gomarus, H. K., Wijers, A. A., Minderaa, R. B., and Althaus, M. (2009). ERP correlates
of selective attention and working memory capacities in children with ADHD and/or PDD-NOS. Clinical
Neurophysiology, 120(1):60–72.
[Gore, 2003] Gore, J. C. (2003). Principles and practice of functional MRI of the human brain. Journal of
Clinical Investigation, 112(1):4–9.
[Gorodnitsky et al., 1995] Gorodnitsky, I. F., George, J. S., and Rao, B. D. (1995). Neuromagnetic source imag-
ing with FOCUSS: a recursive weighted minimum norm algorithm. Electroencephalography and Clinical
Neurophysiology, 95(4):231–251.
[Greischar et al., 2004] Greischar, L. L., Burghy, C. A., van Reekum, C. M., Jackson, D. C., Pizzagalli, D. A.,
Mueller, C., and Davidson, R. J. (2004). Effects of electrode density and electrolyte spreading in dense
array electroencephalographic recording. Clinical Neurophysiology: Official Journal of the International
Federation of Clinical Neurophysiology, 115(3):710–720.
[Gulrajani et al., 2017] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. (2017). Im-
proved Training of Wasserstein GANs. arXiv:1704.00028 [cs, stat].
[Guo et al., 2015] Guo, S., Lin, S., and Huang, Z. (2015). Feature extraction of P300s in EEG signal with dis-
crete wavelet transform and fisher criterion. In 2015 8th International Conference on Biomedical Engi-
neering and Informatics (BMEI), pages 200–204.
[Guo et al., 2013] Guo, X., Wu, X., Gong, X., and Zhang, L. (2013). Envelope detection based on online ICA
algorithm and its application to motor imagery classification. In 2013 6th International IEEE/EMBS Con-
ference on Neural Engineering (NER), pages 1058–1061. ISSN: 1948-3554.
[Haemaelaeinen and Sarvas, 1989] Haemaelaeinen, M. S. and Sarvas, J. (1989). Realistic conductivity geom-
etry model of the human head for interpretation of neuromagnetic data. IEEE transactions on bio-medical
engineering, 36(2):165–171.
[Hallez et al., 2006] Hallez, H., Vergult, A., Phlypo, R., Van Hese, P., De Clercq, W., D’Asseler, Y., Van de Walle,
R., Vanrumste, B., Van Paesschen, W., Van Huffel, S., and Lemahieu, I. (2006). Muscle and eye movement
artifact removal prior to EEG source localization. Conference proceedings: ... Annual International Con-
ference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology
Society. Annual Conference, 2006:1002–1005.
100
[Hamedi et al., 2014] Hamedi, M., Salleh, S.-H., Noor, A. M., and Mohammad-Rezazadeh, I. (2014). Neural
network-based three-class motor imagery classification using time-domain features for BCI applications.
In 2014 IEEE REGION 10 SYMPOSIUM, pages 204–207.
[Hammer et al., 2012] Hammer, E. M., Halder, S., Blankertz, B., Sannelli, C., Dickhaus, T., Kleih, S., Müller,
K.-R., and Kübler, A. (2012). Psychological predictors of SMR-BCI performance. Biological Psychology,
89(1):80–86.
[Hardwick et al., 2017] Hardwick, R. M., Caspers, S., Eickhoff, S. B., and Swinnen, S. P. (2017). Neural Corre-
lates of Motor Imagery, Action Observation, and Movement Execution: A Comparison Across Quantitative
Meta-Analyses. Pages: 198432 Section: New Results.
[Harmening et al., 2022] Harmening, N., Klug, M., Gramann, K., and Miklody, D. (2022). HArt-
MuT—modeling eye and muscle contributors in neuroelectric imaging. Journal of Neural Engineering,
19(6):066041. Publisher: IOP Publishing.
[Hartmann et al., 2018] Hartmann, K. G., Schirrmeister, R. T., and Ball, T. (2018). EEG-GAN: Generative ad-
versarial networks for electroencephalograhic (EEG) brain signals. arXiv:1806.01875 [cs, eess, q-bio, stat].
[Haselsteiner and Pfurtscheller, 2000] Haselsteiner, E. and Pfurtscheller, G. (2000). Using time-dependent
neural networks for EEG classification. IEEE Transactions on Rehabilitation Engineering, 8(4):457–463.
Conference Name: IEEE Transactions on Rehabilitation Engineering.
[Hassanpour et al., 2019] Hassanpour, A., Moradikia, M., Adeli, H., Khayami, S. R., and Sham-
sinejadbabaki, P. (2019). A novel end-to-end deep learning scheme for classifying multi-
class motor imagery electroencephalography signals. Expert Systems, 36(6):e12494. _eprint:
https://onlinelibrary.wiley.com/doi/pdf/10.1111/exsy.12494.
[Haufe et al., 2014] Haufe, S., Meinecke, F., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., and Bieß-
mann, F. (2014). On the interpretation of weight vectors of linear models in multivariate neuroimaging.
NeuroImage, 87:96–110.
[He et al., 2018] He, B., Sohrabpour, A., Brown, E., and Liu, Z. (2018). Electrophysiological Source Imaging:
a Noninvasive Window to Brain Dynamics. Annual review of biomedical engineering, 20:171–196.
[He et al., 2021] He, C., Liu, J., Zhu, Y., and Du, W. (2021). Data Augmentation for Deep Neural Networks
Model in EEG Classification Task: A Review. Frontiers in Human Neuroscience, 15.
[He et al., 2016] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity Mappings in Deep Residual Networks.
arXiv:1603.05027 [cs].
[Hinton, 2012] Hinton, G. E. (2012). A Practical Guide to Training Restricted Boltzmann Machines. In Mon-
tavon, G., Orr, G. B., and Müller, K.-R., editors, Neural Networks: Tricks of the Trade, volume 7700, pages
599–619. Springer Berlin Heidelberg, Berlin, Heidelberg. Series Title: Lecture Notes in Computer Science.
[Hochberg et al., 2012] Hochberg, L. R., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J.,
Haddadin, S., Liu, J., Cash, S. S., Smagt, P. v. d., and Donoghue, J. P. (2012). Reach and grasp by people with
tetraplegia using a neurally controlled robotic arm. Nature, 485(7398):372. Publisher: NIH Public Access.
[Hochreiter and Schmidhuber, 1997] Hochreiter, S. and Schmidhuber, J. (1997). Long Short-Term Memory.
Neural Computation, 9(8):1735–1780.
[Houmani et al., 2018] Houmani, N., Vialatte, F., Gallego-Jutglà, E., Dreyfus, G., Nguyen-Michel, V.-H., Mar-
iani, J., and Kinugawa, K. (2018). Diagnosis of Alzheimer’s disease with Electroencephalography in a
differential framework. PLOS ONE, 13(3):e0193607. Publisher: Public Library of Science.
[Hsu, 2011] Hsu, W.-Y. (2011). EEG-based motor imagery classification using enhanced active segment se-
lection and adaptive classifier. Computers in Biology and Medicine, 41(8):633–639.
[Hu and Zhang, 2019] Hu, L. and Zhang, Z., editors (2019). EEG Signal Processing and Feature Extraction.
Springer, Singapore.
101
[Huang et al., 1998] Huang, N. E., Shen, Z., Long, S. R., Wu, M. C., Shih, H. H., Zheng, Q., Yen, N.-C., Tung,
C. C., and Liu, H. H. (1998). The Empirical Mode Decomposition and the Hilbert Spectrum for Nonlinear
and Non-Stationary Time Series Analysis. Proceedings: Mathematical, Physical and Engineering Sciences,
454(1971):903–995. Publisher: The Royal Society.
[Hämäläinen and Ilmoniemi, 1994] Hämäläinen, M. S. and Ilmoniemi, R. J. (1994). Interpreting magnetic
fields of the brain: minimum norm estimates. Medical & Biological Engineering & Computing, 32(1):35–
42.
[Höhne et al., 2011] Höhne, J., Schreuder, M., Blankertz, B., and Tangermann, M. (2011). A Novel 9-Class
Auditory ERP Paradigm Driving a Predictive Text Entry System. Frontiers in Neuroscience, 5.
[Ieracitano et al., 2020] Ieracitano, C., Mammone, N., Hussain, A., and Morabito, F. C. (2020). A novel multi-
modal machine learning based approach for automatic classification of EEG recordings in dementia.
Neural Networks, 123:176–190.
[Ingolfsson et al., 2020] Ingolfsson, T. M., Hersche, M., Wang, X., Kobayashi, N., Cavigelli, L., and Benini, L.
(2020). EEG-TCNet: An Accurate Temporal Convolutional Network for Embedded Motor-Imagery Brain-
Machine Interfaces. arXiv:2006.00622 [cs, eess, stat].
[Iturrate et al., 2009] Iturrate, I., Antelis, J. M., Kubler, A., and Minguez, J. (2009). A Noninvasive Brain-
Actuated Wheelchair Based on a P300 Neurophysiological Protocol and Automated Navigation. IEEE
Transactions on Robotics, 25(3):614–627. Conference Name: IEEE Transactions on Robotics.
[Iturrate et al., 2015] Iturrate, I., Chavarriaga, R., Montesano, L., Minguez, J., and Millán, J. d. R. (2015).
Teaching brain-machine interfaces as an alternative paradigm to neuroprosthetics control. Scientific Re-
ports, 5(1):13893. Number: 1 Publisher: Nature Publishing Group.
[Iturrate et al., 2013] Iturrate, I., Montesano, L., and Minguez, J. (2013). Shared-control brain-computer in-
terface for a two dimensional reaching task using EEG error-related potentials. In 2013 35th Annual In-
ternational Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 5258–5262,
Osaka. IEEE.
[Jalil Razavi et al., 2015] Jalil Razavi, M., Zhang, T., Liu, T., and Wang, X. (2015). Cortical Folding Pattern and
its Consistency Induced by Biological Growth. Scientific Reports, 5(1):14477. Number: 1 Publisher: Nature
Publishing Group.
[Javaid et al., 2022] Javaid, H., Kumarnsit, E., and Chatpun, S. (2022). Age-Related Alterations in EEG Net-
work Connectivity in Healthy Aging. Brain Sciences, 12(2):218.
[Kalaganis et al., 2020] Kalaganis, F. P., Laskaris, N. A., Chatzilari, E., Nikolopoulos, S., and Kompatsiaris,
I. (2020). A Data Augmentation Scheme for Geometric Deep Learning in Personalized Brain–Computer
Interfaces. IEEE Access, 8:162218–162229. Conference Name: IEEE Access.
[Kalunga et al., 2015] Kalunga, E., Chevallier, S., and Barthélemy, Q. (2015). Data augmentation in Rieman-
nian space for Brain-Computer Interfaces.
[Kandel et al., 2000] Kandel, E. R., Schwartz, J. H., and Jessell, T. M. (2000). Principles of neural science.
McGraw-Hill, Health Professions Division, New York, 4th ed edition. OCLC: 42073108.
[Kang et al., 2009] Kang, H., Nam, Y., and Choi, S. (2009). Composite Common Spatial Pattern for Subject-
to-Subject Transfer. Signal Processing Letters, IEEE, 16:683–686.
[Kaper et al., 2004] Kaper, M., Meinicke, P., Grossekathoefer, U., Lingner, T., and Ritter, H. (2004). BCI com-
petition 2003-data set IIb: support vector machines for the P300 speller paradigm. IEEE Transactions on
Biomedical Engineering, 51(6):1073–1076. Conference Name: IEEE Transactions on Biomedical Engineer-
ing.
[Kavanagh et al., 1978] Kavanagh, R. N., Darcey, T. M., Lehmann, D., and Fender, D. H. (1978). Evaluation of
methods for three-dimensional localization of electrical sources in the human brain. IEEE transactions
on bio-medical engineering, 25(5):421–429.
102
[Kellis et al., 2010] Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., and Greger, B. (2010). Decoding
spoken words using local field potentials recorded from the cortical surface. Journal of neural engineering,
7(5):056007.
[Kennedy, 2010] Kennedy, J. (2010). Particle Swarm Optimization. In Sammut, C. and Webb, G. I., editors,
Encyclopedia of Machine Learning, pages 760–766. Springer US, Boston, MA.
[Kennedy et al., 2000] Kennedy, P., Bakay, R., Moore, M., Adams, K., and Goldwaithe, J. (2000). Direct control
of a computer from the human central nervous system. IEEE Transactions on Rehabilitation Engineering,
8(2):198–202.
[Khosla et al., 2020] Khosla, A., Khandnor, P., and Chand, T. (2020). A comparative analysis of signal pro-
cessing and classification methods for different applications based on EEG signals. Biocybernetics and
Biomedical Engineering, 40(2):649–690.
[Kim et al., 2006] Kim, S.-J., Manyam, S., Warren, D., and Normann, R. (2006). Electrophysiological Mapping
of Cat Primary Auditory Cortex with Multielectrode Arrays. Annals of biomedical engineering, 34:300–9.
[Kingma and Welling, 2013] Kingma, D. P. and Welling, M. (2013). Auto-Encoding Variational Bayes.
arXiv:1312.6114 [cs, stat] version: 1.
[Ko et al., 2019] Ko, W., Jeon, E., Lee, J., and Suk, H.-I. (2019). Semi-Supervised Deep Adversarial Learning
for Brain-Computer Interface. In 2019 7th International Winter Conference on Brain-Computer Interface
(BCI), pages 1–4. ISSN: 2572-7672.
[Ko et al., 2018] Ko, W., Yoon, J., Kang, E., Jun, E., Choi, J.-S., and Suk, H.-I. (2018). Deep recurrent spatio-
temporal neural network for motor imagery based BCI. In 2018 6th International Conference on Brain-
Computer Interface (BCI), pages 1–3. ISSN: 2572-7672.
[Kong and Ping, 2021] Kong, Z. and Ping, W. (2021). On Fast Sampling of Diffusion Probabilistic Models.
[Kong et al., 2023] Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. (2023). DiffWave: A Versatile
Diffusion Model for Audio Synthesis.
[Krell and Kim, 2017] Krell, M. M. and Kim, S. K. (2017). Rotational data augmentation for electroencephalo-
graphic data. In 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biol-
ogy Society (EMBC), pages 471–474. ISSN: 1558-4615.
[Krol et al., 2022] Krol, L. R., Klaproth, O. W., Vernaleken, C., Russwinkel, N., and Zander, T. O. (2022). Chap-
ter 4 - Towards neuroadaptive modeling: assessing the cognitive states of pilots through passive brain-
computer interfacing. In Fairclough, S. H. and Zander, T. O., editors, Current Research in Neuroadaptive
Technology, pages 59–73. Academic Press.
[Kronegg et al., 2007] Kronegg, J., Chanel, G., Voloshynovskiy, S., and Pun, T. (2007). EEG-Based Synchro-
nized Brain-Computer Interfaces: A Model for Optimizing the Number of Mental Tasks. IEEE Transactions
on Neural Systems and Rehabilitation Engineering, 15(1):50–58. Conference Name: IEEE Transactions on
Neural Systems and Rehabilitation Engineering.
[Kumar et al., 2019] Kumar, A., Gao, L., Pirogova, E., and Fang, Q. (2019). A Review of Error-Related
Potential-Based Brain–Computer Interfaces for Motor Impaired People. IEEE Access, 7:142451–142466.
Conference Name: IEEE Access.
[Kumar et al., 2021] Kumar, S., Sharma, R., and Sharma, A. (2021). OPTICAL+: a frequency-based deep
learning scheme for recognizing brain wave signals. PeerJ Computer Science, 7:e375. Publisher: PeerJ
Inc.
[Lai et al., 2005] Lai, Y., van Drongelen, W., Ding, L., Hecox, K. E., Towle, V. L., Frim, D. M., and He, B. (2005).
Estimation of in vivo human brain-to-skull conductivity ratio from simultaneous extra- and intra-cranial
electrical potential recordings. Clinical Neurophysiology, 116(2):456–465.
[Lawhern et al., 2012] Lawhern, V., Hairston, W. D., McDowell, K., Westerfield, M., and Robbins, K. (2012).
Detection and classification of subject-generated artifacts in EEG signals using autoregressive models.
Journal of Neuroscience Methods, 208(2):181–189.
103
[Lawhern et al., 2018] Lawhern, V. J., Solon, A. J., Waytowich, N. R., Gordon, S. M., Hung, C. P., and Lance,
B. J. (2018). EEGNet: a compact convolutional neural network for EEG-based brain–computer interfaces.
Journal of Neural Engineering, 15(5):056013. Publisher: IOP Publishing.
[Ledoit and Wolf, 2004] Ledoit, O. and Wolf, M. (2004). A well-conditioned estimator for large-dimensional
covariance matrices. Journal of Multivariate Analysis, 88(2):365–411.
[Lee et al., 2020] Lee, T., Kim, M., and Kim, S.-P. (2020). Data augmentation effects using borderline-SMOTE
on classification of a P300-based BCI. In 2020 8th International Winter Conference on Brain-Computer
Interface (BCI), pages 1–4. ISSN: 2572-7672.
[Lee et al., 2009] Lee, W. H., Liu, Z., Mueller, B. A., Lim, K., and He, B. (2009). Influence of white matter
anisotropic conductivity on EEG source localization: Comparison to fMRI in human primary visual cor-
tex. Clinical Neurophysiology, 120(12):2071–2081.
[Lemley et al., 2017] Lemley, J., Bazrafkan, S., and Corcoran, P. (2017). Smart Augmentation - Learning an
Optimal Data Augmentation Strategy. IEEE Access, 5:5858–5869. arXiv:1703.08383 [cs, stat].
[Lemm et al., 2005] Lemm, S., Blankertz, B., Curio, G., and Muller, K.-R. (2005). Spatio-spectral filters for
improving the classification of single trial EEG. IEEE Transactions on Biomedical Engineering, 52(9):1541–
1548. Conference Name: IEEE Transactions on Biomedical Engineering.
[Lemm et al., 2004] Lemm, S., Schafer, C., and Curio, G. (2004). BCI competition 2003-data set III: prob-
abilistic modeling of sensorimotor /spl mu/ rhythms for classification of imaginary hand movements.
IEEE Transactions on Biomedical Engineering, 51(6):1077–1080. Conference Name: IEEE Transactions on
Biomedical Engineering.
[León-Carrión et al., 2012] León-Carrión, J., León-Domínguez, U., León-Carrión, J., and León-Domínguez,
U. (2012). Functional Near-Infrared Spectroscopy (fNIRS): Principles and Neuroscientific Applications. In-
techOpen. Publication Title: Neuroimaging - Methods.
[Li and Zhang, 2010] Li, J. and Zhang, L. (2010). Bilateral adaptation and neurofeedback for brain computer
interface system. Journal of Neuroscience Methods, 193(2):373–379.
[Li et al., 2013] Li, Y., Pan, J., Wang, F., and Yu, Z. (2013). A Hybrid BCI System Combining P300 and SSVEP
and Its Application to Wheelchair Control. IEEE Transactions on Biomedical Engineering, 60(11):3156–
3166. Conference Name: IEEE Transactions on Biomedical Engineering.
[Li et al., 2020] Li, Y., Yang, H., Li, J., Chen, D., and Du, M. (2020). EEG-based intention recognition with
deep recurrent-convolution neural network: Performance and channel selection by Grad-CAM. Neuro-
computing, 415:225–233.
[Li et al., 2019] Li, Y., Zhang, X.-R., Zhang, B., Lei, M.-Y., Cui, W.-G., and Guo, Y.-Z. (2019). A Channel-
Projection Mixed-Scale Convolutional Neural Network for Motor Imagery EEG Decoding. IEEE transac-
tions on neural systems and rehabilitation engineering: a publication of the IEEE Engineering in Medicine
and Biology Society, 27(6):1170–1180.
[Liao et al., 2012] Liao, L.-D., Chen, C.-Y., Wang, I.-J., Chen, S.-F., Li, S.-Y., Chen, B.-W., Chang, J.-Y., and Lin,
C.-T. (2012). Gaming control using a wearable and wireless EEG-based brain-computer interface device
with novel dry foam-based sensors. Journal of NeuroEngineering and Rehabilitation, 9(1):5.
[Liao et al., 2007] Liao, X., Yao, D., Wu, D., and Li, C. (2007). Combining spatial filters for the classification of
single-trial EEG in a finger movement task. IEEE transactions on bio-medical engineering, 54(5):821–831.
[Lin and She, 2020] Lin, J.-S. and She, B.-H. (2020). A BCI System with Motor Imagery Based on Bidirectional
Long-Short Term Memory. IOP Conference Series: Materials Science and Engineering, 719(1):012026. Pub-
lisher: IOP Publishing.
[Lotte, 2015] Lotte, F. (2015). Signal Processing Approaches to Minimize or Suppress Calibration Time in
Oscillatory Activity-Based Brain–Computer Interfaces. Proceedings of the IEEE, 103(6):871–890.
104
[Lotte et al., 2018] Lotte, F., Bougrain, L., Cichocki, A., Clerc, M., Congedo, M., Rakotomamonjy, A., and Yger,
F. (2018). A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update.
Journal of Neural Engineering, 15(3):031005.
[Lotte et al., 2015] Lotte, F., Bougrain, L., and Clerc, M. (2015). Electroencephalography (EEG)-Based Brain-
Computer Interfaces. In Wiley Encyclopedia of Electrical and Electronics Engineering, pages 1–20. John
Wiley & Sons, Inc., Hoboken, NJ, USA.
[Lotte et al., 2007] Lotte, F., Congedo, M., Lécuyer, A., Lamarche, F., and Arnaldi, B. (2007). A review of clas-
sification algorithms for EEG-based brain–computer interfaces. Journal of Neural Engineering, 4(2):R1–
R13.
[Lotte and Guan, 2010] Lotte, F. and Guan, C. (2010). Learning from other subjects helps reducing Brain-
Computer Interface calibration time. In 2010 IEEE International Conference on Acoustics, Speech and
Signal Processing, pages 614–617, Dallas, TX, USA. IEEE.
[Lotte and Guan, 2011] Lotte, F. and Guan, C. (2011). Regularizing Common Spatial Patterns to Improve BCI
Designs: Unified Theory and New Algorithms. IEEE Transactions on Biomedical Engineering, 58(2):355–
362. Conference Name: IEEE Transactions on Biomedical Engineering.
[Lotte et al., 2009] Lotte, F., Guan, C., and Ang, K. K. (2009). Comparison of designs towards a subject-
independent brain-computer interface based on motor imagery. Annual International Conference of the
IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. An-
nual International Conference, 2009:4543–4546.
[Lu et al., 2017] Lu, N., Li, T., Ren, X., and Miao, H. (2017). A Deep Learning Scheme for Motor Imagery
Classification based on Restricted Boltzmann Machines. IEEE Transactions on Neural Systems and Re-
habilitation Engineering, 25(6):566–576. Conference Name: IEEE Transactions on Neural Systems and
Rehabilitation Engineering.
[Luhman and Luhman, 2021] Luhman, E. and Luhman, T. (2021). Knowledge Distillation in Iterative Gen-
erative Models for Improved Sampling Speed. arXiv:2101.02388 [cs].
[Luo et al., 2021] Luo, S., Chen, R., Yang, Z.-T., and Li, K. (2021). Intelligence level might be predicted by the
characteristics of EEG signals at specific frequencies and brain regions. Journal of Mechanics in Medicine
and Biology, 21:2140047.
[Luo et al., 2020] Luo, Y., Zhu, L.-Z., Wan, Z.-Y., and Lu, B.-L. (2020). Data augmentation for enhancing EEG-
based emotion recognition with deep generative models. Journal of Neural Engineering, 17(5):056021.
[Maaten and Hinton, 2008] Maaten, L. v. d. and Hinton, G. (2008). Visualizing Data using t-SNE. Journal of
Machine Learning Research, 9(86):2579–2605.
[Majidov and Whangbo, 2019] Majidov, I. and Whangbo, T. (2019). Efficient Classification of Motor Imagery
Electroencephalography Signals Using Deep Learning Methods. Sensors, 19(7):1736. Number: 7 Pub-
lisher: Multidisciplinary Digital Publishing Institute.
[Malmivuo and Plonsey, 1995] Malmivuo, J. and Plonsey, R. (1995). Bioelectromagnetism: Principles and
Applications of Bioelectric and Biomagnetic Fields. Oxford University Press.
[Mammone et al., 2020] Mammone, N., Ieracitano, C., and Morabito, F. C. (2020). A deep CNN approach
to decode motor preparation of upper limbs from time–frequency maps of EEG signals at source level.
Neural Networks, 124:357–372.
[Matsuura and Okabe, 1995] Matsuura, K. and Okabe, Y. (1995). Selective minimum-norm solution of the
biomagnetic inverse problem. IEEE transactions on bio-medical engineering, 42(6):608–615.
[McConnell et al., 2009] McConnell, G. C., Rees, H. D., Levey, A. I., Gutekunst, C.-A., Gross, R. E., and Bel-
lamkonda, R. V. (2009). Implanted neural electrodes cause chronic, local inflammation that is correlated
with local neurodegeneration. Journal of Neural Engineering, 6(5):056003.
105
[McFarland et al., 1997] McFarland, D. J., McCane, L. M., David, S. V., and Wolpaw, J. R. (1997). Spatial
filter selection for EEG-based communication. Electroencephalography and Clinical Neurophysiology,
103(3):386–394.
[McMenamin et al., 2010] McMenamin, B. W., Shackman, A. J., Maxwell, J. S., Bachhuber, D. R., Koppen-
haver, A. M., Greischar, L. L., and Davidson, R. J. (2010). Validation of ICA-based myogenic artifact cor-
rection for scalp and source-localized EEG. NeuroImage, 49(3):2416–2432.
[Michel and Brunet, 2019] Michel, C. M. and Brunet, D. (2019). EEG Source Imaging: A Practical Review of
the Analysis Steps. Frontiers in Neurology, 10.
[Michel and He, 2019] Michel, C. M. and He, B. (2019). Chapter 6 - EEG source localization. In Levin, K. H.
and Chauvel, P., editors, Handbook of Clinical Neurology, volume 160 of Clinical Neurophysiology: Basis
and Technical Aspects, pages 85–101. Elsevier.
[Michel and Murray, 2012] Michel, C. M. and Murray, M. M. (2012). Towards the utilization of EEG as a brain
imaging tool. NeuroImage, 61(2):371–385.
[Michel et al., 2004] Michel, C. M., Murray, M. M., Lantz, G., Gonzalez, S., Spinelli, L., and Grave de Peralta,
R. (2004). EEG source imaging. Clinical Neurophysiology: Official Journal of the International Federation
of Clinical Neurophysiology, 115(10):2195–2222.
[Minguillon et al., 2017] Minguillon, J., Lopez-Gordo, M. A., and Pelayo, F. (2017). Trends in EEG-BCI for
daily-life: Requirements for artifact removal. Biomedical Signal Processing and Control, C(31):407–418.
[Minh et al., 2021] Minh, T. N., Sinn, M., Lam, H. T., and Wistuba, M. (2021). Automated Image Data Prepro-
cessing with Deep Reinforcement Learning. arXiv:1806.05886 [cs].
[Mirza and Osindero, 2014] Mirza, M. and Osindero, S. (2014). Conditional Generative Adversarial Nets.
arXiv:1411.1784 [cs, stat].
[Moghadamfalahi et al., 2015] Moghadamfalahi, M., Orhan, U., Akcakaya, M., Nezamfar, H., Fried-Oken,
M., and Erdogmus, D. (2015). Language-model assisted brain computer interface for typing: A compari-
son of matrix and rapid serial visual presentation. IEEE Transactions on Neural Systems and Rehabilitation
Engineering, 23(5):910–920.
[Mogren, 2016] Mogren, O. (2016). C-RNN-GAN: Continuous recurrent neural networks with adversarial
training. arXiv:1611.09904 [cs].
[Mohsenvand et al., 2020] Mohsenvand, M. N., Izadi, M. R., and Maes, P. (2020). Contrastive Representation
Learning for Electroencephalogram Classification. In Proceedings of the Machine Learning for Health
NeurIPS Workshop, pages 238–253. PMLR. ISSN: 2640-3498.
[Momen et al., 2007] Momen, K., Krishnan, S., and Chau, T. (2007). Real-Time Classification of Forearm
Electromyographic Signals Corresponding to User-Selected Intentional Movements for Multifunction
Prosthesis Control. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 15(4):535–542.
Conference Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering.
[Mosher and Leahy, 1999] Mosher, J. and Leahy, R. (1999). Source localization using recursively applied and
projected (RAP) MUSIC. IEEE Transactions on Signal Processing, 47(2):332–340. Conference Name: IEEE
Transactions on Signal Processing.
[Mosher et al., 1992] Mosher, J., Lewis, P., and Leahy, R. (1992). Multiple dipole modeling and localization
from spatio-temporal MEG data. IEEE Transactions on Biomedical Engineering, 39(6):541–557. Confer-
ence Name: IEEE Transactions on Biomedical Engineering.
[Mulder, 2007] Mulder, T. (2007). Motor imagery and action observation: cognitive tools for rehabilitation.
Journal of Neural Transmission, 114(10):1265–1278.
[Neuper and Pfurtscheller, 2001] Neuper, C. and Pfurtscheller, G. (2001). Event-related dynamics of cortical
rhythms: frequency-specific features and functional correlates. International Journal of Psychophysiol-
ogy: Official Journal of the International Organization of Psychophysiology, 43(1):41–58.
106
[Neuper et al., 2005] Neuper, C., Scherer, R., Reiner, M., and Pfurtscheller, G. (2005). Imagery of motor ac-
tions: differential effects of kinesthetic and visual-motor mode of imagery in single-trial EEG. Brain Re-
search. Cognitive Brain Research, 25(3):668–677.
[Nguyen et al., 2017] Nguyen, D., Tran, D., Sharma, D., and Ma, W. (2017). On The Study of EEG-based
Cryptographic Key Generation. Procedia Computer Science, 112:936–945.
[Nguyen, 2013] Nguyen, T. (2013). Total Number of Synapses in the Adult Human Neocortex. Undergradu-
ate Journal of Mathematical Modeling: One + Two, 3(1).
[Nichol and Dhariwal, 2021] Nichol, A. and Dhariwal, P. (2021). Improved Denoising Diffusion Probabilistic
Models. arXiv:2102.09672 [cs, stat].
[Niediek et al., 2016] Niediek, J., Boström, J., Elger, C. E., and Mormann, F. (2016). Reliable Analysis of Single-
Unit Recordings from the Human Brain under Noisy Conditions: Tracking Neurons over Hours. PLoS
ONE, 11(12):e0166598.
[Nikulin et al., 2011] Nikulin, V. V., Nolte, G., and Curio, G. (2011). A novel method for reliable and fast ex-
traction of neuronal EEG/MEG oscillations on the basis of spatio-spectral decomposition. NeuroImage,
55(4):1528–1535.
[Odena et al., 2017] Odena, A., Olah, C., and Shlens, J. (2017). Conditional Image Synthesis With Auxiliary
Classifier GANs. arXiv:1610.09585 [cs, stat].
[Oikonomou et al., 2017] Oikonomou, V. P., Georgiadis, K., Liaros, G., Nikolopoulos, S., and Kompatsiaris, I.
(2017). A Comparison Study on EEG Signal Processing Techniques Using Motor Imagery EEG Data. In
2017 IEEE 30th International Symposium on Computer-Based Medical Systems (CBMS), pages 781–786,
Thessaloniki. IEEE.
[Omedes et al., 2015] Omedes, J., Iturrate, I., Chavarriaga, R., and Montesano, L. (2015). Asynchronous De-
coding of Error Potentials during the Monitoring of a Reaching Task. In 2015 IEEE International Confer-
ence on Systems, Man, and Cybernetics, pages 3116–3121.
[Oord et al., 2016] Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner,
N., Senior, A., and Kavukcuoglu, K. (2016). WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499
[cs].
[Ortiz-Echeverri et al., 2019] Ortiz-Echeverri, C. J., Salazar-Colores, S., Rodríguez-Reséndiz, J., and Gómez-
Loenzo, R. A. (2019). A New Approach for Motor Imagery Classification Based on Sorted Blind Source
Separation, Continuous Wavelet Transform, and Convolutional Neural Network. Sensors, 19(20):4541.
Number: 20 Publisher: Multidisciplinary Digital Publishing Institute.
[Padfield et al., 2019] Padfield, N., Zabalza, J., Zhao, H., Masero, V., and Ren, J. (2019). EEG-Based Brain-
Computer Interfaces Using Motor-Imagery: Techniques and Challenges. Sensors (Basel, Switzerland),
19(6):1423.
[Palaniappan, 2005] Palaniappan, R. (2005). Brain Computer Interface Design Using Band Powers Extracted
During Mental Tasks. In Conference Proceedings. 2nd International IEEE EMBS Conference on Neural En-
gineering, 2005., pages 321–324, Arlington, Virginia, USA. IEEE.
[Pan et al., 2022] Pan, J., Chen, X., Ban, N., He, J., Chen, J., and Huang, H. (2022). Advances in P300
brain–computer interface spellers: toward paradigm design and performance evaluation. Frontiers in
Human Neuroscience, 16.
[Panwar et al., 2020] Panwar, S., Rad, P., Jung, T.-P., and Huang, Y. (2020). Modeling EEG Data Distribution
With a Wasserstein Generative Adversarial Network to Predict RSVP Events. IEEE Transactions on Neu-
ral Systems and Rehabilitation Engineering, 28(8):1720–1730. Conference Name: IEEE Transactions on
Neural Systems and Rehabilitation Engineering.
[Parra et al., 2005] Parra, L. C., Spence, C. D., Gerson, A. D., and Sajda, P. (2005). Recipes for the linear anal-
ysis of EEG. NeuroImage, 28(2):326–341.
107
[Pascual-Marqui, 2002] Pascual-Marqui, R. D. (2002). Standardized low-resolution brain electromagnetic
tomography (sLORETA): technical details. Methods and Findings in Experimental and Clinical Pharma-
cology, 24 Suppl D:5–12.
[Pascual-Marqui et al., 1994] Pascual-Marqui, R. D., Michel, C. M., and Lehmann, D. (1994). Low resolu-
tion electromagnetic tomography: a new method for localizing electrical activity in the brain. Interna-
tional Journal of Psychophysiology: Official Journal of the International Organization of Psychophysiology,
18(1):49–65.
[Perez and Wang, 2017] Perez, L. and Wang, J. (2017). The Effectiveness of Data Augmentation in Image
Classification using Deep Learning. arXiv:1712.04621 [cs].
[Pfurtscheller and Neuper, 1997] Pfurtscheller, G. and Neuper, C. (1997). Motor imagery activates primary
sensorimotor area in humans. Neuroscience Letters, 239(2):65–68.
[Pfurtscheller and Neuper, 2001] Pfurtscheller, G. and Neuper, C. (2001). Motor imagery and direct brain-
computer communication. Proceedings of the IEEE, 89(7):1123–1134. Conference Name: Proceedings of
the IEEE.
[Pillette et al., 2021] Pillette, L., Roc, A., N’Kaoua, B., and Lotte, F. (2021). Experimenters’ Influence
on Mental-Imagery based Brain-Computer Interface User Training. International Journal of Human-
Computer Studies, 149:102603.
[Portillo-Lara et al., 2021] Portillo-Lara, R., Tahirbegi, B., Chapman, C. A. R., Goding, J. A., and Green, R. A.
(2021). Mind the gap: State-of-the-art technologies and applications for EEG-based brain–computer in-
terfaces. APL Bioengineering, 5(3):031507. Publisher: American Institute of Physics.
[Pravdich-Neminsky, 1913] Pravdich-Neminsky, V. (1913). Experiments on the registration of the electrical
phenomena of the mammalian brain. Zbl. Physiol, 27:951–960.
[Radford et al., 2016] Radford, A., Metz, L., and Chintala, S. (2016). Unsupervised Representation Learning
with Deep Convolutional Generative Adversarial Networks. arXiv:1511.06434 [cs].
[Rao, 2013] Rao, R. P. N. (2013). Brain-Computer Interfacing: An Introduction. Recording and Stimulating
the Brain. Cambridge University Press, 1 edition.
[Rashid et al., 2020] Rashid, M., Sulaiman, N., P. P. Abdul Majeed, A., Musa, R. M., Ab. Nasir, A. F., Bari, B. S.,
and Khatun, S. (2020). Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer
Interface: A Comprehensive Review. Frontiers in Neurorobotics, 14.
[Rawnaque et al., 2020] Rawnaque, F. S., Rahman, K. M., Anwar, S. F., Vaidyanathan, R., Chau, T., Sarker,
F., and Mamun, K. A. A. (2020). Technological advancements and opportunities in Neuromarketing: a
systematic review. Brain Informatics, 7(1):10.
[Robinson, 1999] Robinson, S. (1999). Functional neuroimaging by Synthetic Aperture Magnetometry
(SAM). Recent Advances in Biomagnetism.
[Rodríguez-Bermúdez and García-Laencina, 2012] Rodríguez-Bermúdez, G. and García-Laencina, P. J.
(2012). Automatic and adaptive classification of electroencephalographic signals for brain computer in-
terfaces. Journal of Medical Systems, 36 Suppl 1:S51–63.
[Rommel et al., 2022a] Rommel, C., Moreau, T., Paillard, J., and Gramfort, A. (2022a). CADDA: Class-wise
Automatic Differentiable Data Augmentation for EEG Signals. arXiv:2106.13695 [cs].
[Rommel et al., 2022b] Rommel, C., Paillard, J., Moreau, T., and Gramfort, A. (2022b). Data augmenta-
tion for learning predictive models on EEG: a systematic comparison. Journal of Neural Engineering,
19(6):066020. Publisher: IOP Publishing.
[Roy et al., 2019] Roy, Y., Banville, H., Albuquerque, I., Gramfort, A., Falk, T. H., and Faubert, J. (2019). Deep
learning-based electroencephalography analysis: a systematic review. Journal of Neural Engineering,
16(5):051001.
108
[Rubaiy, 2017] Rubaiy, H. N. (2017). A Short Guide to Electrophysiology and Ion Channels. Journal of Phar-
macy & Pharmaceutical Sciences, 20:48–67.
[Rumelhart et al., 1986] Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations
by back-propagating errors. Nature, 323(6088):533–536. Number: 6088 Publisher: Nature Publishing
Group.
[Ru¸sanu et al., 2020] Ru¸sanu, O. A., Cristea, L., and Luculescu, M. C. (2020). Arduino based mobile robot
controlled by voluntary eye-blinks using LabVIEW GUI & NeuroSky Mindwave Mobile Headset. IOP Con-
ference Series: Materials Science and Engineering, 997(1):012059.
[Saeed et al., 2021] Saeed, A., Grangier, D., Pietquin, O., and Zeghidour, N. (2021). Learning From Hetero-
geneous Eeg Signals with Differentiable Channel Reordering. In ICASSP 2021 - 2021 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1255–1259. ISSN: 2379-190X.
[Saha and Baumert, 2020] Saha, S. and Baumert, M. (2020). Intra- and Inter-subject Variability in EEG-
Based Sensorimotor Brain Computer Interface: A Review. Frontiers in Computational Neuroscience, 13.
[Saibene and Gasparini, 2023] Saibene, A. and Gasparini, F. (2023). Genetic algorithm for feature selection
of EEG heterogeneous data. Expert Systems with Applications, 217:119488.
[Salami et al., 2022] Salami, A., Andreu-Perez, J., and Gillmeister, H. (2022). EEG-ITNet: An Explainable In-
ception Temporal Convolutional Network for Motor Imagery Classification. IEEE Access, 10:36672–36685.
arXiv:2204.06947 [cs, q-bio].
[Salamon and Bello, 2017] Salamon, J. and Bello, J. P. (2017). Deep Convolutional Neural Networks and
Data Augmentation for Environmental Sound Classification. IEEE Signal Processing Letters, 24(3):279–
283. Conference Name: IEEE Signal Processing Letters.
[Salimans and Ho, 2022] Salimans, T. and Ho, J. (2022). Progressive Distillation for Fast Sampling of Diffu-
sion Models.
[Samuel et al., 2017] Samuel, O. W., Geng, Y., Li, X., and Li, G. (2017). Towards Efficient Decoding of Multiple
Classes of Motor Imagery Limb Movements Based on EEG Spectral and Time Domain Descriptors. Journal
of Medical Systems, 41(12):194.
[Sannelli et al., 2019] Sannelli, C., Vidaurre, C., Müller, K.-R., and Blankertz, B. (2019). A large scale screening
study with a SMR-based BCI: Categorization of BCI users and differences in their SMR activity. PLoS ONE,
14(1):e0207351.
[Scarapicchia et al., 2017] Scarapicchia, V., Brown, C., Mayo, C., and Gawryluk, J. R. (2017). Functional Mag-
netic Resonance Imaging and Functional Near-Infrared Spectroscopy: Insights from Combined Record-
ing Studies. Frontiers in Human Neuroscience, 11:419.
[Schirrmeister et al., 2017] Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M.,
Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., and Ball, T. (2017). Deep learning with convo-
lutional neural networks for EEG decoding and visualization. Human Brain Mapping, 38(11):5391–5420.
[Schlögl et al., 2010] Schlögl, A., Vidaurre, C., and Müller, K.-R. (2010). Adaptive Methods in BCI Research -
An Introductory Tutorial. In Graimann, B., Pfurtscheller, G., and Allison, B., editors, Brain-Computer Inter-
faces: Revolutionizing Human-Computer Interaction, The Frontiers Collection, pages 331–355. Springer,
Berlin, Heidelberg.
[Schmidt et al., 2012] Schmidt, N. M., Blankertz, B., and Treder, M. S. (2012). Online detection of error-
related potentials boosts the performance of mental typewriters. BMC Neuroscience, 13(1):19.
[Schwabedal et al., 2019] Schwabedal, J. T. C., Snyder, J. C., Cakmak, A., Nemati, S., and Clifford, G. D. (2019).
Addressing Class Imbalance in Classification Problems of Noisy Signals by using Fourier Transform Sur-
rogates. arXiv:1806.08675 [nlin, q-bio, stat].
[Schäfer and Strimmer, 2005] Schäfer, J. and Strimmer, K. (2005). A shrinkage approach to large-scale co-
variance matrix estimation and implications for functional genomics. Statistical Applications in Genetics
and Molecular Biology, 4:Article32.
109
[Sekihara et al., 2001] Sekihara, K., Nagarajan, S. S., Poeppel, D., Marantz, A., and Miyashita, Y. (2001). Re-
constructing spatio-temporal activities of neural sources using an MEG vector beamformer technique.
IEEE transactions on bio-medical engineering, 48(7):760–771.
[Shi et al., 2012] Shi, J.-h., Shen, J.-z., Ji, Y., and Du, F.-l. (2012). A submatrix-based P300 brain-computer
interface stimulus presentation paradigm. Journal of Zhejiang University SCIENCE C, 13(6):452–459.
[Shorten and Khoshgoftaar, 2019] Shorten, C. and Khoshgoftaar, T. M. (2019). A survey on Image Data Aug-
mentation for Deep Learning. Journal of Big Data, 6(1):60.
[Shovon et al., 2019] Shovon, T. H., Nazi, Z. A., Dash, S., and Hossain, M. F. (2019). Classification of Motor
Imagery EEG Signals with multi-input Convolutional Neural Network by augmenting STFT. In 2019 5th
International Conference on Advances in Electrical Engineering (ICAEE), pages 398–403. ISSN: 2378-2692.
[Silvoni et al., 2011] Silvoni, S., Ramos-Murguialday, A., Cavinato, M., Volpato, C., Cisotto, G., Turolla, A.,
Piccione, F., and Birbaumer, N. (2011). Brain-Computer Interface in Stroke: A Review of Progress. Clinical
EEG and Neuroscience, 42(4):245–252. Publisher: SAGE Publications Inc.
[Singh et al., 2021] Singh, A., Hussain, A. A., Lal, S., and Guesgen, H. W. (2021). A Comprehensive Review on
Critical Issues and Possible Solutions of Motor Imagery Based Electroencephalography Brain-Computer
Interface. Sensors (Basel, Switzerland), 21(6):2173.
[Solhjoo and Moradi, 2004] Solhjoo, S. and Moradi, M. (2004). Mental Task Recognition: A Comparison be-
tween Some of Classification Methods. Biosignal. Pages: 26.
[Stancin et al., 2021] Stancin, I., Cifrek, M., and Jovic, A. (2021). A Review of EEG Signal Features and Their
Application in Driver Drowsiness Detection Systems. Sensors, 21(11):3786. Number: 11 Publisher: Multi-
disciplinary Digital Publishing Institute.
[Subirats et al., 2018] Subirats, L., Allali, G., Briansoulet, M., Salle, J. Y., and Perrochon, A. (2018). Age and
gender differences in motor imagery. Journal of the Neurological Sciences, 391:114–117.
[Sur and Sinha, 2009] Sur, S. and Sinha, V. K. (2009). Event-related potential: An overview. Industrial Psychi-
atry Journal, 18(1):70–73.
[Symeonidou et al., 2018] Symeonidou, E.-R., Nordin, A. D., Hairston, W. D., and Ferris, D. P. (2018). Effects
of Cable Sway, Electrode Surface Area, and Electrode Mass on Electroencephalography Signal Quality
during Motion. Sensors, 18(4):1073. Number: 4 Publisher: Multidisciplinary Digital Publishing Institute.
[Tabar and Halici, 2016] Tabar, Y. R. and Halici, U. (2016). A novel deep learning approach for classification
of EEG motor imagery signals. Journal of Neural Engineering, 14(1):016003. Publisher: IOP Publishing.
[Taheri et al., 2020] Taheri, S., Ezoji, M., and Sakhaei, S. M. (2020). Convolutional neural network based
features for motor imagery EEG signals classification in brain–computer interface system. SN Applied
Sciences, 2(4):555.
[Tangermann, 2007] Tangermann, M. W. (2007). Feature Selection for Brain-Computer Interfaces. Disserta-
tion, Universität Tübingen. Accepted: 2007-09-26.
[Tangermann et al., 2008] Tangermann, M. W., Krauledat, M., Grzeska, K., Sagebaum, M., Vidaurre, C.,
Blankertz, B., and Müller, K.-R. (2008). Playing pinball with non-invasive BCI. In Proceedings of the 21st
International Conference on Neural Information Processing Systems, NIPS’08, pages 1641–1648, Red Hook,
NY, USA. Curran Associates Inc.
[Tayeb et al., 2019] Tayeb, Z., Fedjaev, J., Ghaboosi, N., Richter, C., Everding, L., Qu, X., Wu, Y., Cheng, G., and
Conradt, J. (2019). Validating Deep Neural Networks for Online Decoding of Motor Imagery Movements
from EEG Signals. Sensors (Basel, Switzerland), 19(1):210.
[Tian and Liu, 2019] Tian, G. and Liu, Y. (2019). Simple Convolutional Neural Network for Left-Right Hands
Motor Imagery EEG Signals Classification. International Journal of Cognitive Informatics and Natural
Intelligence, 13(3):36–49.
110
[Tomioka and Müller, 2010] Tomioka, R. and Müller, K.-R. (2010). A regularized discriminative framework
for EEG analysis with application to brain–computer interface. NeuroImage, 49(1):415–432.
[Torma and Szegletes, 2023] Torma, S. and Szegletes, D. L. (2023). Brain Signal Generation and Data Aug-
mentation with a Single-Step Diffusion Probabilistic Model. ICLR 2023.
[Torres et al., 2020] Torres, E. P., Torres, E. A., Hernández-Álvarez, M., and Yoo, S. G. (2020). EEG-Based BCI
Emotion Recognition: A Survey. Sensors, 20(18):5083. Number: 18 Publisher: Multidisciplinary Digital
Publishing Institute.
[Tosato et al., 2023] Tosato, G., Dalbagno, C. M., and Fumagalli, F. (2023). EEG Synthetic Data Generation
Using Probabilistic Diffusion Models. arXiv:2303.06068 [cs, eess, q-bio].
[Townsend et al., 2010] Townsend, G., LaPallo, B., Boulay, C., Krusienski, D., Frye, G., Hauser, C., Schwartz,
N., Vaughan, T., Wolpaw, J., and Sellers, E. (2010). A novel P300-based brain-computer interface stimulus
presentation paradigm: moving beyond rows and columns. Clinical neurophysiology : official journal of
the International Federation of Clinical Neurophysiology, 121(7):1109–1120.
[Treder and Blankertz, 2010] Treder, M. S. and Blankertz, B. (2010). (C)overt attention and visual speller
design in an ERP-based brain-computer interface. Behavioral and Brain Functions, 6. Place: United
Kingdom Publisher: BioMed Central Limited.
[Trujillo-Barreto et al., 2004] Trujillo-Barreto, N. J., Aubert-Vázquez, E., and Valdés-Sosa, P. A. (2004).
Bayesian model averaging in EEG/MEG imaging. NeuroImage, 21(4):1300–1319.
[Ujma et al., 2019] Ujma, P. P., Konrad, B. N., Simor, P., Gombos, F., Körmendi, J., Steiger, A., Dresler, M., and
Bódizs, R. (2019). Sleep EEG functional connectivity varies with age and sex, but not general intelligence.
Neurobiology of Aging, 78:87–97.
[Urigüen and Garcia-Zapirain, 2015] Urigüen, J. A. and Garcia-Zapirain, B. (2015). EEG artifact
removal—state-of-the-art and guidelines. Journal of Neural Engineering, 12(3):031001. Publisher: IOP
Publishing.
[van der Waal et al., 2012] van der Waal, M., Severens, M., Geuze, J., and Desain, P. (2012). Introducing the
tactile speller: an ERP-based brain-computer interface for communication. Journal of Neural Engineer-
ing, 9(4):045002.
[Van Erp et al., 2012] Van Erp, J., Lotte, F., and Tangermann, M. (2012). Brain-Computer Interfaces: Beyond
Medical Applications. Computer, 45(4):26–34.
[Van Essen et al., 2019] Van Essen, D. C., Donahue, C. J., Coalson, T. S., Kennedy, H., Hayashi, T., and Glasser,
M. F. (2019). Cerebral cortical folding, parcellation, and connectivity in humans, nonhuman primates,
and mice. Proceedings of the National Academy of Sciences of the United States of America, 116(52):26173–
26180.
[Van Veen et al., 1997] Van Veen, B. D., van Drongelen, W., Yuchtman, M., and Suzuki, A. (1997). Localization
of brain electrical activity via linearly constrained minimum variance spatial filtering. IEEE transactions
on bio-medical engineering, 44(9):867–880.
[Vansteensel et al., 2016] Vansteensel, M. J., Pels, E. G., Bleichner, M. G., Branco, M. P., Denison, T., Freuden-
burg, Z. V., Gosselaar, P., Leinders, S., Ottens, T. H., Van Den Boom, M. A., Van Rijen, P. C., Aarnoutse, E. J.,
and Ramsey, N. F. (2016). Fully Implanted Brain–Computer Interface in a Locked-In Patient with ALS. The
New England journal of medicine, 375(21):2060–2066.
[Veitinger et al., 2023] Veitinger, S., DeRose, J., and Greb, C. (2023). What is the Patch-Clamp Technique?
[Vidal, 1973] Vidal, J. J. (1973). Toward Direct Brain-Computer Communica-
tion. Annual Review of Biophysics and Bioengineering, 2(1):157–180. _eprint:
https://doi.org/10.1146/annurev.bb.02.060173.001105.
[Vidaurre et al., 2011a] Vidaurre, C., Kawanabe, M., von Bünau, P., Blankertz, B., and Müller, K. R. (2011a).
Toward Unsupervised Adaptation of LDA for Brain–Computer Interfaces. IEEE Transactions on Biomedi-
cal Engineering, 58(3):587–597. Conference Name: IEEE Transactions on Biomedical Engineering.
111
[Vidaurre et al., 2009] Vidaurre, C., Krämer, N., Blankertz, B., and Schlögl, A. (2009). Time Domain Parame-
ters as a feature for EEG-based Brain–Computer Interfaces. Neural Networks, 22(9):1313–1319.
[Vidaurre et al., 2011b] Vidaurre, C., Sannelli, C., Müller, K.-R., and Blankertz, B. (2011b). Co-adaptive cali-
bration to improve BCI efficiency. Journal of Neural Engineering, 8(2):025009.
[Vidaurre et al., 2011c] Vidaurre, C., Sannelli, C., Müller, K.-R., and Blankertz, B. (2011c). Machine-
Learning-Based Coadaptive Calibration for Brain-Computer Interfaces. Neural Computation, 23(3):791–
816.
[Villamar et al., 2018] Villamar, M. F., Al-Bakri, A. F., Haddix, C., Albuja, A. C., Bensalem-Owen, M., and Sun-
deram, S. (2018). T157. Seizure prediction with autonomic measurements versus intracranial EEG in
patients with refractory epilepsy. Clinical Neurophysiology, 129:e63.
[Visconti et al., 2008] Visconti, G., Seno, B. D., Matteucci, M., and Mainardi, L. (2008). Automatic Recogni-
tion of Error Potentials in a P300-Based Brain-Computer Interface.
[Wakai, 2014] Wakai, R. T. (2014). The atomic magnetometer: A new era in biomagnetism. pages 46–54,
León, Guanajuato, Mexico.
[Wang et al., 2020] Wang, F., Wu, S., Zhang, W., Xu, Z., Zhang, Y., Wu, C., and Coleman, S. (2020). Emotion
recognition with convolutional neural network and EEG-based EFDMs. Neuropsychologia, 146:107506.
[Wang et al., 2018a] Wang, F., Zhong, S.-h., Peng, J., Jiang, J., and Liu, Y. (2018a). Data Augmentation for
EEG-Based Emotion Recognition with Deep Convolutional Neural Networks. In Schoeffmann, K., Chal-
idabhongse, T. H., Ngo, C. W., Aramvith, S., O’Connor, N. E., Ho, Y.-S., Gabbouj, M., and Elgammal, A.,
editors, MultiMedia Modeling, Lecture Notes in Computer Science, pages 82–93, Cham. Springer Interna-
tional Publishing.
[Wang et al., 1992] Wang, J. Z., Williamson, S. J., and Kaufman, L. (1992). Magnetic source images deter-
mined by a lead-field analysis: the unique minimum-norm least-squares estimation. IEEE transactions
on bio-medical engineering, 39(7):665–675.
[Wang et al., 2009] Wang, W., Degenhart, A. D., Collinger, J. L., Vinjamuri, R., Sudre, G. P., Adelson, P. D.,
Holder, D. L., Leuthardt, E. C., Moran, D. W., Boninger, M. L., Schwartz, A. B., Crammond, D. J., Tyler-
Kabara, E. C., and Weber, D. J. (2009). Human Motor Cortical Activity Recorded with Micro-ECoG Elec-
trodes During Individual Finger Movements. Conference proceedings : ... Annual International Conference
of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society.
Conference, 2009:586–589.
[Wang et al., 2014] Wang, Y., Li, X., Li, H., Shao, C., Ying, L., and Wu, S. (2014). Feature extraction of motor
imagery electroencephalography based on time-frequency-space domains. Sheng Wu Yi Xue Gong Cheng
Xue Za Zhi = Journal of Biomedical Engineering = Shengwu Yixue Gongchengxue Zazhi, 31(5):955–961.
[Wang et al., 2018b] Wang, Z., Cao, L., Zhang, Z., Gong, X., Sun, Y., and Wang, H. (2018b). Short
time Fourier transformation and deep neural networks for motor imagery brain computer inter-
face recognition. Concurrency and Computation: Practice and Experience, 30(23):e4413. _eprint:
https://onlinelibrary.wiley.com/doi/pdf/10.1002/cpe.4413.
[Wang et al., 2019] Wang, Z., Yu, Y., Xu, M., Liu, Y., Yin, E., and Zhou, Z. (2019). Towards a Hybrid BCI Gaming
Paradigm Based on Motor Imagery and SSVEP. International Journal of Human–Computer Interaction,
35(3):197–205. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10447318.2018.1445068.
[Williams, 2006] Williams, J. (2006). Regularised CSP for Sensor Selection in BCI. Proceedings of the 3rd
International Brain-Computer Interface Workshop and Training Course 2006.
[Williamson et al., 2017] Williamson, J., Schneiders, M., Sciascio, C. d., Mueller-Putz, G., Ofner, P., Schwarz,
A., Pereira, J., Hessing, B., Luzhnica, G., Veas, E., Stein, S., Murray-Smith, R., Escolano, C., Montesano, L.,
and Rupp, R. (2017). Moregrasp: Restoration Of Upper Limb Function In Individuals With High Spinal
Cord Injury By Multimodal Neuroprostheses For Interaction In Daily Activities. 7th Graz Brain-Computer
Interface Conference 2017. ISBN: 9783851255331 Publisher: Verlag der Technischen Universität Graz.
112
[Woehrle et al., 2015] Woehrle, H., Krell, M. M., Straube, S., Kim, S. K., Kirchner, E. A., and Kirchner, F.
(2015). An Adaptive Spatial Filter for User-Independent Single Trial Detection of Event-Related Potentials.
IEEE Transactions on Biomedical Engineering, 62(7):1696–1705. Conference Name: IEEE Transactions on
Biomedical Engineering.
[Xie and Oniga, 2023] Xie, Y. and Oniga, S. (2023). Classification of Motor Imagery EEG Signals Based on
Data Augmentation and Convolutional Neural Networks. Sensors, 23(4):1932. Number: 4 Publisher: Mul-
tidisciplinary Digital Publishing Institute.
[Xu et al., 2020] Xu, J., Zheng, H., Wang, J., Li, D., and Fang, X. (2020). Recognition of EEG Signal Motor Im-
agery Intention Based on Deep Multi-View Feature Learning. Sensors, 20(12):3496. Number: 12 Publisher:
Multidisciplinary Digital Publishing Institute.
[Xu et al., 2004] Xu, X.-L., Xu, B., and He, B. (2004). An Alternative Subspace Approach to EEG Dipole Source
Localization. Physics in Medicine and Biology 49: 327-343. Physics in medicine and biology, 49:327–43.
[Yang et al., 2018] Yang, C., Wu, H., Li, Z., He, W., Wang, N., and Su, C.-Y. (2018). Mind Control of a Robotic
Arm With Visual Fusion Technology. IEEE Transactions on Industrial Informatics, 14(9):3822–3830. Con-
ference Name: IEEE Transactions on Industrial Informatics.
[Yang et al., 2020a] Yang, D., Liu, Y., Zhou, Z., Yu, Y., and Liang, X. (2020a). Decoding Visual Motions from
EEG Using Attention-Based RNN. Applied Sciences, 10(16):5662. Number: 16 Publisher: Multidisciplinary
Digital Publishing Institute.
[Yang and Honavar, 1998] Yang, J. and Honavar, V. (1998). Feature Subset Selection Using a Genetic Algo-
rithm. In Liu, H. and Motoda, H., editors, Feature Extraction, Construction and Selection: A Data Min-
ing Perspective, The Springer International Series in Engineering and Computer Science, pages 117–136.
Springer US, Boston, MA.
[Yang et al., 2020b] Yang, J., Ma, Z., Wang, J., and Fu, Y. (2020b). A Novel Deep Learning Scheme for Motor
Imagery EEG Decoding Based on Spatial Representation Fusion. IEEE Access, 8:202100–202110.
[Yang et al., 2022] Yang, S., Xiao, W., Zhang, M., Guo, S., Zhao, J., and Shen, F. (2022). Image Data Augmen-
tation for Deep Learning: A Survey. arXiv:2204.08610 [cs].
[Yao, 2000] Yao, D. (2000). Electric potential produced by a dipole in a homogeneous conducting sphere.
IEEE Transactions on Biomedical Engineering, 47(7):964–966. Conference Name: IEEE Transactions on
Biomedical Engineering.
[Yger et al., 2017] Yger, F., Berar, M., and Lotte, F. (2017). Riemannian Approaches in Brain-Computer Inter-
faces: A Review. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 25(10):1753–1762.
Conference Name: IEEE Transactions on Neural Systems and Rehabilitation Engineering.
[Yin and Zhang, 2017] Yin, Z. and Zhang, J. (2017). Cross-session classification of mental workload levels
using EEG and an adaptive deep learning model. Biomedical Signal Processing and Control, 33:30–47.
[Yu et al., 2014] Yu, X., Chum, P., and Sim, K.-B. (2014). Analysis the effect of PCA for feature reduction in
non-stationary EEG based motor imagery of BCI system. Optik, 125(3):1498–1502.
[Yun et al., 2013] Yun, Y., Gu, I. Y.-H., and Aghajan, H. (2013). Riemannian manifold-based support vector
machine for human activity classification in images. In 2013 IEEE International Conference on Image
Processing, pages 3466–3469. ISSN: 2381-8549.
[Zhang et al., 2021a] Zhang, C., Kim, Y.-K., and Eskandarian, A. (2021a). EEG-Inception: An Accurate and
Robust End-to-End Neural Network for EEG-based Motor Imagery Classification. arXiv:2101.10932 [cs,
eess].
[Zhang et al., 2020] Zhang, K., Xu, G., Han, Z., Ma, K., Zheng, X., Chen, L., Duan, N., and Zhang, S. (2020).
Data Augmentation for Motor Imagery Signal Classification Based on a Hybrid Neural Network. Sensors,
20(16):4485. Number: 16 Publisher: Multidisciplinary Digital Publishing Institute.
113
[Zhang et al., 2015] Zhang, R., Xu, P., Chen, R., Li, F., Guo, L., Li, P., Zhang, T., and Yao, D. (2015). Predict-
ing Inter-session Performance of SMR-Based Brain–Computer Interface Using the Spectral Entropy of
Resting-State EEG. Brain Topography, 28(5):680–690.
[Zhang et al., 2021b] Zhang, R., Zong, Q., Dou, L., Zhao, X., Tang, Y., and Li, Z. (2021b). Hybrid deep neu-
ral network using transfer learning for EEG motor imagery decoding. Biomedical Signal Processing and
Control, 63:102144.
[Zhang et al., 2006] Zhang, Y., Ding, L., Drongelen, W. v., Hecox, K., Frim, D., and He, B. (2006). A Cortical
Potential Imaging Study from Simultaneous Extra- and Intra-cranial Electrical Recordings by Means of
the Finite Element Method. NeuroImage, 31(4):1513–1524.
[Zhang et al., 2019] Zhang, Z., Duan, F., Solé-Casals, J., Dinarès-Ferran, J., Cichocki, A., Yang, Z., and Sun,
Z. (2019). A Novel Deep Learning Approach With Data Augmentation to Classify Motor Imagery Signals.
IEEE Access, 7:15945–15954. Conference Name: IEEE Access.
[Zhu et al., 2019] Zhu, K., Wang, S., Zheng, D., and Dai, M. (2019). Study on the effect of different electrode
channel combinations of motor imagery EEG signals on classification accuracy. The Journal of Engineer-
ing, 2019(23):8641–8645. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1049/joe.2018.9073.
114
List of Figures
1 A typical BCI system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Schematic anatomy of a typical neuron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 An action potential . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
4 Synapse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5 Brain structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
6 Functional areas of the human brain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
7 Temporal and spatial resolution in neural recording . . . . . . . . . . . . . . . . . . . . . . . . . 7
8 Patch clamp recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
9 Intracellular and extracellular electrodes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
10 Microelectrode arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
11 Standard 10−20 system of positioning of EEG electrodes on the scalp. . . . . . . . . . . . . . . 10
12 Magnetoencephalography recording setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
13 Recording technologies based on the detection of hemodynamic response to neural activity . 11
14 Positron emission tomography scanner ("PET/CT scanner" by Frank Kehren, licensed under CC BY-
NC-ND 2.0.). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
15 Primary motor cortex area in the right hemisphere with approximate locations of the corre-
sponding EEG channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
16 Power spectral density (PSD) of EEG signals measured during motor imagery tasks . . . . . . . 14
17 Types of temporal filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
18 Spatial filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
19 SSD applied to real MI EEG data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
20 Forward and inverse EEG problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
21 The LDA hyperplane separates the data representing two different classes. . . . . . . . . . . . . 30
22 The optimal SVM hyperplane is found by maximizing the margin. . . . . . . . . . . . . . . . . . 31
23 Linear tangent space to the Riemannian manifold . . . . . . . . . . . . . . . . . . . . . . . . . . 32
24 Perceptron predicts the output ˆ
yby applying an activation function gto the weighted sum of
input features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
25 Examples of non-linear activation functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
26 Example of an MLP architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
27 A typical convolutional neural network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
28 Unfolded RNN loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
29 Autoencoder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
30 Restricted Boltzmann machine with four visible and three hidden units. . . . . . . . . . . . . . 38
31 VAE architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
32 Generative adversarial network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
33 Segmentation and recombination method for EEG data augmentation . . . . . . . . . . . . . . 44
34 Sliding windows approach allows extracting multiple segments from a single EEG trial. . . . . 45
35 The process of generating EEG spectral images from Gaussian noise with a diffusion model . . 48
36 Design of the trial for MI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
37 The process of dipole manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
38 Generating data with augmented patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
39 Generating data with recombination of motor- and noise-related components . . . . . . . . . 55
40 DeepNN architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
41 ShallowNN architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
42 Cross-subject baseline classification results on k. . . . . . . . . . . . . . . . . . . . . . . . . . . 60
43 Classification accuracy (percent correct) with different classification models when 20 original
trials per subject were available (k=20) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
44 Classification results with different models trained on the original and augmented datasets,
depending on k............................................... 62
45 Cross-subject classification results of the models trained on original data depending on the
number of available training subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
46 Classification accuracy (percent correct) with different classification models when all original
trials from 5 subjects were available (n=5) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
115
47 Classification results with different models trained on the original and augmented datasets,
depending on n............................................... 65
48 Baseline classification accuracy (percent correct) with different classification models in the
mixed scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
49 Accuracy gain after adding augmented data to the training dataset in the mixed scenario . . . 66
50 Within-subject classification results of the models trained on original trials depending on the
number of available training trials, denoted as k. . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
51 Classification accuracy (percent correct) with three different classification models for k=20,
k=60, and k=90 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
52 Classification results with different models trained on the original and augmented datasets,
depending on k............................................... 68
53 Within-subject classification results of the models trained on augmented trials depending on
the number of original training trials, denoted as k. . . . . . . . . . . . . . . . . . . . . . . . . . 69
54 Localized dipoles for the first five SSD components in subject 7 and subject 1 . . . . . . . . . . 70
55 Localized dipoles corresponding to the motor imagery task for all 18 subjects . . . . . . . . . . 71
56 Modified dipoles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
57 Original and augmented feature vectors projected onto the first two PCA components . . . . . 71
58 t-SNE embedding of original and augmented feature vectors . . . . . . . . . . . . . . . . . . . . 72
59 PCA and t-SNE applied to original and augmented data of all subjects . . . . . . . . . . . . . . . 72
60 Original and augmented feature vectors projected onto the first two PCA components . . . . . 72
61 t-SNE embedding of original and augmented feature vectors . . . . . . . . . . . . . . . . . . . . 73
62 PSD of original and augmented data in Laplacian channels C3 and C4 . . . . . . . . . . . . . . 73
63 PSD of original and augmented data in Laplacian channels C3 and C4 . . . . . . . . . . . . . . 73
64 Adjustments to the shifted pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
65 Effect of data augmentation on different classification models for k=5 . . . . . . . . . . . . . . 83
66 Effect of data augmentation on different classification models for k=20 . . . . . . . . . . . . . 83
67 Effect of data augmentation on different classification models for k=90 . . . . . . . . . . . . . 84
68 Effect of data augmentation on different classification models for k=120 . . . . . . . . . . . . 84
69 Effect of data augmentation on different classification models for n=3 . . . . . . . . . . . . . . 86
70 Effect of data augmentation on different classification models for n=5 . . . . . . . . . . . . . . 86
71 Effect of data augmentation on different classification models for n=10 . . . . . . . . . . . . . 87
72 Effect of data augmentation on different classification models for n=17 . . . . . . . . . . . . . 87
73 Accuracy loss after adding augmented data to the training dataset in the mixed scenario . . . . 88
74 Classification accuracy on the original (left) and augmented (middle) datasets, depending on
the total number of original training trials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
75 Classification accuracy (percent correct) with three different classification models for k=30
and k=120 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
76 Active brain areas during motor imagery, motor observation, and motor execution (from [Hard-
wick et al., 2017]). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
77 Localized dipoles for the first five SSD components for each subject . . . . . . . . . . . . . . . . 93
78 PSD of original and augmented data for all subjects . . . . . . . . . . . . . . . . . . . . . . . . . . 94
116
List of Tables
1 Frequency bands in EEG signals and corresponding mental states. . . . . . . . . . . . . . . . . . 9
2 Common feature extraction methods for MI EEG . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3 Common EEG data augmentation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4 Data generation for cross-subject classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5 Changes in classification accuracy after data augmentation in few-trial scenario . . . . . . . . 62
6 Changes in classification accuracy after data augmentation in few-subject scenario . . . . . . 65
7 Changes in within-classification accuracy after data augmentation . . . . . . . . . . . . . . . . 69
8 Effect of data augmentation on classification accuracy . . . . . . . . . . . . . . . . . . . . . . . . 85
9 Changes in within-classification accuracy after data augmentation (kfrom 20 to 70) . . . . . . 90
10 Changes in within-classification accuracy after data augmentation (kfrom 75 to 120) . . . . . 91
117
