FEEDBACK VISUALIZATION INFLUENCE ON A BRAIN-COMPUTER INTERFACE PERFORMANCE

Summary

This paper presents our progress in development of a brain computer interface. We used existing system which was extended to support synchronous experiments with feedback. We use the simplest possible methods as we focus on feedback influence. We examined influence of the way how the feedback is visualized. The abstract feedback, the feedback based on computer game, and the feedback showing real photographs were tested. The classification score 88.5% was achieved with one subject even with simple classification method.

Keywords:
BCI, Real-time processing, feedback

Introduction

Our group has been dealing with a research in the field of movement-related EEG recognition towards developing a Brain-Computer Interface (BCI). We have shown that off-line single trial classification of extension and flexion movements of right index finger is possible; classification score of up to 95 % was achieved [1]. The logical next step was to move towards a real-time processing; therefore we have been designing a modular real time processing system [2], [3] and [4].

In the paper [4] we examined how a BCI system is influenced by the presence of feedback. We decided to advance to the online classification of left and right hand movement imagery with visual feedback. This contribution examines the influence of the way how the feedback is presented to the user.

First movement related BCIs

We started with a research on methods used in the oldest movement related BCIs. The first movement related BCIs used synchronous experimental protocol and movements on the opposite side of the body. Synchronous setup means that the user is provided with instruction when and which mental activity he has to perform, in opposite to asynchronous when user performs activities in his own pace.

Works [5],[6],[7], and [8] use the following experimental protocol: at first, the fixation cross is presented to let the subject concentrate, then there is an acoustic beep prior to the instruction of movement to imagine (arrow). Then, the feedback in a form of extending bar is presented, and finally a randomized resting interval is used to avoid adaptation of the subject. Based on these findings we decided to adopt experimental protocol as shown in Figure 1.

Electrode placement was adopted from work [7]. We recorded 3 differential channels connected to the following electrodes (positive - negative): FC3 - CP3, FCZ - CPZ and FC4 - CP4. These channels are simply called C3, C4, and CZ in further text. Linked ears were used as reference.

EPP Architecture

Basic design of our distributed modular EEG Processing Pipeline (EPP) was presented in works [2] and [4]. The system is composed of independent standalone modules connected by a network interface, see Figure 2.

The modules are independent on used operation system and hardware as implemented in the Java language.

The communication protocol is based on Real-time Transport Protocol (RTP) to achieve real-time data transfer over the network. Intended purposes of RTP are real-time audio and video streaming. These have very similar requirements as EEG data transfer. And RTP packets are usually prioritized in network devices such as routers.

The latency is most critical parameter for real-time applications. The latency of the pipeline was tested in [2] and was found sufficient for the applications with feedback.

The communication protocol enables to transfer optional parameters defined by the common configuration system. Logging system can store all the communication including all states, messages and commands.

Following modules were implemented:

• BIOPAC Bridge module provides support for a EEG device manufactured by the BIOPAC Company which is available in our department. All the experiments presented in this paper were conducted using this device with a sampling rate of 200 Hz.
• Control module serves for real-time control of the running system, including switching states of the system and sending command to other modules.
• Visualization module serves for visual representation of classification to the experimental subject.
• Perceptron module implements a real-time classification algorithm – a simple one layer Perceptron.
• Feature Extraction module implements a general FIR filtration and a short time power estimation by a leaking integrator filter.
• Detection module computes the power in the same way as the Feature Extraction module and compares the value with a threshold.
• The Data Flow Monitoring module was extended to be capable of showing given experiment setting and all the inter-module communication.

Following modules were implemented in order to support synchronous experiments:

• The Trigger module serves for time synchronization. The module generates sequence of states from the predefined experimental protocol, see Figure 1.
• Feedback module provides instructions and feedback for the experimental subject. Current state of the feedback is picked up from the data flow. The feedback can be displayed as extending horizontal bar, moving image or animation.
• Arkanoid present feedback in the style based on the popular computer game Arkanoid.
• General purpose Generator module, which loads data from log file of any other module and simulates its behavior.
• Threshold module was implemented for the purposes of online classification experiments with asymmetric ratio [9]. The ratio A is defined as

where R is the feature extracted from a channel recorder over right hemisphere and L over left hemisphere.

We found out the ratio is usually biased. The bias is subject dependent and differs with each experiment even with same subject. It is caused by electrode placement, contact quality or by asymmetry of brain signal itself. We decided to make normalization of features before computing the ratio. We use balance constant b and we multiply one feature with 1 + b and the other with 1 − b

The constant can be set by the configuration file or it can be automatically computed. For automatic balancing we presume long term averages of the normalized features should be equal.

We found out that the modular architecture concept of the system as published in [2] was the right choice because we could easily extend it for new types of experiments.

Synchronous experiments

Synchronous experiments followed the experimental protocol as shown in Figure 1. Modules were connected as shown in Figure 3. The first synchronous experiments were with simulated feedback and the classification was performed offline. The purpose of these experiments was to find simple classification algorithm. Movement imagery of only right hand was used at first as we want to investigate the movement related activity. Stronger movement related activity compared to asynchronous experiment was detected which is in compliance with work [10]. Then two classification algorithms which used signal filtered by an 8-40 Hz band pass filter were tested. 

The first algorithm used the Trigger module to compare power from left and right hemisphere. The second algorithm used Support Vector Machine (SVM) and Hjorth features (Activity, Mobility and Complexity). The achieved scores (ratio of correctly classified trials and all trials) were 73 % for power comparison and 75 % for SVM. As the achieved score for SVM was not significantly better we decided to use the Trigger module for feedback comparison experiments [4].

The results with simulated feedback were promising therefore we have attempted to use real feedback based on an online classification. Threshold classification module was activated during the movement period thus the classification results were presented directly to the subject. Subjects reported that it was hard to concentrate on the movement imagery while watching the feedback as the classification results were quite frequently changing during the single movement imagery. The subjects felt more like they are responding to the presented feedback rather than that the system is responding to their imagery. This degraded the signal as offline classification scores have fallen to around 64 %.

We examined the influence of the feedback behavior [11]. The same setting as in previous experiments was used. The Threshold module was used for classification. The feedback was operating in two modes. One was realistic when the feedback bar was extending according to the classification result. The second one was so called optimistic it was expanding when the result of classification was the same as the command and it was still when the result was different. The optimistic feedback was found easier to control and achieved score was better [11].

The next experiment examined the feedback visualization itself. We were using three different feedback types (Fig. 4).

We were using optimistic feedback as it achieved best results in previous experiment. The experiment was performed with three subjects. These subjects were healthy and had no previous experience with the BCI. Each subject was measured in three sessions in different days. Session consists of 4 to 5 minutes long measurements with short breaks between them. One feedback type was tested in one measurement. There were nine measurements in each session (three of each feedback type). The videos from experiments can be found on research group website [12].

Each cycle command-feedback was considered right or wrong. Right cycle had more samples classified according to the command. The score was computed as ratio of right cycles and all cycles. The score for each feedback type can be found in Table 1.

The best results were with feedback based on Arkanoid probably because it is much more motivating than the others. The Arkanoid feedback shows whatever the user was successful and it even displays score. The subjective opinion about the feedback visualizations differs among subjects. The subject one claimed there is not big difference between feedback visualizations. The subject two founds the Bar feedback most easy to control but he has better results for the Arkanoid. He seemed very motivated when controlling the Arkanoid. The subject three claimed she does not like the Arkanoid feedback because it is much more stressful than the others. But the results for Arkanoid was also better than for other feedback types. We found out the score is very subject dependent. Our second subject have significantly better results than other two subjects (Table 2).

Conclusion and next steps

We found out that even the simple methods can be used for left and right hand imagined movement classification of EEG. We proven the BCI can be operated even with relatively simple equipment (only two channels EEG) and without need of very special environment such as shielded room. The novel method of asymmetric ratio automatic balancing was achieving high successful rates. This method was used to compare different feedback visualization types. The best results were with feedback which was based on computer game. The second was feedback based on real photographs and last was abstract visualization. The large differences between individual subjects was found. The average score of the best subject was 88.5% but the worst average score was only 56.3% (50% is a baseline).

The significant influence of momental psychical state of subject was also observed. For example stress or insufficient sleep can lower classification score. Subjects also easily lose focus or motivation. This can be partially avoided by right experiment setup and communication with subject.

The subjects were able to use the system almost without training. The results close to average were achieved after less than ten minutes from start of their first experience with BCI.

Because the system is relatively simple it provides large space for future improvements. More advanced classification methods can be used. The modular processing system enables easy adaptation of the new classification methods. The data from realized experiments can be used for design and testing of these methods. The first step in future improvements will probably be adding new class to classification. Thus the classification classes will be left, right, and neutral. This can be done with existing classification with asymmetric ratio even without changes in the source code.

Acknowledgement

Jakub Šťastný is supported by the Grant Agency of the Czech Republic through project P-102/11/1795: Novel selective transforms for non-stationary signal processing and along with the rest of the team by Grant Agency of the Czech Technical University in Prague, grant No. SGS12/143/OHK3/2T/13.

Vladimír Černý

Biological Signal Lab.,

Faculty of Electrical Engineering,

Czech Technical University in Prague,

Technická 2, Prague 6, 166 27 Czech Republic

E-mail: cernyvl3@fel.cvut.cz

Phone: +420 603 512 421