• TABLE OF CONTENTS
HIDE
 Title Page
 Copyright
 Dedication
 Acknowledgement
 Table of Contents
 Abstract
 Overview of CPVER
 Implementation of CPVER--the data...
 Implementation of CPVER data processing...
 Implementation of CPVER data collection/processing...
 Applications of CPVER--a study...
 Concluding remarks and future...
 References
 Appendices
 Equipment checklist procedure
 CPVER software user's guide
 CPVER module listings
 Biographical sketch
 Copyright






Title: Implementation, collection, and processing of visual evoked responses (CPVER) for cognitive studies
CITATION THUMBNAILS PAGE IMAGE ZOOMABLE
Full Citation
STANDARD VIEW MARC VIEW
Permanent Link: http://ufdc.ufl.edu/UF00089538/00001
 Material Information
Title: Implementation, collection, and processing of visual evoked responses (CPVER) for cognitive studies
Physical Description: viii, 281 leaves : ill. ; 28 cm.
Language: English
Creator: Arroyo, Amauri Antonio, 1949-
Publication Date: 1981
 Subjects
Subject: Evoked potentials (Electrophysiology) -- Experiments   ( lcsh )
Cognition -- Experiments   ( lcsh )
Electrical Engineering thesis Ph. D
Dissertations, Academic -- Electrical Engineering -- UF
Genre: bibliography   ( marcgt )
non-fiction   ( marcgt )
 Notes
Thesis: Thesis (Ph. D.)--University of Florida, 1981.
Bibliography: Bibliography: leaves 174-177.
Statement of Responsibility: by Amauri Antonio Arroyo.
General Note: Typescript.
General Note: Vita.
 Record Information
Bibliographic ID: UF00089538
Volume ID: VID00001
Source Institution: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
Resource Identifier: aleph - 000316118
oclc - 08557480
notis - ABU2909

Table of Contents
    Title Page
        Page i
    Copyright
        Page ii
    Dedication
        Page iii
    Acknowledgement
        Page iv
        Page v
    Table of Contents
        Page vi
        Page vii
    Abstract
        Page viii
    Overview of CPVER
        Page 1
        Page 2
        Page 3
        Page 4
        Page 5
        Page 6
        Page 7
        Page 8
        Page 9
        Page 10
        Page 11
        Page 12
        Page 13
        Page 14
        Page 15
        Page 16
        Page 17
        Page 18
        Page 19
        Page 20
        Page 21
    Implementation of CPVER--the data collection subsystem
        Page 22
        Page 23
        Page 24
        Page 25
        Page 26
        Page 27
        Page 28
        Page 29
        Page 30
        Page 31
        Page 32
        Page 33
        Page 34
        Page 35
        Page 36
        Page 37
        Page 38
        Page 39
        Page 40
        Page 41
        Page 42
        Page 43
    Implementation of CPVER data processing subsystem
        Page 44
        Page 45
        Page 46
        Page 47
        Page 48
        Page 49
        Page 50
        Page 51
        Page 52
        Page 53
        Page 54
        Page 55
        Page 56
        Page 57
        Page 58
        Page 59
        Page 60
        Page 61
        Page 62
        Page 63
        Page 64
        Page 65
        Page 66
        Page 67
        Page 68
        Page 69
        Page 70
        Page 71
    Implementation of CPVER data collection/processing interface
        Page 72
        Page 73
        Page 74
        Page 75
        Page 76
        Page 77
    Applications of CPVER--a study of cognitive processes
        Page 78
        Page 79
        Page 80
        Page 81
        Page 82
        Page 83
        Page 84
        Page 85
        Page 86
        Page 87
        Page 88
        Page 89
        Page 90
        Page 91
        Page 92
        Page 93
        Page 94
        Page 95
        Page 96
        Page 97
        Page 98
        Page 99
        Page 100
        Page 101
        Page 102
        Page 103
        Page 104
        Page 105
        Page 106
        Page 107
        Page 108
        Page 109
        Page 110
        Page 111
        Page 112
        Page 113
        Page 114
        Page 115
        Page 116
        Page 117
        Page 118
        Page 119
        Page 120
        Page 121
        Page 122
        Page 123
        Page 124
        Page 125
        Page 126
        Page 127
        Page 128
        Page 129
        Page 130
        Page 131
        Page 132
        Page 133
        Page 134
        Page 135
        Page 136
        Page 137
        Page 138
        Page 139
        Page 140
        Page 141
        Page 142
        Page 143
        Page 144
        Page 145
        Page 146
        Page 147
        Page 148
        Page 149
        Page 150
        Page 151
        Page 152
        Page 153
        Page 154
        Page 155
        Page 156
        Page 157
        Page 158
        Page 159
        Page 160
        Page 161
        Page 162
        Page 163
        Page 164
        Page 165
        Page 166
        Page 167
        Page 168
        Page 169
        Page 170
    Concluding remarks and future work
        Page 171
        Page 172
        Page 173
    References
        Page 174
        Page 175
        Page 176
        Page 177
    Appendices
        Page 178
    Equipment checklist procedure
        Page 179
        Page 180
        Page 181
        Page 182
        Page 183
    CPVER software user's guide
        Page 184
        Page 185
        Page 186
        Page 187
        Page 188
        Page 189
    CPVER module listings
        Page 190
        Page 191
        Page 192
        Page 193
        Page 194
        Page 195
        Page 196
        Page 197
        Page 198
        Page 199
        Page 200
        Page 201
        Page 202
        Page 203
        Page 204
        Page 205
        Page 206
        Page 207
        Page 208
        Page 209
        Page 210
        Page 211
        Page 212
        Page 213
        Page 214
        Page 215
        Page 216
        Page 217
        Page 218
        Page 219
        Page 220
        Page 221
        Page 222
        Page 223
        Page 224
        Page 225
        Page 226
        Page 227
        Page 228
        Page 229
        Page 230
        Page 231
        Page 232
        Page 233
        Page 234
        Page 235
        Page 236
        Page 237
        Page 238
        Page 239
        Page 240
        Page 241
        Page 242
        Page 243
        Page 244
        Page 245
        Page 246
        Page 247
        Page 248
        Page 249
        Page 250
        Page 251
        Page 252
        Page 253
        Page 254
        Page 255
        Page 256
        Page 257
        Page 258
        Page 259
        Page 260
        Page 261
        Page 262
        Page 263
        Page 264
        Page 265
        Page 266
        Page 267
        Page 268
        Page 269
        Page 270
        Page 271
        Page 272
        Page 273
        Page 274
        Page 275
        Page 276
        Page 277
        Page 278
        Page 279
        Page 280
    Biographical sketch
        Page 281
        Page 282
        Page 283
    Copyright
        Copyright
Full Text













IMPLEMENTATION, COLLECTION, AND PROCESSING OF
VISUAL EVOKED RESPONSES (CPVER) FOR COGNITIVE STUDIES
















BY


AMAURI ANTONIO ARROYO


A DISSERTATION PRESENTED TO THE GRADUATE COUNCIL
OF THE UNIVERSITY OF FLORIDA IN
PARTIAL FULFILLMENT OF THE REQUIREMENTS
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY


UNIVERSITY OF FLORIDA


1981



































Copyright 1981

by

Amauri Antonio Arroyo





































TO LYDIA















ACKNOWLEDGEMENTS


The author wishes to express his gratitude, appreciation and high

esteem to his advisor, Dr. D. G. Childers, for his direction, support,

guidance, and determination throughout the course of this research,

and throughout all of the author's graduate studies.

The author is particularly grateful to Dr. R. G. Selfridge, whose

moral support and encouragement have been a source of strength during

the last decade.

He is also thankful to Dr. N. W. Perry, Jr., and Dr. I. S. Fischler

for many stimulating discussions. The author is grateful to

Dr. J. R. Smith and Dr. L. W. Couch for serving on the committee and

for their guidance. He also wishes to express his appreciation to

fellow graduate students P. A. Bloom and T. Achariyapaopan for their

many stimulating discussions, encouragement, and moral support.

The author is thankful to Mrs. Edna Aquino, Mrs. Gloria Luiggi,

Mrs. Amarilys Quevedo, Miss Arlene Arroyo and Mr. Armando Quevedo for

their encouragement, prayers, and their help in many ways.

The author wishes to thank his wife Marilyn, who not only edited

and typed this dissertation, but whose love, patience, understanding

and support have made this work possible.

The author is MOST grateful and eternally indebted to his mother,

Lydia. Her hard work, sacrifice, love, patience, support, and encour-

agement have been the motivating force behind the author's accomplish-

ments, life, and work.











The author acknowledges, that above and beyond all, the acceptance

of the Lord Jesus Christ as his personal Saviour has been the key to

any and all fruit of his endeavor and reserves any and all glory and

praise to the Almighty, Yahweh God.













TABLE OF CONTENTS



Page

ACKNOWLEDGEMENTS . . . . . .. iv

ABSTRACT . . . .. . . viii

CHAPTER

I OVERVIEW OF CPVER . . . . .. 1

1.1 Introduction and General Objectives . .
1.2 Single Visual Evoked Responses!
A Signal Processing Perspective . . 5

1.2.1 Signal Extraction Methodology . 7
1.2.2 Signal Classification Methods . 9
1.2.3 A Framework for Real-Time Classification 10

1.3 Technical Approach . . . ... 14

1.3.1 Functional Specification of CPVER .. 14
1.3.2 Software Overview . . ... 17
1.3.3 Equipment Overview . . ... 19

1.4 Preview of Remaining Chapters . . .. 21

II IMPLEMENTATION OF CPVER--THE DATA COLLECTION
SUBSYSTEM . . . . ... .... .22

2.1 Introduction . . . ... .22
2.2 Data Collection Subsystem Overview . .. 22
2.3 Experimental Application Overview . .. 23
2.4 Real-Time Stimulus Presentation and EEG
Data Collection . . . ... 25
2.5 Data Averaging and Display . . ... 37

2.5.1 The EEG Data Averaging Module . .. 37
2.5.2 EEG Data Display Module . . .. 39
2.5.3 Single Epoch Display . . .. 41

2.6 Data Storage Conventions . . ... 41

III IMPLEMENTATION OF CPVER DATA PROCESSING SUBSYSTEM 44

3.1 Introduction . . . .... .44









3.2 Data Processing Subsystem Overview . ... 45
3.3 Experimental Overview. . . . ... 46
3.4 Feature Extraction Methodology . ... 49
3.5 Feature Extraction Software . . ... 53

3.5.1 Covariance Estimator Module--COVEST .. 57
3.5.2 Discriminant Vector Estimator Module--
DVEST . . . ... .62

3.6 ERP Classification Software--
The SCATTER Module . . . ... 64
3.7 Data Processing/Data Storage Conventions ... 69

IV IMPLEMENTATION OF CPVER DATA COLLECTION/PROCESSING
INTERFACE . . . . ... ... 72

4.1 "Off-Line" Implementation of CPVER . ... 72
4.2 "Real-Time" Implementation of CPVER. . .. 74

V APPLICATIONS OF CPVER--A STUDY OF COGNITIVE PROCESSES 78

5.1 Experimental Design . . . .. 78
5.2 Summary of CPVER Data Processing Results ... 85

VI CONCLUDING REMARKS AND FUTURE WORK . ... 171

REFERENCES . . . . ... . .... 174

APPENDICES . . . ... . ... 178

1 Equipment Checklist Procedure . . ... 179
2 CPVER Software User's Guide . . ... 184
3 CPVER Module Listings . . . ... 190

3A CPVER Data Collection Software . ... 191
3B CPVER Data Processing Software . ... 206
3C CPVER Support Modules Software . ... 227
3D CPVER File Details Software . . ... 259
3E CPVER Command Modules Software . ... 266

BIOGRAPHICAL SKETCH . . . . ... . 282


vii
















Abstract of Dissertation Presented to the Graduate Council
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy








IMPLEMENTATION, COLLECTION, AND PROCESSING OF
VISUAL EVOKED RESPONSES (CPVER) FOR COGNITIVE STUDIES

By

Amauri Antonio Arroyo

December 1981


Chairman: D. G. Childers
Major Department: Electrical Engineering

Single Visual Evoked Responses (SVERs) elicited from human

subjects can be used to study sensory and perceptual processes.

The philosophy behind the design and implementation of a computer

laboratory facility to collect and process visual evoked responses

(unaveraged or single EEG records) is described and expounded.

The resulting system has been used to collect and process Evoked

Potentials (EPs) for a series of cognitive experiments. The "off-

line" data processing subsystem has been used to process the SVERs

with classification error rates as low as 0% (perfect) for both

design and test sets. Operator guides for the software and hardware

procedures are described in the appendices.


viii















CHAPTER I
OVERVIEW OF CPVER



1.1 Introduction and General Objectives


The need for understanding the neural processes underlying human

perception and behavior has motivated the recent unification of

psychology, neurology, engineering and Computer Science into inter-

disciplinary teams studying.electrical responses evoked from the human

brain. The experimental approach has been to measure the electrical

activity of localized populations of neurons in the brain during the

performance of specific perceptual or behavioral tasks. The activity

of neurons generates electric currents that are transmitted through

the skull, thereby causing minute voltage changes known as the electro-

encephalogram (EEG). These neuronal responses can be recorded with

non-invasive disk electrodes taped to the scalp. The EEG reveals

little about sensory perception because of the absence of specific

correlation to sensory stimuli. The evoked potential technique attempts

to focus on the neuronal potential that is directly evoked to specific

sensory stimuli. This process supposedly recruits and synchronizes

numerous neurons so that the evoked response to a particular stimulus

exceeds the on-going background activity.

The availability of the evoked potential has helped to reveal

specific brain activity where information is handled simultaneously

in multichannel fashion; it can be used to distinguish organic











disorders from psychogenic ones; and it can provide an objective indica-

tion of sensory function where conventional perceptual tests are un-

reliable or impractical (patients, infants, the severely handicapped,

etc.).

The main problem in recording evoked potentials (EP) is detecting

them at all, since they are imbedded in the composite electrical

activity of the brain as a whole (EEG). Whereas the EEG signal

reaches amplitudes of 50 to 100 microvolts, single evoked potentials

(EPs) are often in the range of 1 to 5 microvolts and may be as small

as 0.5 microvolts. The signal energy appears confined mostly to low

frequencies, e.g., around the 10 Hz alpha rythm. This continuous

activity is spatially distributed over the scalp and can be, at best,

described as a piecewise stationary process. The evoked potential

(EP) (evoked response (ER) or event related potential (ERP)) is the

desired signal and to the ER investigator the EEG signal represents

unwanted and overwhelming background noise.

Event related potentials (ERPs) may be induced by other than

sensory events. Short latency ERPs are those whose presence is most

prominent in the 50 ms to 150 ms range following the stimuli and

include sensory ERPs elicited by visual (VER), auditory (AER), soma-

sensory (SER) and olphatory (OER) as well as by direct electrical

stimulation (EER). Motor ERPs are responses found accompanying

voluntary movement that may in fact precede the actual behavioral

event. Examples of motor ERPs include phonation, eye movement and

limb movement. Long latency ERPs refer to potentials whose presence

is detected in the 250 to 450 ms range following the stimuli. Most










prominent in the literature is the positive deflection occurring

around 300 ms (P300) and more recently, the negative component

around 400 ms (N400) [1] has been discussed. These components

are enhanced by subject expectation (novelty or familiarity) and

task relevance to stimuli.

Artifacts are potential fluctuations of non-neuronal origin such

as electroocular potentials (EOG), muscle potentials from neck, scalp

or face (such as eyeblinks), electrocardiographic signals (ECG), and

external electric or magnetic field interference.

Because of the large differences in magnitude between the ERP and

the on-going EEG signal, unrelated to the event of interest, the ERP

competes with the EEG for "signal space" and as such the EEG constitutes

or plays the role of noise. If a deterministic, repetitive signal

is added to random noise and the time occurrence of the signal is

accurately known, averaging becomes an effective tool for enhancement

of the signal. Conventional averaging, however, is subject to the

following assumptions about the entire EEG process. First, the moni-

tored composite scalp brain wave f(t) is composed of an ERP signal

component s(t) and an additive noise component n(t) represented by

the on-going EEG waveform; e.g., the noise is purely additive. Second,

the ERP signal s(t) is usually assumed to be deterministic; e.g., that

for the same stimulus and subject, the ERP signal will faithfully

repeat itself given the same stimuli. Third, the noise component is

not time-locked to the presentation of the stimulus. Aunon et al.[2]

give a detailed discussion of conventional averaging as applied to

evoked potential research.











Evidence exists, however, that individual evoked responses s(t)

do indeed change from stimulus to stimulus, thereby rendering conven-

tional averaging an ineffective tool for signal enhancement. Ideally,

one would like to recover the event related information from the elec-

trical potential on the basis of single events (individual responses

or "single epochs" in real time) as they occur during an experiment.

This represents a major challenge when considering that effects of

instrumentation noise and artifacts are more critical in the absence

of averaging.

The study of cognitive brain function is currently a major area

of interest in experimental psychophysiology and behavioral psychology.

The possibility exists that single ERPs can provide an objective

measure of brain processes involved in learning, memory functioning

and problem solving. The ability to discriminate or classify single

ERPs, on-line, and provide feedback to the subject for conscious

validation could bring to this field the same advantages enjoyed by

other classical behavioral experiments, mainly that of subject parti-

cipation. These biofeedback-type experiments could give much insight

into psychological behavior and could have applications to such areas

as communication with the severely handicapped and infants or improved

automatic response aids for airline pilots and astronauts, to name a

few.

The major objective of this research is the development and

implementation of an integrated ERP collection and processing labora-

tory facility to be used in conjunction by psychologists, scientists

and biomedical engineers to study the cognitive functions of the human











brain. Keeping in mind the desire for "on-line" classification and

biofeedback, the goal was to develop and standardize procedures, soft-

ware, optimize hardware participation, minimize operator intervention,

reduce total subject participation time, and design enough flexibility

and generality into the system to accommodate new and novel experimental

paradigms [3]. In addition, the need for making it simple to learn and

operate the software has become a significant area of interest, and as

such a secondary goal for the University of Florida (UF) collection

and processing of visual evoked response (UF-CPVER) system.



1.2 Single Visual Evoked Responses!
A Signal Processing Perspective


Evoked responses have been used to find clues to the functioning

of the human brain [4, 5, 6]. However, the use of evoked potentials

has not become wide-spread due to major problems such as large inter-

subject variability [7, 8], within subject variability [9], artifact

contamination etc.

If individual evoked responses change from stimulus to stimulus,

then the time average may contribute very little to the characteriza-

tion of the response. The single evoked potential may then be a viable

alternative. This is particularly true in certain kinds of cognitive

studies. One such study requires the subjects to perform a "sentence

verification" task that has been used extensively in behavioral re-

search of human semantic memory [1, 10, 11, 12]. The stimuli in these

experiments cannot be repeated numerous times because, presumably,











the response to previous sentences could become overlearned. The

response could then become one of simple association rather than a

decision process employing long-term memory, incorporation of overall

meaning and evaluation with respect to the correctness of a proposition.

The detection of the single evoked potential represents a formidable

challenge to the signal processor especially in the presence of such

experimental constraints.

Recent articles by John, Ruchkin, and Vidal [13], Freeman [14],

and Brillinger [15] concisely document methods of ERP analysis. Five

major types of measurement problems have been identified by John et al.

[13] as follows:

1) The extraction of ERP "signals" from EEG "noise." Signal

classification methods are better suited for handling non-

homogeneous signals, such as the SVER, than averaging, but

averages can be used for artifact identification and for

short latency ERP contributions.

2) Extraction of quantified features from, or a quantitative

description of, signals after separation from noise. Special

feature extraction algorithms can be used to describe specific

aspects of the ERP. Examples include principal components,

Fourier Analysis, the eigenvalue/eigenvector methods of

Roucos and Childers [12], etc.

3) Analysis of relationships between signals as dependent

variables and experimental manipulations of stimuli, environ-

mental conditions or state of the system as independent

variables. ANOVAS, t-tests and stepwise discriminant analysis

are examples of techniques used in this area.











4) Analysis of relati

regions caused by

signals in the sam

stimuli. Factor

discriminant analy

successive or diff

5) Analysis of relati

caused by the same

Cluster Analysis,

inant functions ar

tween groups of su

"a-priori" classes

1.2.1 Signal Extraction Me

The simplest model of

an (assumed) invariant sign

generous data the method of

for extracting the signal f

model is quite valid for an

latency contributions to ER

least valid for the longer

activity. The average ERP,

ification of certain types


nships between signals in different

he same stimulus or between successive

region caused by successive or different

alysis, sorting methods and stepwise

is can be used to classify ERPs to

rent stimuli.

nships between signals in the same regions

stimuli delivered to different individuals.

actor Analysis, Multiple ANOVA or discrim-

methods used to describe differences be-

jects identified as members of different



hodology

n evoked response consists of the sum of

1 and random background noise. For homo-

esponse averaging may be a valid procedure

om the noise in which it is imbedded. This

sthetized subjects, and for the short

s which reflect sensory processes. It is

latency components associated with cognitive

however, may be quite useful in the ident-

of artifacts.


1) The following methods are recommended as precautionary

measures for the detection of artifacts by John et al. [13].

First, it is highly advisable to examine visually at least

a representative sample of single ERPs. The vertical array











display of stimulus-aligned (stimulus-lock) data described by

John et al. [16] is easily implementable. An alternative

implementation is obtained by displaying the ERP in a three-

dimensional coordinate system, where "surface" effects are

more readily identified [11, 17, 18]. Second, it is extremely

useful to compute the standard deviation or the variance at

each latency point in the epoch.

2) Prefiltering of evoked responses prior to visual inspection

of single trial records may be profitable if the filter

attenuates activity in the following range(s) where the noise

is dominant. Digital filtering techniques are a convenient

way of implementing such filters. Other filtering techniques

of interest includes the Linear Prediction, Autoregression,

and Maximum Entropy Analysis [19, 20], the MMSE and MSNR method

of Aunon and McGillem [21], Matched Filtering [22], Weiner

Filtering [23, 24, 25], Cross and Autocorrelation [26], and

Adaptive Filtering [27, 28, 29].

3) Signal averaging across response-aligned (response-locked)

ERPs can potentially enhance the evoked response and/or the

artifact. Problems will arise if ERPs persist beyond the

time of occurrence of subsequent stimuli, therefore as a rule

of thumb, the effective time span of the range of variation

of the interstimulus intervals should be greater than the

period of the lowest frequency component of the overlapping

activity to be attenuated by averaging.










4) Adaptive Correlation Detection [28,29] is a method that can

be used for averaging variable latency (non-homogeneous)

signals. McGillem and Aunon [25] have devised a near optimal

algorithm that uses pre-filtering techniques to attenuate the

signal in frequency ranges where the S/N ratio is relatively

low. However, these strategies cannot deal accurately with

cases in which there exist significant signal components

within the analysis epoch which are not phase-locked to the

major signal peak. Classification procedures may be the only

way to analyze such data.



1.2.2 Signal Classification Methods


Although signal classification methods are more difficult, and

feature extraction and selection is not straightforward, they are often

capable of providing accurate results in cases where it would be hope-

less to use signal-extraction methods. For single ERPs, these methods

may be used for: 1) Identification of "protypic" responses and 2)

identification of sequences-of-states in the process. Sorting methods

and stepwise discriminant analysis are procedures suitable for the

identification of prototypic responses corresponding to each state

(of the subject-environment system) of interest. In the latter case,

there may exist non-random variations in the nature of the signal which

reflect different stages in some process, such as decision-making. The

features of the signal are not only themselves of interest, but each

different form of the signal may reflect some characteristic response

mode of the system.










Single epoch classification with no training data (sorting methods)

require two general steps: 1) Determination of the characteristics

of the non-homogeneities and 2) classification of the individual ERPs

into homogeneous subgroups. Quantitative assessment of non-homogeneity

can be made by computing for each time point a chi-squared measure

for the fit of the amplitude histogram to a normal distribution, and

the number of modes of the profile of the histogram. Such methods

must be used with care for if misused, the method can say "yes" to

almost any proposition [13].

Single epoch classification with training data discriminantt

analysis) was pioneered by Donchin et al. in the late 1960's [30].

Studies by Vidal [9], Childers et al. [11,12,17,19] and others [20]

have shown unequivocably that under suitable conditions, extremely

reliable classification of single ERPs can be performed. These

methods can open the door to a new realm of experiments in which

cognitive variables that vary from trial to trial may be manipulated

and sorted out. The ultimate goal of the signal classification

methodology is to recognize single visual evoked responses to meaning-

ful stimuli in real-time.



1.2.3 A Framework for Real-Time Classification

The following constitutes a general framework of procedures to be

considered in the generation of a system to do "on-line" real-time

classification. The summary in John et al. [13] provided a starting

basis for this framework.










Artifact rejection. This procedure detects electroocular and

movement artifacts (EOG artifacts, eyeblinks, head movement, etc.)

before and during the ERP epoch. On-line artifact rejection is

fundamental to the real-time approach and is essential to the overall

strategy. Peak detection algorithms with the width of the window and

threshold value as adjustable parameters are used for this purpose.

Artifact monitoring must continue throughout the data acquisition

period. The detection of an artifact then aborts the trial and re-

schedules the stimulus event.

Preprocessing--Wiener filtering. Real-time filtering may be

performed to improve the signal-to-noise ratio based on covariance

information [21,23,24,25].

Selection of a feature vector. Data reduction techniques are

applied to the digitized EEG data in order to reduce the dimensionality

of the measurement vector. In cases where the number of samples/class

is small compared to the dimensionality of the observation vector, the

data classification method may be unreliable [11,12,31]. This ratio

must be increased in order to improve the reliability of classification.

As features are selected, the probability of error will decrease, up

to the point where additional unreliable features are added. Beyond

this point the population PEwill increase [11,12,32-38]. Care must be

taken when using eigenvector expansions, because the largest principal

component may yield a very poor feature for some data sets, particularly

those where two classes have different means but equal covariances.

The mean difference along this principal component may be zero [11].

Cognitive studies demand special consideration because the number of











samples/class cannot be increased given a certain experimental

paradigm. For these small sample cases few feature extraction algo-

rithms are available.

Stepwise selection of best features. As mentioned earlier, there

may exist an optional set of features that achieves the lowest PE.

Selection of these features is based upon Fisher's ratio [39], a type

of class separability [40]. Algorithms developed by Childers et al.

[11,12] solve the stepwise selection problem.

Design or training set. Discriminant analysis-type methods

require a "training set," e.g., a set of labelled epochs of known

classes to guide the data reduction (feature extraction) procedure in

the selection of an optimal set of features.

Epoch classification and outlier rejection. A linear Bayesian

decision rule is calculated for each pair of hypotheses (classes).

The performance of this rule (PE) is estimated using the elements of

the design set. The PE thus obtained gives an optimistic value for

the true PE of the system. The classifier design algorithm attempts

to obtain the lowest PE for the design set and establishes a rejection

region for outliers. Interactive graphic packages, clustering tech-

niques, nearest neighbor rules etc., are useful procedures to use for

the automatic design of an "optimum" classifier [41-43].

Test set or real-time defaulting. Once the decision rule is

established on the design set, real-time classification ("testing")

can proceed with a minimum of computing time between epoch acquisition.

The only calculation involved is the evaluation of the linear decision

rule. This task can, of course, be performed on an independent set











of trials commonly labelled the "test set." The probability of error

(PE) thus obtained gives a pessimistic value for the true PE of the

system. Therefore the bounds (PEd, PEt) establish the performance

of the combination feature extractor/classifier algorithm.

Decision rule updating. By making the procedure recursive,

blocks of epochs can sequentially serve as training sets for the

next string. This capability allows for decision rule updating

where the "behavior" of a subject may be monitored and investigated

and an objective measure obtained of such brain processes as learning

and problem solving. Experimental paradigms involving biofeedback

may then become plausible alternatives in cognitive research.













1.3 Technical Approach


Attention is now focused on the major components that comprise

an ERP collection and processing laboratory facility. For design

purposes, the application is partitioned into three levels: 1) The

development of a complete set of functional requirements (specifica-

tions) that describes, in precise terms, the overall performance of the

integrated system; 2) the development of a set of software modules to

supervise and control the ERP collection phase, and signal processing

modules to recognize the single visual evoked response to meaningful

stimuli; 3) identification of the hardware (equipment) used to imple-

ment the system and the development of special-purpose software (for

example a display routine to drive an oscilloscope) to use this equip-

ment in EEG applications.

1.3.1 Functional Specification of CPVER


The major functional blocks of the CPVER system are depicted in

Figure 1-1. The figure shows general system flow, hardware components,

software components, and external or environmental interaction (the

operator, for instance). The overall (technical, objective and func-

tional) specification is made up of varying amounts of (emphasis on)

efficiency, maintainability, modifiability, generality, flexibility and

utility. These are components of objective system quality, e.g., the

best implementation in established terms, within recognized limitations,

while making compromises required by working in the real world.































A
r(t)


Figure 1-1 CPVER Functional Diagram










Broadly speaking, the following criteria were incorporated into

the overall performance requirements of the system:

1) To reduce total subject participation time in order to mini-

mize fatigue and to meet the experimental paradigm requirements.

2) To reduce the number of personnel required to supervise and

control the data collection effort to a minimum of one or

two persons.

3) To increase the participation of the computer in all aspects

of data collection and processing, e.g., to make the system

more automatic and to optimize hardware participation.

4) To design enough generality (the system's ability to be

widely used and adaptable to many variations in application

and/or environment) into the system for its use in a broad

class of cognitive experiments.

5) To build enough flexibility into the system by increasing

the extent to which more control over the behavior of the

system resides in the input data or the resident data

(specifications tables) than in the instructions themselves.

Also by increasing the number and variety of internal controls

(flags) that are made externally available.

6) To modularize the implementation in order to increase the

reliability of the components. System reliability is also

increased by reducing the number of decisions and inputs

required of the operator.

7) To use structured design concepts and structured programming

in order to influence the cost of keeping the system viable










in the face of changing requirements modifiabilityy), while

keeping the productivity of the system in a relatively

constant environment maintainabilityy).

8) To make the system simple to use, e.g., to make it "menu"

or "a la carte" driven.

9) To increase the utility of the system by standardizing

procedures and software modules, especially those which

control peripheral devices such as the A/D converter, oscillo-

scope, three-dimensional display, etc. This is a very impor-

tant consideration in a research laboratory facility.

10) To perform single VER extraction and classification in real-

time or in near real-time.

All of these factors have been given consideration in the design

of the CPVER system. However, in the event that a compromise between

criteria was required, items 10, 1, 2, 3, 8 and 9 were given precedence.



1.3.2 Software Overview


The software modules of CPVER consist of: 1) A "real-time" stim-

ulus presentation and EEG data collection module, 2) a set of data

averaging and data display modules, 3) feature extraction modules,

4) an ERP classifier module, and 5) a set of special-purpose modules

to control non-standard peripherals or to enhance the participation of

standard peripherals (such as the use of the "off-line" capability of

the Hewlett-Packard (HP) 2648A terminal).










The "real-time" stimulus presentation and EEG data collection

module schedules the statement propositions, collects the digitized

EEG, monitors the EOG signal for eyeblink artifacts, reschedules

stimulus presentation, schedules and collects a calibration signal

from each EEG channel, and schedules and presents photic stimuli in

order to obtain an "averaged" VER which is used for a baseline deter-

mination.

The data averaging and display routines display the collected EEG

signal from a given channel for operator inspection and signal valida-

tion. Evoked response averages are used to detect certain types of

artifacts not readily observable from single epoch inspection, and to

establish the adherence of the collected EEG to "expected" behavior

(there should be a signal peak on the stimulus-aligned averaged VER,

shortly after the onset of each stimulus segment presentation).

The feature extraction software consists of two major routines,

a discriminant vector estimator module and a covariance matrix esti-

mator module. The purpose of feature extraction is to reduce the

dimensionality of the measurement vector in order to improve the

probability of correct classification [11,12].

The ERP classification module derives a linear Bayesian rule from

the projection of a subset of the ERP trials (design set) onto the

selected feature space. The module produces a design scatter plot

from the points obtained by forming the dot product between the fea-

ture vectors and the data [11,12]. This is followed by a search algo-

rithm which shifts and/or rotates the perpendicular bisector of the










centroids of the two data classes (the decision boundary line) until

the probability of error for the design set is minimized [11].

Special-purpose software modules were developed for the following

applications: 1) Analog-to-digital conversion for a set of signal

channels on programmed input-output mode, synchronized to either an

internal clock, an external clock, the CPU basic instruction cycle or

no synchronization, 2) digital-to-analog conversion using either pro-

grammed I/O mode or the data channel (DMA) facility for one or two

channels with or without blanking pulses, 3) oscilloscope driver,

4) DEVELCO 3-D display driver, 5) keyboard interrupt facility, 6) inter-

rupt disabled FORTRAN "TYPE" statement, 7) graphics and device control

drivers for the HP 2648A terminal, and 8) a pulse generator for trigger-

ing the Grass Instruments xenon photo stimulator.



1.3.3 Equipment Overview-- Hardware


The various functions performed by the hardware are also depicted

in Figure 1-1. CPVER uses both instruments that are available "off-

the-shelf" in most electrical engineering laboratories and custom-built

equipment. The implementation of the CPVER system is greatly influenced

by the type of computer that is available, its operating system, and its

attached peripherals. In many instances, a desired capability was

obtained by interconnecting a module from a given piece of equipment

to another.

The following equipment was used to implement CPVER:










1) DGC NOVA 4 minicomputer with 256 KB semiconductor memory and

the floating-point hardware option running the MAPPED RDOS,

Revision 6.62, operating system and an Inter-Processor Buffer.

2) DGC NOVA 2 minicomputer with 64 KB core memory and a hardware

multiply/divide option running the RDOS, Revision 6.52, oper-

ating system and an Inter-Processor Buffer.

3) A DGC 6070 Moving Head Disk storage facility (2-10 MB disk

cartridges, one removable, one non-removable).

4) A DGC 4047B Moving Head Disk storage facility (2-2.4 MB

removable disk cartridges).

5) Hewlett-Packard 2648A Graphics Terminal with printer/plotter

attachment.

6) B-W TV monitor with variable horizontal high-frequency

adjustment.

7) EEG amplifiers with 1 to 50 Hz bandwith and 50% attenuation

at 1 and 50 Hz.

8) DEVELCO 3-D Axis Display consisting of a control and A/D

unit, a core memory unit and a high-resolution video monitor.

9) Grass Instruments polygraph recorder.

10) Grass Instruments xenon photo stimulator.

11) DGC Model 4330-A Digital Analog Subsystem.

12) Analog Digital Subsystem for the NOVA 2 computer [44].

13) Two Televideo, Model TVI-912B CRT terminals.

14) Oscilloscopes, low-pass filters, high-pass filters, ampli-

fiers, power supplies, pulse generator, EEG signal calibrator,

subject response pulse generator, scalp electrodes and associ-

ated supplies, audio "white" noise generator, a ventilated











Faraday shielded room, and various types and lengths of

cabling.

Prior to any data collection effort, the equipment must be in-

spected, prepared and tested for proper operation in order to minimize

instrumentation interference and prevent equipment failures during the

experimental sessions. A standardized procedure and equipment check-

list (see Appendix 1) has been developed for this purpose. This pro-

cedure has been found effective in minimizing unexpected equipment

failures, and in the "debugging" of the hardware in the event a mal-

function is detected and the faulty component is not obvious.



1.4 Preview of Remaining Chapters


The remaining chapters of this work present a survey of the CPVER

implementation, results, and future research.

Chapter II gives the details of the implementation of the data

collection subsystem. Chapter III discusses the data processing

subsystem. Chapter IV discusses the interfacing of data collection

and processing into an integrated "menu" driven system. Chapter V

provides details on the experiments conducted using the CPVER system.

Chapter VI presents some concluding remarks.

The appendices includes the hardware and equipment checklist, the

software user's guide and program listings.
















CHAPTER II
IMPLEMENTATION OF CPVER
THE DATA COLLECTION SUBSYSTEM



2.1 Introduction


The implementation details of the EEG data collection subsystem

are described in this chapter. Software philosophies and experimental

procedures are explained from the system designer's point of view. The

chapter is divided into the following subsections: 1) Data Collection

Overview, 2) Experimental Application Overview, 3) Real-Time Stimulus

Presentation and EEG Data Collection, 4) Data Averaging and Display,

5) Data Storage Conventions.



2.2 Data Collection Subsystem Overview


The goal of the CPVER data collection subsystem is to monitor

and digitize the EEG signal, from a number of scalp electrodes, that is

the response to a given stimulus presentation. The system must be

capable of eliminating trials that are found to have been contaminated

by artifacts, and reschedule the stimulus presentation. The data

collection software must be designed as an "on-line," "real-time,"

memory-resident package that has the capability to respond to various

kinds of external interrupts without abnormally terminating.










The data collection software is divided into two classes of

computer programs: One type is the "real-time" stimulus presentation

and EEG data collection and storage routines; the other is a set of

data display and averaging modules. The latter is used to validate

the collected data base by establishing general adherence to "expected

behavior." An example of this type behavior is the signal peaks that

are observable in averaged stimulus-aligned VERs shortly after the on-

set of the visual stimulus.

The data collection subsystem is invoked by the pseudo-macros:

COLLECT.MC, START.MC, VERAVERAGE.MC, or VERDISPLAY.MC. These RDOS

command modules set the necessary flags and bring into execution the

program modules necessary for the task at hand. The appendices provide

detailed program listings of the pseudo-macro commands and program

modules.



2.3 Experimental Application Overview


The details of the experimental applications that have been

conducted using the CPVER system are discussed in Chapter V. A brief

discussion of the experiments is included in this chapter for complete-

ness and to make this an independent chapter for reader convenience.

In the cognitive experiments conducted, cortical potentials were

recorded while subjects read and decided if simple propositions such

as "A robin is a bird" or "My name is Tony" were true or false. The

subjects were to decide rapidly and respond via switch activation. The

major purpose of the experiments was to collect the EEG, extract the










ERP and classify the single epochs into one of two possible classes,

e.g., correct or incorrect.

Five scalp electrode locations, F3, F4, C3, C4, Cz in the 10/20

EEG system, were to be monitored and digitized using nonpolarizing cup

electrodes. A timing/response channel, an EMG channel, and a diagonal

EOG channel were collected as well. Each EEG location was a monopolar

derivation to the linked mastoids. The data were to be digitized at

125 samples/second/channel (8 msec sampling interval on 1,000 Hz

sampling frequency over 8 channels). The amplifiers had a bandwith

from 1 to 50 Hz, with 50% attenuation at 1 and 50 Hz [45].

Prior to the conduction of the experiment, the operator performed

pre-operational checks on the equipment. The procedure and checklist

in Appendix 1 was used for this purpose. On the day of the experiment,

subjects were given appropriate instructions relevant to the particular

experiment. A 50 microvolt, 10 Hz calibrate signal was connected to the

input of all the EEG amplifiers. The operator verified proper operation

of the amplifiers by observing each channel on an oscilloscope and by

collecting and digitizing 512 samples from each channel.

The subject was then seated in a ventilated Faraday shielded

screen room and photic evoked response (ER) trials were collected

using a Grass Instruments xenon photo stimulator. The ER trials were

used to calculate an "average" visual evoked response that was dis-

played on an oscilloscope and visually inspected for proper shape and

magnitude. An invalid VER signal indicated probable amplifier mal-

function or improper electrode placement.










A practice session, using sentence presentations not used in the

experiment, was given to familiarize the subject with the task. The

visual stimulus was presented via a black and white TV monitor located

approximately 32 inches from the subject's eyes. The contrast and

brightness were adjusted to each subject's specifications for viewing

comfort. Each proposition was presented under computer control via

HP 2648A graphics terminal to which the TV monitor was connected.

The sentence, in the form of a proposition, was presented in three

segments to minimize eye movements, e.g.,

My name

is

Tony

The sequencing of the presentation of the sentence segments, with

respect to data acquisition, is presented in Table 2-1. The timing of

the segments and the subject's response were collected on one digitized

data channel. F3, Cz timing and the EOG were also monitored, via a

Grass Instruments polygraph, on-line during the experiments.

More than ten subjects have participated in various experiments

[17]. Experimental results and conclusions are also described by

Childers et al. [11,12,17,19] and Fishier et al. [1].



2.4 Real-Time Stimulus Presentation
And EEG Data Collection


The EEG experiment driver software is a set of FORTRAN IV routines

implemented on a Data General NOVA 4 computer using NOVA FORTRAN, Revi-

sion 5.52 and NOVA Assembler, Revision 6.6, running under the MAPPED










Table

Timing


2-1

Diagram


MSEC


EXAMPLE

1-1


EVENT

Fixation Box Appears

Begin Data Collection

Subject Presented

Subject Erased

Verb Presented

Verb Erased

Object Presented

Object Erased

Response Executed

End Data Collection

Message (If Any) Presented


Message Erased

Write Data To Disk And Start
Next Presentation


A Robin



Is Not



A Bird


0

400

800

1200

1600

2000

2400

2000-4048

4048

4048


4448

4448-7500










RDOS, Revision 6.62, operating system. The driver presents the state-

ment propositions in a three-segment format on the 2648A HP graphics

terminal, and collects the digitized EEG signal from eight A/D channels

operating at a composite 1000 Hz sampling frequency. It monitors the

EOG signal for eyeblinks and rejects and reschedules the stimulus

presentation, if the subject blinks during the response period. In

addition, the driver collects a digitized calibration signal from

each channel and presents photic stimuli to obtain an "averaged" VER

for a specified channel.

The structure chart [46] for the equipment driver is shown in

Figure 2-1. Each module represents a logical grouping of related

operations that will be described in this section.

Module Al's purpose is to define the default values of program

parameters and loop counters, to initialize the value of the experiment

statistics accumulators, and to input the experiment series parameter

table, subject initials, and experimental session number. The experi-

ment series parameter table is shown in Table 2-2. The first eight

lines contain the character values of the instructions stored in the

function keys of the 2648A HP graphics terminal. These are discussed

in conjunction with Module A6 in a later paragraph. The experiment

designer supplies most of the remaining information.

The concept behind this philosophy is that once an experimental

paradigm is designed, the values of the parameters that describe the

experimental session are fixed and the computer operator should not

have the option of respecifying them. This area is especially sensi-

tive when the experiments involve interdisciplinary teams, such as







































S A9: ON-LINE
CLASSIF.
I j


Figure 2-1 EEG Experiment Driver Structure Chart






29




Table 2-2

Experiment Series Parameter Table


Line
Number Contents Format

1-8 Value of H.P. 2648A Terminal, Function Keys 1-8 S80

9 Series designator, file types (1:10) 40(A1,1X)

10 No response feedback message S15

11 Eyeblink feedback message S15

12 Wrong answer feedback message S15

13 Reference channel,.default scale factors
eyeblink channel, eyeblink threshold, timing (II, IX, 8F4.1,
channel number, four-letter names for the eight 12, F8.2, 12,
channels digitized 16A2)

14 First channel number, last channel number,
number of photic stimuli 2014

15 Number of sentences in set, number of trials,
starting sentence number, number of sentence
classes, number of sentences per class, number
of trial classes, number of trials per class 2014

16 Sentence master filename S15

17 Delay parameters (3); number of samples to
collect after: 1) fixation box, 2) subject
term, 3) verb term; polarity of the response
{+1, -1} 2014

18 Number of partitions, number of samples/
partition, rectangular window shift for
stimulus and response lock, horizontal and
vertical scale dimensions of the graphics
terminal, number of channels to process,
channel number of each channel to be pro-
cessed 2014










those already described in the introduction. The experiment designer

not only designs the experimental paradigm, but also must interpret the

results; the engineers, computer scientists, technicians and operators

play a supporting role in this phase. The control of these parameters

stays with the experiment designer. Cognitive experiments must adhere

to this "modus operandi."

In line 9 of the parameter table, the experiment designer assigns

a one-character alphanumeric code to the series of experiments to be

conducted, e.g., S for the semantic experiments, E for the episodic

experiments, R for the replication of the semantic studies using the

same subjects as the espisodic experiments [17,19].

The file-type designators are assigned by the system designer.

These are arbitrarily assigned one alphanumeric character code desig-

nation for each different type of data file required by the entire

system. The current file-type designator convention is: D for EEG

files, C for covariance files, T for Toeplitz estimator files, U for

mean vector files, and V for discriminant vector files. All system

data files use the maximum ten-character filename allowed by the RDOS

operating system. The first five file-type designators represent a

choice for the first character of the filename.

Lines 10, 11 and 12 contain rejection feedback messages to be

displayed back to the subject, for example, EYEBLINK! These lines may

be left blank (no subject feedback) at the discretion of the experiment

designer. In the episodic experiment series, messages were found to

improve the data collection effort and had a "calming" effect on the











subjects. The number and frequency of occurrence of eyeblinks gives

an objective measure of whether or not the subject is "good."

A reference channel number, default scale factors, eyeblink chan-

nel number and threshold, the timing channel number and names for each

channel are supplied in line 13. The reference channel number and

default scale factors are described with the A2 and A3 Modules, the

eyeblink channel number and threshold, with the A7 Module. The choice

of a timing channel number is arbitrary, and is implied by the naming

of each channel, e.g., Channel 1 (F3), channel 2(F4), channel 3 (C3),

channel 4 (C4), channel 5 (EMG), channel 6 (Cz), channel 7 (EOG),

channel 8 (TIM).

Line 14 specifies a channel or sequence of channels where the

evoked response to the specified number of photic stimuli is collected,

digitized and averaged. In the episodic experiment series, fifty

repetitions collected on channel 6 (Cz) were used for this purpose.

Experiment-specific data are supplied in lines 15 and 16 of the

table. For the semantic experiment series, 144 different sentences

involving general knowledge constituted the master semantic set. The

subject was presented either four sessions of 72 sentences each or two

sessions of 144 sentences each. The semantic set consisted of sen-

tences of four types (classes); true affirmative (TA) (e.g., A hammer

is a tool), true negative (TN) (e.g., A hammer is not a plant), false

affirmative (FA) (e.g., A hammer is a plant), and false negative (FN)

(e.g., A hammer is not a tool). Each class contained 36 sentences, 4

classes, 18 sentences per class. The two-session set consisted of 144

sentences, 4 classes and 36 sentences per class.










In the episodic experiments [17,19], 72 sentences involving facts

about the individual subjects were used, e.g., My name is Tony. The

sentences consisted only of two types (classes) (TA and FA). Each

type contained 36 sentences. Subjects were presented four different

sessions of 72 sentences, 2 classes, 36 sentences per class, each.

The filename of the master set of sentences is supplied in line 16 and

is normally five characters long. If a longer name is supplied, the

system will automatically replace characters 6 8 with the subject's

initials. This feature is useful for subject dependent sentence files,

as those used in the episodic experiments.

The delay parameters, the sample collection breakdown, and re-

sponse switch polarity are supplied in line 17 of the table. These

parameters give the experiment designer and the biomedical engineer

flexibility in the control of the segment-presentation/data-collection,

A6 Module.

Line 18 contains specifications for the data processing subsystem.

These parameters are discussed in Section 3.4 of this work.

The operator is requested to input the subject initials (3) and

session number. The system constructs the filename for the first EEG

record as follows: The character stored in the first file-type desig-

nator, the three subject initials, the experiment series designator,

the session number, and a four digit (including nonsignificant zeroes,

e.g.,0001) sentence number. For example, in the episodic experiments,

the data for sentence number 24 of session 3 for subject RCP are stored

in the file whose filename is DRCPE30024; in the semantic experiments,

the data for sentence number 17 of session 4 for subject MWF are stored










in filename DMWFS40017; in the semantic replication experiments, the

data for subject JNL, sentence number 142 of session 2 are stored in

filename DJNLR20142, and so on.

Module A2 is operator enabled/disabled and is typically enabled

once for each subject. The module defaults the loop control parameters

of Module A5 and line 17 of the specifications table, to make a single

pass through Module A6 and enables passage to Module A3 via flag ICA,

e.g., ICA=0 means do not calibrate, ICA=1 means calibration is desired.

A six-character filename consisting of the letter C, the subject's

initials, the experiment series designator and the session number, is

created for the storage of the digitized 512 samples/channel, 50

microvolt, calibrate signal connected to each of the EEG amplifiers.

Module A3 is entered after a single pass through Module A6, which

was enabled by Module A2. The module clears the display memory of the

HP terminal, calculates the maximum absolute value for each of the

digitized channels, and computes a scale factor based on the largest

peak-to-peak difference in the immediate vicinity of the maximum

value (to compensate for amplifier drift). These factors are displayed

on the master console, so a determination of "relative" amplifier per-

formance and stability can be made. Amplifiers may then be recalibrated

and the process repeated.

Module A4 is also operator enabled/disabled. Its purpose is to

present a specified set of photic stimuli and collect photic evoked

response trials, that are averaged and used for a baseline determina-

tion. The subject must be fully prepared in order to proceed with

the module. The system creates a six-character filename in the same










manner as the calibration signal filename, using the letter V for a

first letter, and stores in it the averaged VER, after displaying it

on an oscilloscope. After visually inspecting the signal on the

oscilloscope the electrodes may be repositioned and the process re-

peated.

Module A5 uses the information in the specifications table to set

up the stimulus presentation module. The system then searches for a

file whose name is created from the experiment series designator, the

letters SEQ, and the session number. This file is assumed to contain

the randomized order for sentence presentation. The system prints out

a subset of the default specifications for the session, and creates

the filenames that will contain the digitized EEG signals. By doing

this, the system not only insures that there is enough disk space to

store all of the data (prior to the conduction of the experiment) but

also saves one second of CPU time between each trial presentation (more

time is required to create, allocate space and write disk blocks than

it does to restore data on direct-access files).

Module A6 presents the sentence in three segments, collects the

EEG signal, and stores the digitized data samples in the appropriate

file. The presentation of the sentences in segments is made possible

by the exploitation of the capabilities of the HP 2648A graphics

terminal. This terminal allows the user to load from one to eight,

eighty-character instruction sequences into its memory, and allows

triggering of the sequences by depressing a button (these are called

"Function Keys" Fl to F8) or via command string. Instruction sequen-

ces for one key may contain commands to trigger another key. This










allows for the off-line storage of commands that will, in fact, cause

the HP to perform operations asynchronous from the host computer.

Function keys Fl, F8 and F2 load the subject term into graphics memory

while inhibiting the display; keys F3 and F4 turn on the graphics

display for 400 ms, turn it off, and load the verb into graphics

memory; keys F5 and F6 turn on the graphics display for 400 ms, turn

it off, and load the object term into the graphics memory; and key F7

turns on the graphics display for 400 ms, turns it off, delays for 2

seconds, then places an asterisk on the fixation box. Timing signals

are derived from the BELL character which is triggered by F8, F3, F5,

and twice by F7. After adjusting the real-time clock (A/D system turns

it off), the system writes the data on disk, stores the new sentence

in keys F2, F4 and F6 and triggers key Fl in preparation for another

trial.

Module A7 examines the eyeblink channel for a threshold violation,

the timing channel for missing timing pulses or response pulse,

examines the response pulse for the proper polarity, and determines

if the operator has requested a keyboard interrupt. If any of these

conditions exist, the system will reschedule the trial for later

presentation. The subject is given visual feedback in the case of

eyeblinks, no response, or erroneous response. The system pauses in

the case of missing timing pulses or operator keyboard interrupt

request, and the entire screen is erased. A message that is not

visible to the subject is displayed for operator intervention. The

operator has the option of aborting (in the case of equipment failure)

or making adjustments and continuing with the session.










Module A8 updates the filename for the next sentence to be pre-

sented, and in the event of a rejection, it updates the set of pointers

that index each sentence presentation such that the rejected sentence

is presented again at the end of the session.

Module A9 is only enabled for the "on-line" biofeedback experi-

ments. It contains the "current" parameters of the classifier/feature

extractor algorithm so as to decide on a computer-assigned class for

the single epoch just extracted. Based on this class assignment the

system may now display visual feedback to the subject and further

modify the stimulus presentation. Experimental paradigms that will

take advantage of this mode of operation are currently being developed

by Bloom [3].

Module A10 displays (on the operator's console) the run-time

statistics, for the just completed session, that are to be kept in a

laboratory manual for future reference.

A single experimental session is completed, on the average, in

approximately seven minutes (72 sentences, 10 rejects). The operator's

console displays each sentence to be presented, rejection data and

the value of the current pointer on the upper left hand portion of the

CRT, not visible to the subject. A single operator can monitor the

master console and the polygraph recorder in order to observe the pro-

gress of the experiment once it is begun. The equipment has been

physically arranged to facilitate the single operator concept.

Support routines for the experiment driver module include 1)

multichannel A/D conversion with external clock running in interrupt-

disabled mode, 2) single and dual-channel D/A conversion, 3) software










pulse generator, 4) keyboard interrupt facility, 5) interrupt-disabled

FORTRAN "TYPE" statement, 6) interface to the operating system. These

routines are discussed in the appendices.



2.5 Data Averaging and Display


Nearly all the cognitive ERP work reported in the literature has

been done with averages [11]. Though the CPVER system was designed

for single epoch processing (unaveraged EEG records), the averaged

VER does provide an objective measure of validity, and a basis for

comparison with other research. Certain types of artifacts are better

identified on the averaged VER or on three-dimensional data displays

than by examination of the single epoch. Visual inspection of the

EEG data has been recommended by Childers et al. [17,19] and John

et al. [13].

Experimental session averages are obtained by invoking the

VERAVERAGE.MC pseudo-macro, and experimental session three-dimensional

displays by invoking VERDISPLAY.MC pseudo-macro. Single sentence dis-

plays are obtained by invocation of the SVERDSPL.MC or the SVERTYPE.MC

pseudo-macro. The appendices provide detailed program listings of the

pseudo-macro commands and program modules.



2.5.1 The EEG Data Averaging Module


The EEG data averaging structure chart is depicted in Figure 2-2.

Module Bl's purpose is to define the default value of the loop param-

eters, to initialize the EEG array accumulators, and to input the






































































Figure 2-2 EEG Data Averaging Structure Chart










experiment series parameter table, subject initials and experimental

session number. This table is the same table discussed previously

and depicted in Table 2-2. Refer to Section 2.4 for an expose'of

the parameter table.

Module B2 extracts the stimulus-aligned or response-aligned EEG

for each channel, and accumulates the data in an array. This is re-

peated for each trial in the session.

Module B3 computes the arithmetic mean for each channel and

evaluates the maximum signal amplitude for scaling purposes.

Module B4 sets the graphics display of the HP terminal and plots

the averaged VER signal for each class and desired channel.



2.5.2 EEG Data Display Module


The EEG data display routine structure chart is depicted in

Figure 2-3. Module C1 is analogous to Module Bl of the data averaging

routine of Section 2.5.1.

Module C2 extracts the stimulus-aligned or response-aligned EEG

for a selected channel and Module B3 outputs the EEG signal to the

DEVELCO 3-D unit. The operator examines each single epoch as it is

displayed on the DEVELCO unit, and if desired, issues a keyboard

interrupt in order to zoom in on the desired epoch. Module B4 senses

the keyboard interrupt and gives the operator the option of selecting

any of the last five epochs displayed in the DEVELCO unit, for display

on a Tektronics oscilloscope. The operator is given the option of

making an entry in a sequential file, called the REJECTS file, which



























































Figure 2-3 EEG Data Display Structure Chart










may be used for future reference. The REJECTS file has been used for

"off-line" artifact rejection and EEG session editing [17].



2.5.3 Single Epoch Display


The single epoch data display or type routine structure chart is

depicted in Figure 2-4.

Module Dl requests the subject's initials, the experiment series

designator, the session number, and the trial number. Module D2 dis-

plays the data on an oscilloscope or outputs the digitized sample

values on the operator's console, or both, for a given EEG channel.



2.6 Data Storage Conventions


The DGC 6070 Moving Head Disk storage facility was used to store

the data collected by the CPVER system. Three 10 MB disk packs labelled

DISK 300, DISK 301, and DISK 302 were used for this purpose. Data for

both the episodic experiments and the replication of the semantic

experiments (using two of the subjects who participated in the episodic

experiments, see Chapter V for details) are stored, in subject order,

on these disk packs. Table 2-3 summarizes the data storage convention

adapted for this purpose.



























































Figure 2-4 Single Epoch Display Structure Chart









Table 2-3

CPVER Disk Storage Convention


Disk Experiment Series
No. Designator


Subject Session


First Trial/Last Trial
Filenames


Episodic(E)


Semantic
Replication(R)

Semantic
Replication(R)


Episodic(E)




Episodic(E)




Episodic(E)




Episodic(E)


DHG
TA
SNS
SED
STJ
GMH
SLV

JNL


RCP


JNL




RCP




MAD




CJB


1-4
1-4
1-4
1-4
1-4
1-4
1-4


None*
None*
None*
None*
None*
None*
None*


DJNLR10001-DJNLR10144
DJNLR20001-DJNLR20144

DRCPR10001-DRCPR10144
DRCPR20001-DRCPR20144

DJNLE10001-DJNLE10072
DJNLE20001-DJNLE20072
DJNLE30001-DJNLE30072
DJNLE40001-DJNLE40072

DRCPE10001-DRCPE10072
DRCPE20001-DRCPE20072
DRCPE30001-DRCPE30072
DRCPE40001-DRCPE40072

DMADE10001-DMADE10072
DMADE20001-DMADE20072
DMADE30001-DMADE30072
DMADE40001-DMADE40072

DCJBE10001-DCJBE10072
DCJBE20001-DCJBE20072
DCJBE30001-DCJBE30072
DCJBE40001-DCJBE40072


*Data rejected due to instrumentation failure.
















CHAPTER III
IMPLEMENTATION OF CPVER
DATA PROCESSING SUBSYSTEM



3.1 Introduction


The implementation details of the EEG data processing subsystem

are described in this chapter. The ultimate goal of the signal pro-

cessing task is to recognize single visual evoked responses to meaning-

ful stimuli, and to classify the single epochs into one of two possible

classes. In cognitive studies of semantic memory the stimuli cannot

be repeated numerous times because, presumably, the response to pre-

vious sentence propositions could become overlearned. These experi-

ments demand special consideration since relatively few processing

algorithms are available for the small sample case, and the number of

samples per class cannot be increased.

Signal extraction methods have been used to extract the ERP, but

these methods are least valid for the extraction of longer latency

components associated with cognitive activity. Signal classification

methods, though more difficult and complex, are often capable of

providing accurate results in cases where it would be useless to use

signal extraction methods [13]. CPVER uses the signal classification

methodology developed by Childers et al. [11] to accomplish this task.

A "training set," e.g., a set of labelled epochs of known classes

(design set), is required to guide the data reduction (feature extrac-

tion) procedure in the selection of an optimal set of features, and











to design the decision rule. The performance of the combination fea-

ture extraction/classifier procedure is estimated from the population

probability of classification error (PE). An independent set of trials,

known as the test set, is used to obtain an estimate of the PE. The

(PEd, PEt) determine the performance of the signal classification

method.

These two blocks of epochs must be collected before attempting

any real-time classification (defaulting). The software realization

of the signal classification methodology need not be implemented in

real-time. Chapter IV describes real-time interface considerations

and possible implementation.

The remainder of this chapter details the off-line EEG data pro-

cessing subsystem. The software philosophy is described from the

system designer's point of view. The chapter is subdivided into the

following sections: 1) Data Processing Subsystem Overview, 2) Experi-

mental Application Overview, 3) Feature Extraction Methodology, 4) Fea-

ture Extraction Software, 5) ERP Classification Software, 6) Data

Storage/Data Processing Conventions.



3.2 Data Processing Subsystem Overview


The purpose of the CPVER data processing subsystem is to extract

a number of features from the observed EEG records, that contain the

discriminatory information, required to classify single visual evoked

response trials into one of two general classes. The subsystem yields










the specifications for a linear Bayesian decision rule, along with an

estimate of the decision rule performance (combined feature extraction/

classification procedure) expressed as the probability of classification

error for the population (PE). The software has been designed as an

"off-line" application with respect to the data collection task.

Three major routines comprise the data processing software: A

Covariance Matrix Estimator module (COVEST), a Discriminant Vector

Estimator module (DVEST), and a Scatter Plot and Classifier Generator

module (SCATTER). The latter is used to display the projections of

the observed data onto feature space (scatter diagram).

The data processing subsystem is invoked by the pseudo-macros:

PROCESS.MC, COVARIANCE.MC, FEATURES.MC, CLASSIFY.MC, RESCATTER.MC, or

DESIGN.MC. These RDOS command modules set the necessary flags and

bring into execution the program modules necessary for the completion

of the task. The appendices provide detailed program listings of the

pseudo-macro commands and program modules.



3.3 Experimental Overview


The details of the experimental applications that have been con-

ducted using the CPVER system are discussed in Chapter V. A brief

discussion of the data processing aspects of the experiments is in-

cluded in this chapter for completeness, and to make this chapter

independent for reader convenience.

In the cognitive experiments conducted, cortical potentials were

recorded while subjects read and decided if simple propositions such










as "A hammer is a tool" or "My last name is Arroyo" were true or false.

The subjects were to decide rapidly and respond via switch activation.

The major purpose of the experiments was to collect and digitize the

EEG signal, extract the ERP, and classify the single visually evoked

response (VER) into one of two possible classes, e.g., correct or

incorrect.

Five scalp electrode locations, F3, F4, C3, C4, and Cz in the

10/20 EEG system, were to be monitored and digitized using nonpolarizing

cup electrodes. A timing/response channel, an EMG channel, and a diag-

onal EOG channel were collected as well. Each EEG location was a mono-

polar derivation to the linked mastoids. The data were to be digitized

at 125 samples/second/channel. The amplifiers had a bandwith from 1

to 50 Hz, with 50% attenuation at 1 and 50 Hz.

The sentences, in the form of a proposition, were presented in

three segments to minimize eye movements, e.g.,

A hammer

is

a tool.

The sequencing of the presentation of the sentence segments, with

respect to data acquisition, was presented in Table 2-1. Fifty samples/

channel were collected before the subject term (S-term), one hundred

samples/channel between the S-term and verb, one hundred samples/

channel between the verb and object term (0-term), and two hundred and

sixty-two samples/channel after the 0-term.

In one experiment, there were 144 different sentences involving

general knowledge, which were presented twice to each subject. The










sentences consisted of four types; true affirmative (TA), true negative

(TN), false affirmative (FA) and false negative (FN). Each type con-

tained 36 statements.

In a second experiment, there were 72 different sentences involv-

ing facts about the individual subject, which were presented four times

to each subject. The sentences consisted of two types, TA and FA and

each type contained 36 statements.

The measurement vector was chosen to be a 720 msec (Chapter V)

interval of the EEG. The interval may start with the onset of the

0-term or may be shifted by a specified number of samples from the

0-term (stimulus-lock). Synchronization may also be chosen with re-

spect to the response pulse with/without shifting (response-lock).

Since there were 8 msec between data samples, this gave a 90 component

measurement vector. The observation recorded was partitioned into

segments, such that the ratio of the number of samples per class to

partition size was 3 or 4 [12]. The partition size was chosen as 10,

yielding a reduced nine-element feature set, one feature/partition.

The size 10 was selected after conducting experiments on the correlation

between successive samples, etc.

One or two discriminant vectors were calculated for each partition

of dimension 10. A criterion value (a type of signal-to-noise ratio)

was computed for each feature. The two features with the largest S/N

ratios were chosen for the representation.

The design set was projected onto the selected two-feature space

and a linear classifier rule was derived. The PEd was obtained by

counting the number of trials that were misclassified by this linear










decision boundary. The process was repeated on the test set to obtain

PEt. The parameters of the decision rule may be stored for further

processing.

Experimental results and conclusions described by Childers et al.

[11,12] and Fishier et al. [1] were obtained using the data processing

subsection of the CPVER system.



3.4 Feature Extraction Methodology


The goal of the feature extraction procedure is to obtain a subset

of components from the measurement vector that enhance the chances for

proper classification. An optimal number of features may be found that

minimizes the probability of error. The feature selection algorithm

of Roucos and Childers [12] is basically a special ordering of the

eigenvectors extracted from the covariance matrix, estimated from the

data.

An evaluation of the method is obtained by using the hold-out

method to estimate the PE (a training set is used to design the feature

extraction/classifier and both design and test sets are used to estimate

the respective PEs). A feature discriminantt) vector is obtained by

Equation 3-1 where


-1
d = a W1 u (3-1)



u denotes differences between the mean class vectors (i.e., u=u2-u, for

a two class problem), W is a linear weighting of the covariance matrices









th
C. for the i class, and a is a scale factor that normalizes d to unity.

W is usually chosen as the arithmetic mean of the class covariances (e.g.,

W = (C1 + C2) for a two class problem). Equation 3-1 maximizes the

distance between classes in feature space given by the divergence form-

ulation [32].

The value of d as a feature may be measured by the generalized

Fisher ratio.

J(d) = (d' u) (3-2)
d' W d


A set of P features is obtained by sequentially maximizing

Equation 3-2 as follows:

1) Find the first feature d by maximizing J. This is done
--
with Equation 3-1. d1 is the only eigenvector of W u u'

with a non-zero eigenvalue X..

2) Find feature i < P by maximizing J subject to i-i orthogonality

constraints, i.e.,

d! d = 0 n = 1, 2, ., i- (3-3)
-n


The measurement vector of dimension N is decimated into K parti-

tions, each of dimension L (N = KL) in order to reduce the dimension-

ality of the observed data. For each partition of dimension L, an

optimal feature extractor is used to obtain a features) to represent

the partition. Stepwise selection is applied to the partitions to

obtain a set of P features from the set of features defined for all the

partitions.

The partition size L should be chosen as large as possible for

statistical stability. Given a fixed number M of observations











per class, L is selected such that the ratio M/L is equal to 3 or 4

[11,12].

Consider the alternate expression for the Fisher ratio where e. is


= (eI u)2
Y -. (3-4)
1

an eigenvector of the common covariance matrix C with eigenvalue X.
1

and u is the mean difference vector between two classes.

After ordering the eigenvectors e. by decreasing value of yi (the
-1i

Fisher ratio), the discriminant vector d is given by


L
d = a -i e. (3-5)
i=l .
1

A new feature vector b(m) is defined from Equation 3-5 by choosing

M L components [11,12,33], i.e.,


m
e' u
b(m) = a E -- e. (3-6)
i=l
Xi

The average S/N ratio of b(m) is the average criterion [32], assuming

a design set of M observations was used to estimate the covariance

matrix and the mean vectors.


J(m) = S/N(m) = M F(m) + 2 r(m) (3-7)
M r(m) + 2m

where
m
r(m) = E y (3-8)
i=l


and SNR(d) = J(d) in Equation 3-2 with C = W.











The discriminant vector is obtained from Equation 3-6 by: 1)

Choosing m = mo such that F(mo) does not change by more than 10%

when m is increased, 2) estimating u from the design set, and 3)

having a good estimate of C, e.g.,

m
o e' u
b = b(m ) = a Z -- (3-9)
-- -- (3-9)
i=l A.
1

There are at least two methods to estimate the covariance matrix

when the design set size M is small. First, if the data set is assumed

to be stationary over the measurement interval, then the covariance

matrix estimate may be obtained by averaging the individual partition

matrices. Second, the data set may be assumed to be piecewise station-

ary, e.g., stationary over the duration of a partition. For such a

process, the covariance matrix is Toeplitz. The Toeplitz estimator

replaces all the elements of the matrix diagonals by their arithmetic

mean.

Conventional discriminant analysis is obtained from Equation 3-5,

with the eigenvectors e. derived from the individual partition covariance

matrices Ci, and ordered by decreasing value of the eigenvalues X..

The matrices C. for each partition i, may be assumed to be Toeplitz if

the data is assumed to be piecewise stationary.

To evaluate the feature extraction procedure, the design data set

is projected onto feature space, a linear decision rule is computed,

and an error count is made for the design and test sets. The projection

from data space to feature space is done by forming the dot product

between the feature vectors and the data. For example, suppose d (i) is
-=l










a feature vector for partition i, and d2(j) is a second feature vector

for partition j. The projection becomes

u= (i) x
(3-10)

v= d(j) .

th
where x. is the EEG data for the i partition. The coordinates (u,v)
--1

in feature space may be used to plot the class value for each experiment

trial. A linear decision boundary is then superimposed on the coordinate

space, and a count made of the trials that are misclassified by the de-

cision rule. The error-count-to-number-of-trials ratio is an estimate

of the PE. The decision boundary is chosen as the perpendicular bisector

of the centroids of the two classes, which is shifted and/or rotated

until the design PE is minimized.

Large discrepancies between PEd and PEt may exist due to the bias

and variability of the feature extraction rather than the classification

method.

The feature extraction methodology of the CPVER system is summarized

in Table 3-1 and Table 3-2. Table 3-1 uses the methodology developed by

the University of Florida team of Childers et al. [11,12,17,19]; Table 3-2

is the conventional discriminant analysis method also used by Foley and

Sammon [40, 31] and Roucos [32,33].



3.5 Feature Extraction Software


The feature extraction software implement steps 1 through 4 of

Tables 3-1 and 3-2. Two major routines are used for this purpose: A











Table 3-1
CPVER Feature Extraction/Classification Algorithm--UF Method



Item Instruction

1 Partition the EEG observation vector x of dimension N into K

partitions xk k = 1, 2, ..,K of dimension L, e.g., N = KL

and x = { x1 I x2 j... x }. x is a subset of samples

extracted from the stimulus-aligned or response-aligned single

EEG record.
th
2 Estimate the covariance matrices Cki for the k partition

and the ith class from an experimental session of M samples/

class. Calculate the Toeplitz estimator Tki for each Cki.

Average across partitions the Cki and Tki to obtain C. and

T., e.g.,
SK K
S^ 1 ^ 1 ,
Ci E C T E T
i Kk=1 ki i K k ki


Calculate mean vectors u for each partition.
-k
3 Evaluate the eigenvalues Xk and eigenvectors e of

WA = 1/2(C1 + C2) and WT = 1/2(T1 + T2)

Calculate the Fisher ratios Yk by Equation 3-4, k = 1, 2, ...,K

for each of WA and WT.

4 Sort the eigenvectors ek in decreasing order of Fisher ratio

Yk. Calculate F(m) (Equation 3-8) by increasing m until F(m)

does not change by more than 10% (selected threshold). Eval-

uate b using Equation 3-9 and J(m) by Equation 3-7 for each
partition k. These are our features.
partition k. These are our features.






55




Table 3-1 continued

5 Select the two features with the largest criterion value J(m).

Project the X from the design set onto the selected feature

space. Compute a decision boundary from the perturbation

(position and angular) of the perpendicular bisector of the

centroids of the two classes.

6 Compute the probability of error for the design and test sets

via classification error count.











Table 3-2
CPVER Feature Extraction/Classification Algorithm
Discriminant Analysis


Item Instruction


1 Partition the EEG observation vector x of dimension N into K

partitions Xk k = 1, 2, ...,K of dimension L, e.g., N = KL

and x = {1 1 x2 ... I x} = {xk}. x is a subset of samples

extracted from the stimulus-aligned or response-aligned single

EEG record.

2 Estimate the covariance matrices C for the k partition and
ki
th
the i class from an experimental session with H samples/class.

Calculate the Toeplitz estimator Tki for each C Calculate

mean vectors uk for each partition.

3 Evaluate the eigenvalues ek and eigenvectors Xk of
A A A A
Ck = 1/2(Ckl + Ck2) and Tk = l/2(Tkl + Tk2

and calculate the Fisher ratios by Equation 3-4.

4 Calculate r(m=L) using Equation 3-8 and d by Equation 3-5 for
_=l
each partition. These are our features with F(m=L) as our

criterion value. A second feature vector is calculated sub-

ject to the orthogonality constraints previously discussed.

5 Select the two features with largest criterion value r(L).

Project the k from the design set onto the selected feature

space. Compute a decision boundary from the angular and

positional perturbation of the perpendicular bisector of the

two classes.

6 Compute the probability of error for the design and test sets

via classification error count.











Covariance Estimator module (COVEST) that implements steps one and two,

and a Discriminant Vector Estimator module (DVEST) that implements

steps three and four for both algorithms. The current version of

CPVER extracts a single feature discriminantt vector) for each parti-

tion using the UF algorithm, and two features per partition using the

discriminant analysis algorithm.

The feature extraction modules are invoked by the PROCESS.MC,

COVARIANCE.MC, or FEATURES.MC RDOS pseudo-macros. PROCESS.MC invokes

all the data processing subsystem's modules of CPVER for "off-line"

operation. COVARIANCE.MC and FEATURES.MC invoke COVEST and DVEST,

respectively, for independent execution.

The structure chart for COVEST and DVEST are shown in Figures 3-1

and 3-2, respectively. Each submodule represents a logical grouping

of related operations (a structural block) that will be described in

the following sections.



3.5.1 Covariance Estimator Module-- COVEST


COVEST is a set of FORTRAN IV routines implemented on Data

General's NOVA 4 computer using NOVA FORTRAN,Revision 5.52, running

under the MAPPED RDOS (MRDOS),Revision 6.62,operating system. The

program requests inputs such as the name of the parameter and specifi-

cations table, the subject's initials etc. Outputs generated include

the set of individual partition covariance matrices for each class,

the partition Toeplitz estimator matrix for each class, the average

across partitions matrix for each class and the average Toeplitz

estimator across partitions for each class.





































































Figure 3-1 Covariance Matrix Estimator Structure Chart


































































Figure 3-2 Discriminant Vector Estimator Structure Chart











Submodule El (Figure 3-1) initializes and sets up the COVEST

program. Two externally supplied flags control the operation of the

software: flag INDV is set to enable the output of individual partition

covariances for calculation of features using discriminant analysis,

and flag AUTO is set for automatic, input-free operation. In manual

mode, the system requests the following inputs from the operator:

1) the name of the parameter and specifications table (Table 2-2),

2) the subject's initials, 3) the synchrony point designator, and 4)

the experiment session number. In automatic mode, these inputs are

pre-stored in a sequential character file named ANSWERS.

The submodule opens the specifications table for read-only access,

and extracts the value of the pertinent subset of the parameters that

were previously used to collect the data. The specifications table

was previously described in section 2.4 of this work. The parameters

used by COVEST include: 1) The experiment series and file-type desig-

nators of line 9 of the table; 2) the number of the timing channel

and channel names of line 13; 3) the number of sentences, number of

trials, starting sentence number, number of classes in the master

sentence set, number of sentences per class for the master set, number

of classes presented, and number of sentences per class presented of

line 15; 4) the polarity of the response of line 17; and 5) all of

line 18. This line contains the data processing specifications alluded

to in step one of Tables 3-1 and 3-2, mainly, 1) the number of parti-

tions k, the length of each partition L, the synchrony point displace-

ment windows, the horizontal and vertical scale factors for the graphics

display, and the number of channels and channel numbers to process.










The submodule concludes by displaying portions of the specifications

table and creating the filename for the first covariance matrix to be

generated. This filename adheres to the convention established for

data collection and modified for data processing. This convention is

discussed in Section 3.7.

Submodule E2 extracts the EEG observation vector xk of either

stimulus-aligned or response-aligned data for the design session. The

system automatically generates the correct filename for the digitized

EEG signal, based on the random sequence file of pointers that was

used during the data collection session. A general purpose subroutine

to read the EEG associated with each sentence (SREAD) is invoked for

this purpose (consult the appendices for a detailed description of

SREAD). The submodule accumulates the vector sum, and sum of squares

for each partition/channel combination. This is repeated until the

specified number of sentences/class has been extracted (specifications

table line 15). This number is displayed on the operator's console

upon exit.

Submodule E3 computes the mean vector and covariance for each

partition/channel combination. Submodule E4 calculates the Toeplitz

covariance estimator for the given channel number and partition. Sub-

module E5 accumulates both the individual covariance and individual

Toeplitz covariances. The individual covariances are written to disk

if the external flag INDV is set to one. The mean vector is written

to disk and the process repeated (to block E3) for each partition.

Submodule E6 calculates the average covariance and average Toeplitz

matrix across partitions. These two matrices are written to disk











and the process repeated (to block E3) for each channel. The class

is automatically incremented and another pass is made beginning at

block E2.

COVEST creates (number of partitions K) x (number of channels) x

(number of classes) mean vector files; (1 +(INDV x K)) x (number of

channels) x (number of classes) covariance files; and (1 +(INDV x K))

x (number of channels) x (number of classes) Toeplitz files. Each

mean vector file contains (N + 1) binary real numbers (2 words per

number); each covariance or Toeplitz file contains (1 + N x N) binary

real numbers. Each file requires a minimum of one disk block (256

words) of disk storage plus 18 words of directory space for the RDOS

file pointer. The program requires a 36 KB main memory partition for

execution. In the UF experiments, execution times varied linearly

according to the product (1 + INDV) x (number of channels) from a

minimum of two minutes to a maximum of 12 minutes of total elapsed

time.



3.5.2 Discriminant Vector Estimator Module --DVEST


DVEST is a set of FORTRAN IV routines implemented on Data

General's NOVA 4 computer using NOVA FORTRAN,Revision 5.52, running

under the MAPPED RDOS (MRDOS),Revision 6.62,operating system. The

program requests inputs such as the name of the specifications and

parameters table, the subject's initials, etc. Outputs generated

include a table of criterion J values (S/N ratios) for each partition/

channel combination for both algorithms (UF and Discriminant Analysis

(DA)) and the corresponding features discriminantt vectors).











Submodule Fl initializes and sets up the DVEST program. This

submodule is identical in structure to submodule El, described in

Section 3.5.1. The discriminant vector filenames adhere to the con-

vention discussed in Section 3.7.

Submodule F2 forms the joint average across partitions covariance

matrix W and obtains the eigenvalues and eigenvectors by invoking sub-

routine EIGEN (available from the IBM FORTRAN Scientific Subroutine

Package, see the appendices for detailed documentation). The sub-

module displays the eigenvalues and eigenvectors on the operator's

console upon exit.

Submodule F3 implements steps 3 and 4 of Table 3-2, the discriminant

analysis method. The procedure is implemented with subroutines FEAT,

VECROT and INVERSE. The outputs of the submodule are the criterion

values J (S/N ratios) and the two feature vectors for each partition.

Submodule F4 calculates the Fisher ratios for the UF method using

Equation 3-4, obtains a set of indices for the decreasing order of Fisher

ratios, computes F(mo) by increasing mo until F(mo) does not change by

more than a threshold (5%), and evaluates the criterion J(m ) and b
o -o
(the feature vector). The submodule invokes subroutine SORT to calcu-

late the eigenvector/eigenvalue order according to the decreasing value

of the set of Fisher ratios {yi}.

Submodule F5 displays the filenames used to store the feature

vectors and writes the set of features from the algorithms out to disk.

The process, from submodule F3 on, is repeated for each partition. The

procedure, from submodule F2 on, is repeated for each channel and covar-

iance matrix type combination.











Submodule F6 displays, on the operator's console and on the

attached hardcopy device (HP-IB device), the S/N or criterion J tables

for each algorithm, covariance source type, partition and channel

combination.

DVEST creates 2 x (number of channels) x (1 + INDV) x (K parti-

tions) discriminant vector files of (1 + 2 x N) real numbers. Each

file requires a minimum of one disk block (256 words) of disk storage

plus 18 words of directory space for the RDOS file pointer. The

program requires 36 KB of main memory for execution. In the UF experi-

ments, execution time varied linearly according to the product (1 +

INDV) x (number of channels) from a minimum of 2.5 minutes to a

maximum of 10 minutes of total elapsed time.



3.6 ERP Classification Software
The SCATTER Module


The ERP classification software implements steps 5 and 6 of

Tables 3-1, 2 The SCATTER module displays the projections of the

observed data onto the selected feature space (scatter diagram);

automatically scales the scatter diagram to the dimensions of the

graphics terminal; designs the linear classifier by selecting the

perpendicular bisector of the centroids of the two classes, followed

by shifts and/or rotation of the line until the PE is minimized; re-

draws the design scatter diagram and the classifier line, computes

the design PE, and prints the figure captions; draws the test set

scatter diagram with the classifier line, computes the test set PE,










and prints the figure captions; and stores the parameters of the class-

ifier line in a file for future processing.

The ERP classification module is invoked by the PROCESS.MC,

CLASSIFY.MC, RESCATTER.MC, or DESIGN.MC RDOS pseudo-macros. PROCESS.MC

invokes all the CPVER data processing subsystem modules for "off-line"

operation. CLASSIFY.MC and RESCATTER.MC invoke the SCATTER module for

independent execution. DESIGN.MC invokes the SCATTER module in

"automatic" mode.

The structure chart for SCATTER is shown in Figure 3-3. Each

logical grouping of related operations (structure block or submodule)

is described in the following paragraphs.

Submodule GI initializes and sets up the SCATTER program. Three

externally supplied flags control the operation of the software: flag

INDV is set to enable the discriminant analysis method, flag AUTO is

disabled for manual operation, and flag REDRAW is set to bypass the

recalculation of the design and test set feature space projections,

and assume they are stored in files DESIGN and TEST respectively. In

manual mode, the system requests the following inputs: 1) the symbols

to plot for each class in the scatter diagram, 2) the name of the

specifications table, the subject's initials, the synchrony point and

session numbers for both design and test sets, 3) a selection of the

algorithm and covariance type, 4) the desired channels) and features,

and 5) explicit control of the looping parameters of the bisector

perturbation algorithm in submodule G5. In automatic mode, a single

pass is made through the submodules (design set) and the decision



































































Figure 3-3 Scatter Structure Chart











boundary line parameters are saved in a file named CLASSIFIER. The

structure of submodule GI is essentially that of submodule El or Fl

discussed in the previous section.

Submodule G2 extracts the EEG observation vector xk of either

stimulus-aligned or response-aligned data for the design or test set

or both. The system retrieves the feature vectors for the design set

and forms the dot product of the feature vectors with the observation

vector.

The calculation is repeated for the test set and the projections

are saved in files DESIGN and TEST respectively. The submodule invokes

subroutine SREAD to extract the EEG signal associated with each sentence

presentation.

Submodule G3 requests the channel number and features to use in

the scatter diagram. A scale factor is derived from the projection

maximums for each axis. The submodule exits by requesting the process-

ing mode: design, test or fine tune.

Submodule G4 draws the scatter diagram and calculates the centroids

of the two classes in feature space,for either design (or fine tune) or

test sets. If in fine-tune mode, loop parameters for the perturbation

algorithm are requested. The system calculates the perpendicular

bisector of the centroids of the two classes, then shifts the point of

interception of the perpendicular line with the line drawn between the

two centroids. The perpendicular line is then rotated a maximum of

+900 to -90 from the current angular position.

Submodule G6 calculates the PE, via error count, for each tentative

decision line. Each time the PE of the design set is decreased, the

system "saves" the coordinates of the point of interception, the slope,











the y-intercept, the PE, and draws the new decision line. This process

continues until the design PE is zero or the perturbation algorithm

terminates. The system clears the graphics memory and redraws the

design scatter plot.

Submodule G7 draws the decision line that was "saved last (the

first occurence of the lowest PE that is "closest" to the perpendicular

bisector line), and calculates the PE based on this decision rule.

The submodule concludes by writing in graphics text mode, the scatter

diagram captions in order to obtain a "hardcopy" of the graphics memory.

In automatic mode, the system stores the parameters of the decision

boundary line in the sequential file CLASSIFIER and exits. In manual

mode, the operator is prompted for additional channel/feature combina-

tions as desired.

In the experiments conducted using the CPVER system, the design

and test set scatter diagrams with the classifier boundary and PE

counts, and the corresponding S/N ratio tables, were the outputs de-

sired by the experiment designer. These were used to evaluate the

validity of the postulated hypotheses with regard to semantic memory

processing by Fishier et al. [1], Childers et al. [11,12,17,19,33],

and by Bloom [3].

The SCATTER module requires 46 KB of main memory for execution.

Execution times for typical design/test set applications are, approxi-

mately, 10 minutes for the design and 20 seconds for the test. The

program requires 62 disk blocks to store the files DESIGN and TEST,

and one disk block for the file CLASSIFIER.










3.7 Data Processing/Data Storage Conventions



The DGC 6070 Moving Head Disk storage facility was used to store

the files generated by the CPVER data processing subsystem. These

files are generated in the current system directory at the time of

invocation. The system will automatically search other disk units

for sentence data, but once read, returns to the original directory

for further processing.

All filenames used by the data processing system, with the

exception of the files already mentioned explicitly by name (DESIGN,

TEST, CLASSIFIER, AUTO, INVD, AUTOMATIC, REDRAW, etc.), adhere to

a 10-character naming convention. The first character is always the

file-type designator for the file (described in Section 2.4). Character

positions 2 4 are the subject's initials, character position 5 is the

experiment series designator, position 6 is the session number. Charac-

ter position 7 is either S or R (for Stimulus-aligned or Response-

aligned), position 8 is the class {1-TA, 2-FA, 3-TN, 4-FN} for

covariances or mean vectors, or the letters A, T, D, or M for discrim-

inant vectors {A = UF method using Average across partitions covariance

matrix, T = UF method using the average Toeplitz estimate, D = conven-

tional Discriminant Analysis, and M = Toeplitz Modified Discriminant

Analysis}. Character position 9 is the channel number and character

position 10 is either the partition number or the letter A for average

across partitions. This is summarized in Table 3-3.

As an example, consider the data collected for subject JNL in Ta-

ble 2-3. The data for the semantic replication (R) or episodic (E)










Table 3-3
Filenaming Convention for Covariances,
Mean Vectors and Discriminant Vectors


Filename = 10 alphanumeric characters


{file-type} {III} {E} {N} {sync} {class} {channel} {Pk or A}

Where


{file-type}


{III}


= a one character code for each type of file.
C Covariance, T Toeplitz, U Mean Vector
V Discriminant Vector


= subject's initials (3 letters).


= experiment-series designator (E or S or R).


= session number (1, 2, 3, or 4).


{sync}

{class}


{channel}


= stimulus or response synchronization (S or R).

= the stimulus class for covariances and mean
vectors or the algorithm used for discriminant
vectors. 1 TA, 2 FA, 3 TN, 4 FN,
A UF Averaging, T UF Toeplitz, D Discrim-
inant Analysis, M Modified Discriminant Analysis.


= channel number (1 8).


{P or A} = partition numbers (1 9) or the letter A for
Sthe "average" covariance or "average" Toeplitz
estimator across partitions.










series of experiments reside in disk 301. If we choose to process one

channel, say Cz, using episodic session 1 for design and session 3 for

test, response-lock, the following files will be generated:

Program Number x Size in Bytes Files

COVEST 9 x 404 CJNLE1R161 CJNLE1R169

9 x 404 CJNLE1R261 CJNLE1R269

9 x 404 TJNLE1R161 TJNLE1R169

9 x 404 TJNLE1R261 TJNLE1R269

2 x 404 CJNLE1R16A, CJNLE1R26A

2 x 404 TJNLE1R16A, TJNLE1R26A

9 x 44 UJNLE1R161 UJNLE1R169

9 x 44 UJNLE1R261 UJNLE1R269

DVEST 9 x 84 VJNLE1RA61 VJNLE1RA69

9 x 84 VJNLE1RT61 VJNLE1RT69

9 x 84 VJNLE1RD61 VJNLE1RD69

9 x 84 VJNLE1RM61 VJNLE1RM69

1 x 320 FEATURES

SCATTER 2 x 15552 DESIGN, TEST

1 x 80 CLASSIFIER

For a complete design session (all channels processed), approxi-

mately 1500 disk blocks are required (3/4 MB).















CHAPTER IV
IMPLEMENTATION OF CPVER
DATA COLLECTION/PROCESSING INTERFACE


The implementation details of the EEG data collection interface

to the data processing subsystem, are described in this chapter (from

the system designer's point of view). The chapter is subdivided into

two sections: The "off-line" system implementation and the "real-time"

system implementation.



4.1 "Off-Line" Implementation of CPVER


The term "off-line," when used with the CPVER system, refers to

the data processing task being performed, not in real-time with respect

to the data collection task (which is, of course, always performed

in real-time). In other words, all of the data collection task is

performed prior to the completion of the data processing task, and

there is no biofeedback with respect to the single epoch classification

task. In this mode, any of the experimental sessions are available for

use as either design or test sets (or combinations thereof). The major

disadvantage of this mode is that the knowledge gained through data

processing is not used to influence the data collection, and as a con-

sequence, the homogeneity model assumption becomes less valid. This

must be taken into consideration during the experimental design phase.

For off-line operation, the sequence of events is, typically, as

follows: To begin with, the experiment designer and the biomedical











engineer participate jointly on the development of a realizable para-

digm. A by-product of this step is the generation of the experiment

parameter and specifications table required by the CPVER system. On

the eve of and on the day of the experiment, the laboratory technician

performs the pre-operational checks outlined in Appendix 1. The sub-

jects are trained in the task, and electrodes are attached and tested

for proper operation. Data collection (1st session only) is invoked

by the START.MC pseudo-macro. The system requests the personal infor-

mation (if applicable) obtained from the subject, collects the calibrate

signal, and collects the VER sample prior to sentence presentation.

After the session is completed, the subject is given a brief rest

period while the operator makes an objective evaluation of the session

by invoking any or all of the data averaging and display modules

(VERAVERAGE.MC, VERDISPLAY.MC, SVERDSPL.MC or SVERTYPE.MC). The re-

maining sessions are collected by invoking the COLLECT.MC pseudo-macro

which automatically disables the calibration and VER modules of the

data collection software. The subject is typically given a 15 minute

rest period (outside of the Faraday shielded room) every two sessions.

The data processing is done after all sessions are completed.

The data processing task provides the following outputs to the experi-

ment designer for evaluation: 1) The table of S/N criterion J values

that is produced by the DVEST module, invoked by PROCESS.MC or

FEATURES.MC, preceded by COVARIANCE.MC; 2) the scatter plots for

various channel/feature combinations generated by the SCATTER module,

invoked by PROCESS.MC, CLASSIFY.MC or RESCATTER.MC; 3) a set of VER

"averages" for each channel/session combination. Stimulus-aligned











results, response-aligned results, and choice of feature extraction

algorithm scatter plots have provided the basis for the experimental

results and conclusions reported by Childers et al. [11,12,17,19],

Fishier et al. [1] and Bloom [3].

The ultimate success of a cognitive study is being able to

generalize the conclusions. Inter-subject effects must, necessarily,

be obtained by the use of an "off-line" data processing system. CPVER

must be able to handle such things as designing with one subject and

testing with other subjects, multi-subject design, multi-subject aver-

ages, etc. Versions of the CPVER data processing modules have been

used to generate these types of outputs. In these applications, modules

are invoked independently, "a la carte" so to speak. The pseudo-macro

command invocation provides this capability.



4.2 "Real-Time" Implementation of CPVER


The term "real-time" refers to the interfacing of the data pro-

cessing task to data collection in order to design a classifier,from

a set of labelled epochs, that is made available to the data collection

software for subsequent sessions. The availability of the "on-line"

classification algorithm provides the experimental designer with

plausible alternatives in cognitive research.

In real-time applications, the procedure for the collection of the

data for the first session is identical to the procedure used in off-

line applications described in Section 4.1, using the START.MC pseudo-

macro. The subject is given a brief rest period, long enough for the











operator to obtain a VER average or scan the three-dimensional display,

to make an objective determination of the session. Once the session

is accepted, the system is placed in automatic real-time mode where

the order and invocation of system modules must be performed automati-

cally, with minimum operator intervention.

The following experimental variation is directly implementable

using the CPVER modules: Using the episodic sentence set, design a

classifier with a 72-sentence presentation session and test your

design in real-time with a second session. Provide a classification

error count at the end of the second session. Use the UF feature

extraction algorithm on the stimulus-aligned EEG data.

A possible implementation consists of the following steps:

1) Using any RDOS text editor, create the parameter and

specifications file.

2) Set the AUTO flag and clear the REALTIME, REDRAW, and

INDV flags, using the RDOS command sequence:

DELETE AUTO, REALTIME, INDV, REDRAW

XFER ONE AUTO [set the AUTO flag]

XFER ZERO REALTIME [clear the REALTIME flag]

XFER ZERO INDV [clear the INDV flag]

XFER ZERO REDRAW [clear the REDRAW flag]

3) Collect the data for the first session after creating

the master sentence set. The AUTO flag enables a portion

of code inside the data collection driver module which

creates the file ANSWERS, placing the name of the speci-

fications table, the subject initials, the synchrony











point, the experimental session number and the choice

of feature extraction methodology on a disk file. The

following RDOS command sequence accomplishes this task:

EPIN.SV [creates episodic sentence file]

EXPDRIVER.SV [invokes the data collection
subsystem, creates file ANSWERS]

EEGDSPL.SV [produces 3-D display]

4) If after the visual inspection of the EEG the data is accept-

able, proceed with the automatic classifier design by invoking

the data processing modules with the following RDOS sequence:

COVEST.SV [calculates covariances]

DVEST.SV [calculates discriminant vectors
and creates file FEATURES]

SCATTER.SV [designs classifier and stores it
in file CLASSIFIER]

5) Invoke the data collection subsystem for on-line classification

by setting the REALTIME flag and invoke the experiment driver

module with the following RDOS command sequence:

DELETE REALTIME AUTO

XFER ONE REALTIME

XFER ZERO AUTO

EXPDRIVER.SV

The default "on-line" module for the experiment driver (module

A9 of Figure 2-1) tallies the number of misclassification errors and











reports the count and probability of error automatically, at the

end of the session, on the operator's console. Module A9 can easily

be modified for the display of a feedback message (to the subject)

based on the result of the classification algorithm (Module A7 and

A8 currently provide feedback messages when the subject eyeblinks or

does not respond in adequate time etc. A few (less than 10) FORTRAN

statements in modules A9, A7 and A8 can modify the existing logic for

this purpose). All of these RDOS commands have been stored in the

ONLINEDEMO.MC pseudo-macro, which uses the DEMOEXPER.SV module instead

of the EXPDRIVER.SV module. DEMOEXPER.SV is identical to EXPDRIVER.SV

except it replaces the single statement that writes the collected

EEG data onto disk with the dual statement that reads the EEG data

from disk. In other words, DEMOEXPER.SV performs a real-time simulation

of a previously collected real experimental session, except there will

not be any rejections due to artifacts (these were rejected on-line

when the session is actually collected).

The ONLINEDEMO.MC pseudo-macro is a very valuable tool in the

evaluation of the performance of the data processing software. Newer

revisions of these modules can now be tested "on-line." A revision

that may be optimal in terms of a given performance parameter but that

demands long execution times (over a few minutes) would seriously

degrade biofeedback experiments. Using ONLINEDEMO.MC, this application

took 22 minutes (total elapsed time) to collect and test two episodic

experimental sessions.















CHAPTER V
APPLICATIONS OF CPVER
A STUDY OF COGNITIVE PROCESSES



The motivation for the development of a CPVER system is described

in Childers et al. [17]. In this report, the authors give specific

details of the rationale for the experiments that were to be conducted

using such a system. In section 5.1 a brief, relevant review of the

experimental design is presented for convenience. Section 5.2 gives

a summary of results.



5.1 Experimental Design


Recent studies have suggested that components of ERPs may discrimi-

nate semantic or linguistic "meaning" [1,6,47,48]. In the UF experi-

ments, cortical potentials were to be recorded while subjects decided

if simple propositions such as "A bass is a fish" were true or false.

This "sentence verification" task has been used extensively in behavior-

al studies of semantic memory, since it requires that information

relating the two terms of the sentence be retrieved from long-term

memory, and evaluated with respect to sentence structure. The major

purpose of the experiments was to determine if true and false state-

ments could be discriminated on the basis of the subject's single

ERPs.

Verification of the truth or falsity of even simple propositions

often seems to require effort (consider, for example, "A rose is not











a vegetable"). For one set of experiments, sentence subjects (S-terms)

were 18 concrete nouns (e.g., trout), while sentence objects (0-terms)

were either the immediate (e.g., fish) or superordinate (e.g. animal)

categories of those nouns. Verbs (V-terms) could be either affirmative

("is") or negative ("is not"). The use of both affirmative and negative

sentences, and the inclusion of two possible 0-terms, was intended to

make it less likely that subjects would read the S and V-terms and

then try to anticipate particular words which would make the sentence

true or false. This may reduce the task from semantic verification to

lexical matching. Use of negative sentences also controlled for the

presence of associations between S and O-terms; that is, the class of

true and false sentences each included associated (Robin is/is not-

bird) and unassociated (Robin -is/is not- truck) S-0 pairings.

In the resulting set of 144 sentences, true and false statements

were equally likely, as were affirmative and negative statements. Use

of an equal number of false and true sentences, repetition of terms

within blocks of sentences, and repetition of sentences across blocks,

were intended to keep the false sentences from being any less familiar

or more anomalous than the true statements. Any such differences

would be confounded with the true-false distinction.

The semantic experiment sentence materials are a hierarchy of set-

inclusion statements, a portion of which is outlined as follows:

(Diagram on following page)





80



Level 1 Level 2 Level 3

AI2
B2 1


2 2

1 I



2 Q2


An example from the actual stimuli would be as follows: X =

animal; P1 = bird; P2 = insect; Al = robin; Y = plant; Q1 = flower;

Q2 = tree; etc. The complete hierachy is given in Figure 5-1.

The use of both positive and negative propositions gives four

basic types of statements, e.g.,

True Affirmative (TA): A hammer is a tool.

False Affirmative (FA): A hotel is a tool.

True Negative (TN): A hotel is not a tool.

False Negative (FN): A hammer is not a tool.

The number of TN and FA sentences was restricted to equal the

number of possible sentences in the other classes (i.e. 36 each).

This gives a total of 144 sentences. From a subject's perspective

true and false statements are equiprobable. This serves to disconfound

the response to validity from any probability effects that might occur

with less likely false sentences. For presentation purposes, the

sentences are randomized and divided into two blocks (sessions) of

72 trials. In the semantic experiments, each subject participated in

four distinctly randomized 72-trial sessions. In the replication of










BASS

TROUT

ANT-----
E INSECT ANIMAL
BEE

ROBIN-

SPARROW

OAK----
OTREE
PINE

ROSE
DR FLOWER PLANT
DAISY00

PEA-------
EA VEGETABLE
CARROT

HAMMER
TOOL
SAW

CAR

TRUCK ---= VEHICLE OBJECT

HOTEL
HOUTE -BUILDING
HOUSEe 51 S e


Figure 5-1 Sentence Hierarchy











the semantic experiments, each subject participated in two sessions of

144 trials (in the early experiments, disk storage was limited to the

size of a Data General 4047B disk pack).

In the episodic experiments, 72 sentences involving facts about

the individual subject, were presented in four randomized sessions to

each subject. The sentences consisted of two types, TA and FA, each

type contains 36 statements (the negative statements were eliminated).

These experiments were fundamentally identical to the semantic experi-

ments (the only change was the sentence) described above. The episodic

sentence set is depicted in Figure 5-2.

The rationale for these idiosyncratic experiments was basically

that the task should be easier and faster (this proved to be the case),

and that validity could be tested perhaps more reliably, since the

statements involve terms that are not previously associated. Thus,

the responses to the TA statements should be more easily separated

from the responses to the FA statements because of their particular

idiosyncratic nature and more "obvious" validity.

More than 10 subjects have participated in these experiments

(see Table 2-3)with the use of the CPVER system. Experimental results

have been described by Fishler et al. [1] and Childers et al. [11,

12,17,19,33].

The measurement vector was chosen as a 720 msec interval of the

EEG. The interval may start with the onset of the last sentence seg-

ment that the subject reads or it may be shifted by a specified

number of samples from the 0-term (stimulus-locked). Synchronization

may also be chosen with respect to the response pulse onset, with or
















My first

My last

My father's

My mother's

My major

My class

My hair

My hair

My preferred

My sex

I like

My age

I ride

I work

I was

I was

I live

I live


name is

name is

name is

name is

is

is

is

color is

hand is

is

to play

is over

a

as a

born in

born in

in

in


19

20

S 21

S 22

23

24

25

S 26

27

28

29

30

31

S 32

S 33

S 34

___ 35

36


My parents

My parents

My father

My mother

My eyes

I go

My ancestors

My biggest

My favorite

My worst

I want

My favorite

My shoe

My favorite

My middle

My favorite

My favorite

My residence


lived) in

lived) in

is a

is a

are

to bed

came from

meal is

food is

food is

to be a

color is

size is

music is

name is

author is

actor is

is


Personal Information Data Set


Figure 5-2










without shifting (response-lock). Since there were 8 msec between

samples, this gave a 90 component measurement vector, e.g., N = 90 for

each EEG data channel. This measurement vector dimension was determined

empirically by considering several factors. First, initially it was

desired to include the P300 component in the ERP (i.e., the positive

300 msec peak). Second, it seemed wise to place some upper limit on

the dimensionality of the measurement vector. This observation window

can be shifted at will, therefore, this does not impose any limitation

on exploratory data processing. Thirdly, this 720 msec measurement

interval closely approximates the overall spread of the subject's

responses [11].

The data was divided into two equal sets, a design set of M

samples/class and a test set of M samples/class. Usually D = T = 36

for the experiments conducted (subjects on occasion did not respond

early enough in the measurement interval to allow the full 720 msec

observation window for response-aligned data processing) using CPVER.

The design set is used to determine both the features and the classifier.

To reduce the dimensionality of the observation vector as compared to

the number of samples per class, the measurement vector was partitioned

into K segments of length L (N = KL) such that the ratio M/L > 3. The

partition size was chosen as L = 10, yielding a reduced nine-element

observation set. The size 10 was selected after conducting experiments

on the correlation between successive samples, etc.

One or two discriminant vectors (features) were calculated for

each partition of dimension 10. A criterion value (a type of signal-

to-noise ratio) was computed for each feature. The two features with

the largest S/N ratios were usually chosen for the representation.











The design set was projected onto the selected two-feature space

and a linear classification rule was derived. The PE was determined

empirically for the design set by counting the number of known trials

that fall on the wrong side of the decision boundary. The test set

PE is also determined empirically. This design and test set procedure

is known as the Hold-Out Method for estimating the PE, and gives

pessimistic results, but it does not require much computation.

The desired experimental results were: 1) ERP averages that

demonstrate "gross" effects, such as the evoked response following

each sentence segment presentation, 2) tables of S/N ratios for

stimulus-aligned and response-aligned data, and 3) scatter diagrams

for each design/test set pair with the resulting PE tabulation.

These outputs provided the basis for paradigm evaluation and associated

conclusions.



5.2 Summary of CPVER Data Processing Results


In the following pages, sample 3-D displays, ERP averages, S/N

ratio tables, and scatter diagrams are presented for the four subjects

whose data was "accepted" for processing (see Table 2-3): Mainly,

RCP, JNL, CJB, and MAD. Subjects RCP and JNL participated in both

the episodic experiment and the semantic (replication) experiment.

Subject CJB was asked to reverse the direction of his switch-

activated response (true = left in sessions 1 and 4, true = right in

sessions 2 and 3) movement. An EMG was collected for a preliminary

study designed by Bloom [3].






86




No attempt is made to assess the implicability of these results

to the cognitive processes of the human brain. The goal of this re-

search was to provide a framework for the conduction of experiments

in cognitive psychology studies, e.g., the evaluation of these results

is beyond the scope of this work.






























































Figure 5-3 Subject RCP, Series R, Session 1, S-Lock
















Table 5-1

Feature Vector Criterion












S/H Ratios or Criterion J, Non-Toeplitz.


Page 1


Subj:RCP, Ser:R, Sessal, Chn:6 CZ, S-LOCK C 10), 36 Samp/Cl, 10/25/81

Pk k= 1 2 3 4 5 6 7 8 9
Mk 3 4 4 3 5 3 5 3 3
Gamma 0.482 0.350 0.579 0.791 0.575 0.524 0.897 0.798 0.366

UF -fl 0.400 0.248 0.458 0.699 0.425 0.440 0.728 0.706 0.290
* ---------------------------------------.
DA -fi 0.715 0.463 0.660 1.043 0.521 0.599 1.215 0.877 0.563
DA -f2 0.213 0.143 0.379 0.505 0.385 0.113 0.560 0.512 0.341


















Table 5-2

Feature Vector Criterion












S/N Ratio5 or Criterion J, Toeplitz. Page 1

------------------------------------------------------------------------
Subj:RCP, Ser:R, Sess:1, Chn:6 CZ, S-LOCK C 10), 36 Samp/Cl, 10/25/81
------------------------------------------------------------------------
Pk k= 1 2 3 4 5 6 7 8 9
Mk 2 4 3 3 4 3 5 3 3
Gamma 0.390 0.314 0.611 0.747 0.521 0.455 0.841 0.769 0.498
------------------------------------------------------------------------
UF -fl 0.347 0.216 0.524 0.657 0.404 0.374 0.674 0.678 0.414
------------------------------------------------------------------------
DA -fi 0.594 0.400 0.789 0.837 0.736 0.716 1.196 1.011 0.632
DA -f2 0.218 0.082 0.242 0.463 0.333 0.042 0.439 0.493 0.278
------------------------------------------------------------------------















U H -S 83K + S 7 ANGLE 137


T

FF
FF F FF
F F
F
F F


T


T

T TF T
F T
T
T T

\FT T



T


DESIGN SET, SUBJ:RCP, SER:R, SESSION:1, CHH:6 CZ S-LOCK, SHIFT: 1e, FEAT(8,7), PE=18/72(11-T, 7-F!


Y = -1 83X + 591 75


nuGi r -35


\F T T
F T

F FF
T FF
F FT F T T
FTP FT
F
T TV' T T

FF T


T F I
T T
FF F




T



F
T~~~~~~~T ~ =flfl la~pr ~LUL.~ C~nirU IY. I~ Aln, CuLHI r(J ,rr -


ES SE :II S 0 M M H 0 &6 C S-O, 4 SHIF' : I, FLT87,P=27(oT2 -
Figure 5-4 Scatter plots


UF METHOD (090 COV


UF MEHO CV COY Y- 3 + Cc 7v nC


UF METHOD (090 COY)


Y = -1 93% + 591 75


ANGLE: 135


T













UF METHOD (AVG COY). Y = -1.9X + 686.38 ANGLE: 133

T T


T
F T T
T T
F T TT T T T


T
ST T T T
T F TT T
SF T T

TF
F F" F

FF F T T F
F "
F T


T FF T




























T T
TT F FF F 1 #
TT T

T





FT
F F F
F IF


F F f F

T T T F






TEST SET, SUBJIRCP, SER:R, SESSION:2, CHN:6 CZ S-LOCK, SHIFT: 18, FEAT(7,4), PE=29/72(11-T,18-F
Figure 5-5 Scatter plots













UP METHOD (AVG CO


FF T T
FF

T F
T
F F FF T
FT
SFF T
F F
T FT F
F F
F
F T


T F T

T T

T FT F T T


F F
r FT T
T
ST


DESIGN SET, SUBJ:RCP, SER:R, SESSION;1, CHH:6 CZ S-LOCK, SHIFT: 10, FEAT(7,3), PE=19/72(11-T, 8-FJ
















UF METHOD (RVG COV). Y = -8.69X + 473.29 ANGLE: 146


TEST SET, SUBJ:RCP, SER:R, SESSION:2, CHN:6 Ct S-LOCK, SHIFT: 18, FERT(7,3), PE=33/72(16-T,17-F
Figure 5-4 Scatter plots


T




T


FT r FT
T
T T 1
T T
FTT F

T TTf T
FT T


T F


F F


I I F = -9.69- + 473.i29 nI w I% T


V ~ L~Y r


nuC r A




University of Florida Home Page
© 2004 - 2010 University of Florida George A. Smathers Libraries.
All rights reserved.

Acceptable Use, Copyright, and Disclaimer Statement
Last updated October 10, 2010 - - mvs