VISUALIZATION OF FUNCTIONAL MAGNETIC RESONANCE IMAGES
THROUGH SELF-ORGANIZING MAPS
Didem Gokcay and LiMin Fu
Department of Computer and Information Sciences
University of Florida
didem @cis.ufl.edu, email@example.com
Technical Report: 96-011
Functional activations in the human brain are being studied through various imaging tools, and FMRI is a new tool with
growing popularity. Comparison and interpretation of FMRI image sets collected from different people, through different
imaging platforms, is a difficult problem. In this paper, transformation of FMRI images to an abstract domain, SOM, is
studied as a preliminary step for performing inter-subject image registration on abstract parameters in the future.
Functional magnetic resonance imaging (FMRI) is a promising new neuroimaging tool for understanding the relationships
among brain structure, function and -.iil. ., _.- [;]. FMRI indirectly detects regional signal intensity changes in the brain in
response to miscellaneous functional tasks that affect the motor, visual, auditory ... cortices, by measuring the Cerebral
Blood Flow (CBF) through the change in concentration of oxygenated blood. For detailed information on the data
acquisition process, the readers may refer to .
i,,,. registration" is a common problem for all neuroimaging methods. In image registration, various image datasets
are related, by precisely projecting equivalent areas on top of each other. Considering the differences in the shape, size and
functionality of people's brain, and also considering various different platforms such as MRI, PET, and SPECT from
which the datasets may be obtained, image registration is a difficult problem to handle. When images in two different
modalities such as MRI, and PET are related, the term "multi-modal registration" is used. Two other terms, "intra-
subject registration and "inter-subject registration are used to indicate whether the related image datasets belong to
the same subject or different subjects. Compared to other imaging methods, intra-subject registration is relatively easier in
FMRI, because the anatomic image of the subject is collected during the session in which the functional images are
collected, and these two sets can be superimposed directly. However, inter-subject registration in FMRI is still
problematic, and very error-prone.
A widely accepted approach for inter-subject registration is stereotactic mapping, in which the anatomic coordinates
generated on the subject's brain are projected onto the anatomic coordinates in a brain atlas. In order to relate function to
anatomy, the stereotactic map of the subject's brain and a predetermined standard brain are compared. The comparison is
accomplished by resizing the individual's brain image to fit the dimensions of the standard brain. The correctness of such
a mapping is of course limited to the similarity in shape between the subject's brain and the standard brain, once the size
differences are eliminated . Unfortunately in the current inter-subject registration methods, the basis of cerebral
structure and function relationships can not be captured fully, and adaptation of the standard atlas to the patient's anatomy
is not perfect .
In this study, we propose an abstract platform as an alternative modality for visualizing functional brain activations. The
transformation used here maps FMRI images onto a NxN self-organizing map (SOM), by using features collected from the
given dataset. This study can be viewed as a preliminary step for attempting to perform inter-subject registration on an
abstract model in the future.
2. Background on time-course data processing in FMRI
Before explaining the mapping from FMRI images to SOM, it may be useful to highlight the basic data processing
procedure in FMRI. During data acquisition, subjects are asked to repeat a functional task periodically, interleaved with
rest periods. The time slot in which the subject performs the functional task is referred as the active state, or ON state,
whereas the time slot in which the subject rests is referred as the rest state or OFF state. Hence an FMRI dataset consists of
a sequence of brain images collected during a series of alternating ON and OFF periods. Usually, during each ON/OFF
state, at least 4 images are collected, visualizing the brain activity in at least 2 or more slices of the subject's brain. Slice
thickness, slice location, time spent in scanning each slice and interscan interval time are some of the important
parameters that affect FMRI data acquisition. In figure 1 below, functional task activation protocol is summarized with a
OFF ON OFF ON OFF ON OFF ON OFF ON OFF ON
Figure 1. An example functional task activation protocol in FMRI (6 repetitions, n images for each state)
There are three basic methods used in the interpretation of FMRI image sets: Fourier transform, cross correlation and
subtraction. In all these methods, the underlying principle is the following: The areas of the brain that carry out the
functional task will show a response which is similar to the activation waveform. As an example, consider figure 2 below,
in which the activity at 2 different points, j and k are illustrated. In figure 2a, the location of points j and k are given on an
anatomic image, point j falling into the region of functional activity and point k being a randomly selected point not
participating in the functional task. Figure 2b represents pixel j in the image set, where the graph shows the intensity value
of pixel j through all images in the image set. Note the similarity of the graphs in figure 2b to the box-car waveform.
a. Anatomic image b. Activity at pixel j c. Activity at pixel k
Figure 2. Change of intensities around the neighborhood of 2 different points in a set of FMRI images
Given these characteristics, image analysis procedures are straightforward, as described in detail by . When Fourier
analysis method is used, time-course intensity graphs shown in figure 2 are transformed into frequency domain. Then a
new image is drawn which represents the whole set of FMRI images collected through time, in such a way that, in this
new image, only pixels that respond at a particular frequency -the frequency of the box-car waveform- are highlighted,
whereas other pixels stay black. When cross-correlation method is used, for each pixel, the associated time-course graph
is correlated with the box-car activation waveform. Then a new image, representing the whole set of FMRI images is
generated, in which only the pixels that show a correlation higher than a specific threshold are highlighted, while all other
pixels remain black. Finally, in the subtraction method, intensity difference between averaged ON and OFF states are
calculated for each pixel, and a new image is generated, by plotting each pixel with a gray level corresponding to the
difference in intensity. In figure 3, sample cross-correlation and subtraction images are shown for a simple motor task,
which involves left-hand finger tapping during the ON states and resting during the OFF states. In figure 3a, the cross-
correlation image is superimposed on the anatomic image, and the dark areas show pixels with correlation exceeding
0.55. In figure 3b, the subtraction image is shown, in which, high intensity levels correspond to large differences between
averaged ON and OFF states. Both in figure 3a and 3b, the active area is indicated with boxes.
a. Cross-correlation, threshold=0.55 b. Subtraction
Figure 3. Thresholded cross-correlation and subtraction images for left-hang finger tapping task
(subject's right -active area- is on reader's left)
There are some technical difficulties however, limiting the success of FMRI image analysis. First of all, the signal to noise
ratio is poor. The activity in response to functional task generates approximately 2-5% difference in intensity. Signal to
noise ratio is reported to get better as the power of the MRI machine increases from 1.5T to 4T. Secondly, there are vessel
artifacts in the image, due to blood flow from/to main blood vessels. And finally, imaging takes time and it is hard to
stabilize the head during a long series of scans. Head movement causes a shift in the image pixels, which in turn
introduces errors to the time-course intensity graphs associated with the pixels. For more information on FMRI data
processing, the readers may refer to .
3. Visualization of FMRI datasets through SOM
Given the characteristics and associated technical difficulties in the FMRI datasets, we believe that representing the FMRI
data in an abstract domain is an alternative to Fourier transform, cross-correlation or subtraction images. As an abstract
platform, SOM is chosen, since it preserves the topological characteristics in the data. The SOM layout and learning
algorithm is discussed in detail in , and can be summarized briefly as follows:
A 2-D SOM consists of a grid of cells laid out in 2-D space. Associated with each cell is a weight vector, carrying features
of the entity the SOM represents. Initially, the weight vector of all cells are assigned randomly. Then the SOM is trained
iteratively to represent samples from the input space, by first choosing a sample from the input space, then picking a cell j
in the SOM which contains the closest weight vector to the given sample, and adapting the weight vectors of cell j and
other cells in its neighborhood to represent the sample better. During weight adaptation, weight vectors of the cells in the
neighborhood of cell j are adjusted by a small amount proportional to the Euclidian distance between the weight vector of
j and the presented sample. Initially, neighborhood covers all cells in the grid, but as the algorithm proceeds, the
neighborhood size is reduced, the final neighborhood consisting of only the point j. When the algorithm terminates, the
resulting SOM not only keeps the distribution of samples from the input space, but also preserves their topological
In our study, we explored 2 different SOM models for representing FMRI datasets, both models having a 50x50 grid
layout in 2-D. The only difference between the models are in the feature vector representations, as explained in the
Model I-SOM with cross-correlation and standard deviation features:
In this model, each pixel j in the image set is represented by a 4 dimensional feature vector:
[xj, yj, stdj, corrj]
where x and y are the coordinates of pixel j, std is the standard deviation of the intensity of pixel j through the set of all
images, and corr is a value changing between [-1,1] showing the correlation of the intensity waveform in pixel j with the
box-car activation waveform. Typically, each image in the dataset are 256x256, so there are 256*256 sample vectors in the
input space. If there are a total of 100 images in the dataset, then the standard deviation and correlation values for each
pixel are calculated on 100 values corresponding to the same location at all images.
Model II-SOM with task-state-average features:
In this model, each pixel j in the image set is represented by a k dimensional feature vector:
[xj, yj, avgi(OFF), avgi(ON), avg2(OFF), avg2(OFF),...... avgm(ON) avgm(OFF)]
where x and y are the coordinates of pixel j, and avgi (ON), and avgi (OFF) are the average intensity values of pixel j in the
images that are collected during the ith ON and OFF states respectively. If there are m repetitions of ON/OFF cycles, then k
will be equal to 2 + (2*m). Considering p images are collected during each task-state, each average value in the feature
vector represents intensity values of pixel j over p images.
The SOM algorithm used in this study is a standard SOM algorithm, in which the Euclidian distance metric is used and
weight updates in the neighborhood are flat. However, following improvements can be incorporated in the future, to better
represent the samples, and reduce training time:
SUsing a Gaussian weight adaptation formula in the neighborhood, to increase the sensitivity of the center of activation.
SUsing covariance of pixel j with its neighbors as a multiplier term in the weight adaptation formula, as a means to capture
the cooperation of adjacent pixels in response to task-activation.
. Calculating hit-rate for each cell j in the SOM to show how many times cell j becomes the best-matching unit to the
presented input vector during training, for interpreting the behavior of SOM in representing the input distribution.
Given these vector representations, one last issue that remains to be explained is how to visualize the SOM after training is
complete. Each cell in the SOM carries either a 4 dimensional or k dimensional vector, depending on the model that is
used. In order to visualize the feature vectors associated with each cell, we use a variant of the algorithm presented in .
A 2-D graph is drawn to visualize the SOM as follows: For each cell j in the SOM, plot a node in the graph on locations x,
and y,. Calculate the maximum Euclidian distance, dist, between unit j's and its immediate neighbors' weight vectors,
excluding the x and y components in the vectors. (Immediate neighbors are defined using 8-connectivity). Plot a circle on
locations x, and yj having area proportional to dist1. After plotting circles for each unit j in the SOM, connect the
neighboring units with edges. Intuitively, if the SOM reflects the features of the FMRI dataset closely, we expect to see
large circles at the cells that represent the activation area. Because, features of the cells in the active area will vary highly
from the features of the adjacent cells that represent locations unrelated to the task. Since the SOM is 50x50, there will be
50*50 nodes in the resulting 2-D graph.
4. Results on finger-tapping experiments
The SOM representation suggested in this study is tested on finger tapping experiments using a 1.5T MRI machine with
echo planar imaging. Three basic functional protocols under which data collection is made are: 1) Left-hand finger tap/
rest; 2) Right-hand finger-tap/ rest; 3) Left-hand finger-tap/Right-hand finger-tap. The resulting FMRI datasets from these
tasks are displayed and analyzed through AFNI image processing software, available from the WCM, just to verify the
validity of the magnitude and location of generated response. Once the data of a specific subject is verified, a 50x50 SOM
is trained to represent his/her FMRI dataset, as explained in section 3. In figure 4 below, Model-I and Model-II SOMs
corresponding to the same image set in figure 3 are visualized. In figure 4b, the edges that connect the neighbors are not
shown for clarity. Once again, the functional task in this set is: left-hand finger tap/rest. As seen in the figure, both SOMs
are successful in pinpointing the active area, while displaying almost uniform distribution over all unrelated areas. The
activity around location (125,225) is due to vessel artifacts. The features carried by the Model-I SOM are cross-correlation
and standard deviation, hence we expect this type of SOM to be visually close to the cross-correlation image. On the other
hand, the features represented in the Model-II SOM are task-state averages, so it is expected to have a visual similarity to
the subtraction image. Indeed, in most SOMs trained from the FMRI datasets, the visual representations turned out to be
close to the above expectations.
250 250 .. ..
l'- ',,'I ;' j i1 o 00 .. Bl' ,....-...
''lI : i I ,ii
0 50 100 150 200 0 50 100 150 200
x-dim of neuron i x-dim of neuron i
a. Model-I SOM b. Model-1 SOM
Figure 4. Visualization of SOMs for left-hand finger tapping task (subject's right is on reader's left)
Inter-subject registration is important in clinical situations, to relate a subject's response with the response of a set of
normal patients or to compare a patient's data before and after treatment. This requires a standard reference image, and in
order to relate the image of a subject to the standard image, stereotactic or atlas based approaches are used. Stereotactic
approaches hardly allow comparison and interpretation of functional images across subjects within acceptable error-limits,
because they are based on only physical features and do not take the functional features into account.
5. Conclusion.; .-:' 2. ', ..:': .". '
noralpaiens r o cmprea ptints dtabeorean atertratent Tisreqirs stndrdrefreceimaean i
In this study, we propose yet another modality for representing FMRI datasets: an abstract platform. If implemented in
future, the suggested method may provide means for interpreting FMRI activations across subjects in a better way. In
order to use the abstract platform in inter-subject registration, all image sets must first be transformed into the abstract
domain, and then compared using abstract features.
Some important advantages of using an abstract platform for interpreting functional datasets can be summarized as
. The images are mapped from 256x256 to lesser dimensions
. The abstract domain is independent of the platform in which the images are obtained.
. Transformation from datasets to SOM is based not only on physical features but also on functional features.
STo some degree, head movement artifacts are reduced, since adjacent pixels on the image will be mapped onto the same
cell in SOM.
 P.A. Bandettini, A. Jesmanowicz, E.C.Wong, J.S.Hyde, "Processing Strategies for Time-Course Data Sets in
Functional MRI of the Human Brain", Magnetic Resonance in Medicine, Vol. 30, No. 2, pp.161-173, 1993
 T. Kohonen, "Self-Organizing Maps", Springer Verlag, 1995
 M.A. Kraaijveld, J. Mao, A.K. Jain, "A Nonlinear Projection Method Based on Kohonen's Topology Preserving
Maps", IEEE Transactions on Neural Networks, Vol. 6, No. 3, pp. 548-559, 1995
 S.E. Nadeau, B. Crosson, "A Guide to the Functional Imaging of Cognitive Processes", Neuropsychiatry,
Neuropsychology, and Behavioral Neurology, Vol. 8, No. 3, pp. 143-162, 1995
 S.M. Rao, J.R. Binder, et. al., "Somatotopic Mapping of the Human Primary Motor Cortex with Functional Magnetic
Resonance Imaging", Neurology, Vol. 45, No.5, pp919-924, 1995
 J.A. Sanders, W.W. Orrison, "Functional Magnetic Resonance Imaging", Chapter 7, Functional Brain Imaging,
W.W. Orrison et.al., Mosby Year Book Inc., 1995
 L. Thurfjell, C. Bohm, E. Bengtsson, "CBA: An Atlas-based Software Tool Used to Facilitate the Interpretation of
Neuroimaging Data", Computer Methods and Programs in Biomedicine, Vol.47, No. 1, pp.51-71, 1995
 A.W. Toga, P.K. Banerjee, "Registration Revisited", Journal of Neuroscience Methods, Vol. 48, No. 1-2,pp. 1-13,