EFFECTS OF FIELD INDEPENDENCE, COMPUTER EXPERIENCE,
AND DYNAMIC PICTORIAL ONLINE HELP PRESENTATIONS ON
LEARNING APPLICATION FUNCTION IN A GRAPHICAL USER INTERFACE
JOHN GORDON TYLER
A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL
OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT
OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
UNIVERSITY OF FLORIDA
John Gordon Tyler
All rights reserved.
To my daughter Jessica Ann, for her sweet and joyful love.
May her desire to learn always brightly shine and guide her.
This research project was the culmination of more than a decade of graduate stud-
ies and professional endeavors. During this time I received kind and expert guidance
from many people, without whose gifts of time and talent this goal would never have been
realized. First, I express my deep appreciation to Dr. Lee Mullally for serving as the chair
of my supervisory committee, and for his guidance throughout this research project. His
reassuring support and constructive criticism empowered me to achieve goals beyond my
I also wish to thank the other graduate faculty on my supervisory committee, Drs.
Roy Bolduc, Doug Dankel, and Jeff Hurt, who provided insightful recommendations and
thoughtful questioning during each phase of this research project. They were always
willing to contribute their expertise to help refine the methods and materials applied in
this study. I also want to thank Dr. James Algina for his assistance with the development
and review of the statistical analyses for this study.
I want to express my gratitude to the management team at the IBM Personal Sys-
tems Programming Center in Boca Raton, Flordia. In particular I want to thank Mark
Tempelmeyer, who actively promoted my return to the University of Florida. His con-
stant optimism and wry sense of humor helped to inspire and motivate me during my long
sabbatical from the IBM lab. In addition, I want to thank Dr. Frances Palacio, Oscar
Fleckner, Marty Voss, Janis Walkow, and Tim Shortley for their support during my leave.
Thanks also go to my friends and colleagues at IBM who contributed to this
project: Jack Reese, Jeff Baker, Ray Voigt, Larry Kyralla, Doug Bloch, Larry Mallett,
Tom Greaves, Chris Freeman, Kerry Ortega, Carol Righi, Jim Lewis, and many others
who supported or participated in this study. I especially want to thank Dr. Bob Kamper,
for his thoughtful reviews of this manuscript and his spirited e-mail during this study.
I also thank professors Grandon Gill, Paul Hart, and Randy Coyner of the College
of Business at Florida Atlantic University for consulting on this research and for provid-
ing access to students in their classes. Their support was essential and greatly appreciated.
Finally, I wish to express humble gratitude to my family, who continually gave
encouragement and supported me in every conceivable manner. I want to thank my
father, Dr. Leslie J. Tyler, whose career served as a wonderful example and who instilled
in me the desire to become a research professional. I also thank my mother, Pat, who
reminded me at times to "Stop and smell the roses", and to not be too deeply immersed in
books and computers. Above everyone else, I'm deeply thankful for my wife, Jo, whose
devotion and kindness overcame the many hardships. She gave me her strength and hope
whenever my own were flagging.
TABLE OF CONTENTS
ACKNOW LEDGMENTS............................... ...............................................iv
AHQrTD T Arr
Statement of the Problem
Need for the Study...........
epfinitinn nf Terms
H y potheses .......................................................................................................
Assumptions and Limitations ................................................ .......... ..... 10
2 REVIEW OF LITERATURE ........................ ... .......................... 14
O verv iew ............................................................... .....................................14
Instructional Message Design for Visual Learning ................................... 14
Human-Computer Interface Design ................................................ ........16
Cognitive Style Effects................................ ................ ..........................18
Comparative Expertise Effects on Mental Models.................................... 21
Measuring Computer Expertise ............................... ................. 23
3 METHODOLOGY ....................... ...................................28
Experimental Design.................................................... ......................... 29
Population and Sample............................................... ........................... 37
Instructional Treatment ................................................. ........................ 46
Data Collection ........................................................... ...........................55
4 RESULTS AND ANALYSIS..................... ............................ 58
R esu lts ................................................................................................... .59
5 DISCUSSION AND RECOMMENDATIONS........................................76
Discussion of Findings ..................................... .............. .......................76
Recommendations for Future Research ................................... ........... 82
Sum m ary .................................................................................................. 90
A OS/2 TRAINING SIGN-UP FORM....................... ............................92
B COMPUTER EXPERIENCE AND COMPETENCE SURVEY ................94
C COMPUTER TRAINING LESSONS, PRETEST AND POSTTEST......100
D HELP TRACKING LOG FILE EXAMPLE ............................................115
E OBSERVER LOG FILE EXAMPLE..........................................................118
REFEREN CES............................................. ................................................ 125
BIOGRAPHICAL SKETCH............................. ..... ........................129
Abstract of Dissertation Presented to the Graduate School
of the University of Florida in Partial Fulfillment of the
Requirements for the Degree of Doctor of Philosophy
EFFECTS OF FIELD INDEPENDENCE, COMPUTER EXPERIENCE,
AND DYNAMIC PICTORIAL ONLINE HELP PRESENTATIONS ON
LEARNING APPLICATION FUNCTION IN A GRAPHICAL USER INTERFACE
John Gordon Tyler
Chairman: Lee J. Mullally
Major Department: Instruction and Curriculum
The graphical user interfaces of modem computer applications use dynamic pic-
torial elements to represent application functions. Online help messages assist users
learning to operate those functions. Online help, however, rarely incorporates pictorial
elements. Instructional message design and visual learning theories suggest that pictori-
ally encoded messages should result in greater learning than purely verbal help messages.
H. A. Witkin's theory of cognitive style suggests that learners with greater field indepen-
dence will perform better in complex visual environments, such as those found in
graphical user interfaces. Some researchers suggest that prior computer experience is the
most important determinant of performance in an unfamiliar human-computer interface.
This study was conducted to examine the effects of individuals' level of field
independence and prior computer experience on application task performance in a
graphical user interface. This study also investigated Aptitude x Treatment interaction
effects between computer users' cognitive style (field dependence-independence) or their
level of computer experience and the use of dynamic pictorial message elements in online
From a population of university business college students, 38 volunteer subjects
were randomly assigned to one of two online help treatments: text-only help or text-
with-motion-video help. The two help treatments were identical except for the addition
of digital motion video segments. Field independence was measured using the Group
Embedded Figures Test. Computer experience was assessed using the Computer Experi-
ence and Competence Survey. Time in help was measured as the total time online help
messages were displayed during training. These variables were applied as covariates in a
multiple covariance analysis. Performance on computer application tasks was the
dependent variable. Subjects completed an individualized computer-based training
regimen including a pretest, twelve lessons covering system and spreadsheet application
functions, and a posttest.
The results showed that subjects with higher field independence had significantly
higher task performance scores than subjects with lower field independence. Also, sub-
jects with more computer experience had significantly higher performance scores than
those with less prior experience. No significant differences in performance on application
tasks resulted from the addition of dynamic pictorial message elements in online help.
The results of this study may contribute to the design of adaptive human-computer inter-
faces and online help systems.
The goal of this study was to investigate the effects of aptitude differences
among individual computer users and the effects of dynamic pictorial message presen-
tations in computer-based instruction (e.g., online help). Two distinct characteristics of
computer users, cognitive style (measured as field independence) and computer experi-
ence, were examined to determine whether relationships exist between these
characteristics, the presence of pictorial message content in online help, and perfor-
mance on computer tasks. Specific research questions were raised to examine the
effects of computer users' cognitive styles and levels of computer experience on their
performance on application tasks in a graphical user interface (GUI). In addition, this
study was designed to determine whether Aptitude x Treatment interaction (ATI) effects
occurred between either cognitive style or prior computer experience and the presence
of dynamic pictorial message elements in online help.
Statement of the Problem
The human-computer interface (HCI) provides an environment for the interac-
tion between a human user and the dynamic operations of a computer. A graphical user
interface, one class of HCI, uses a variety of pictorial message elements in addition to
verbal elements (text) to represent computer states and functions (Horton, 1990; Shnei-
derman, 1983). A computer state, or processing condition, may be verbally described
using text message elements, or it may be visually presented by encoding with static or
dynamic pictorial elements.
Learning to recognize computer states and manipulate computer functions often
requires learning from information displayed online, in the form of system or
application help messages. Online help messages typically consist of text messages that
explain a function, state, or procedure. Although computer states and functions are
routinely represented in a GUI using dynamic, pictorial elements, online help messages
rarely incorporate pictorial message elements.
Instructional message design principles suggest that the modality of message
elements employed during instruction should be the same as the modality found in per-
formance situations (Fleming & Levie, 1978). Instructional message design theory also
suggests that the better a symbol system conveys the critical features of a concept or
task, the more appropriate it is during instruction (Salomon, 1978). Reinforcing infor-
mation presented verbally with appropriate visuals has been shown to result in
significantly greater learning over presenting verbal messages alone (Fleming, 1987).
Research on the cognitive style dimension field dependence-independence has
shown that subjects who tend to be more field-independent identify and distinguish
objects in a complex visual field more readily than less field-independent subjects
(Witkin, Moore, Goodenough, & Cox, 1977). Additional studies by Witkin et al. (1977)
have shown that a learner's verbal abilities are only marginally correlated with his or
her level of field independence. These findings suggest that when using an unfamiliar
graphical interface, users with higher field independence should perform better than
users with lower field independence. According to Witkin's theory, this would result
from increased comprehension of the complex visual environment by individuals with
higher field independence.
Comparative expertise research has shown that when introduced to new com-
puter systems or applications, computer users differ widely in initial performance,
depending primarily on the extent of their prior computer experience, and to a lesser
degree on the difference between the type of interface encountered previously and the
one being introduced (Whiteside, Jones, Levy, & Wixon, 1985). These findings suggest
that with increased prior experience, where the potential for computer operation skill
transfer exists, initial performance will be higher than without prior experience.
The problem addressed by this study was the lack of experimental evidence to
demonstrate theoretically anticipated effects of field independence and computer expe-
rience on learning application functions in a GUI. In addition, there was a lack of
evidence regarding what effects the use of dynamic pictorial message elements in online
help information would have on learning application functions in a GUI.
Prior research has not provided clear answers to the following instructional
design and human-computer interface design questions: Does either the user's level of
field independence or level of computer experience moderate the effectiveness of spe-
cific message designs for online help? In particular, does a user's level of field
independence interact with varying levels of pictorial content in online help to influence
performance on tasks in a GUI? Would the extent of a user's prior computer experience
interact with varying levels of pictorial content in online help to influence performance
on application tasks? Would certain combinations of field independence and computer
experience interact in unique ways with the presence of dynamic pictorial elements in
online help to influence performance on tasks? This study was designed and conducted
to answer these questions.
Need for the Study
Graphical User Interfaces and Visual Learning
Pictorial or graphical user interfaces have been rapidly replacing textual inter-
faces as the primary means for manipulating computer system and application functions.
By representing state and function using nonverbal graphical elements (both static and
dynamic), graphical user interfaces have achieved greater expressive power than text-
based user interfaces (Horton, 1990). By improving the fidelity of representation for
computer state and function, graphical user interfaces may improve computer users'
ability to learn and control a wider range of functions than would be expected when
text-based user interfaces are employed.
Online help systems display documentation for computer operations and appli-
cation functions via video display devices. Online help messages consist primarily of
textual (verbal-digital) message elements that verbally describe the help topic. Online
help messages rarely incorporate pictorial (visual-iconic) elements that visually illustrate
the help topic. This would be an appropriate design for help messages in a primarily
textual user interface. Instructional message design principles suggest, however, that
online help messages describing the use and manipulation of visual-iconic elements in a
GUI should also contain visual-iconic elements that refer to and depict the computer
operations and application functions. Specifically, Fleming and Levie (1978) stated:
"In general, the modality used in the final testing or application situation should be the
modality employed during instruction" (p. 106). Affirming the utility of this principle
applied to the design of online help messages was one goal of this study.
As computer users interact with their systems using graphical, direct-
manipulation techniques, learning computer functions shifts from a primarily verbal
paradigm to an increasingly visual one. Strong verbal language skills cease to be the
sole aptitude required for successful learning and competent task completion. Visual,
nonverbal cognitive skills should take on added value and have greater influence on
concept attainment. This expectation follows from Paivio's dual-coding theory and has
been supported by research on the use of graphics in human-computer interfaces
(Rieber, 1990; Rieber & Kini, 1991).
Two aspects of dual-coding theory are important when considering online help
message design. First, verbal and visual cognitive mechanisms are interrelated; their
learning effects are symbiotic. That is, information that is coded in both verbal and
visual modes is remembered more readily and more accurately than information
encoded in only one mode (Fleming, 1987). Dual-coding theory also suggests that
visual stimuli are encoded more frequently in both modes than are verbal stimuli, thus
strengthening the value of visual, nonverbal stimuli for instruction.
Visual learning theory differentiates between presentations using static and
dynamic graphical presentations. Graphics have been shown to be effective attention-
gaining devices (Rieber & Kini, 1991). When appropriately designed, graphics may
enhance learning during computer-based instruction. Animated, or dynamic, graphical
images are fundamentally different from static graphics. Animation in computer-based
instruction involves rapidly updated computer screen displays, presenting an illusion of
motion. Because computer states and functions are themselves dynamic, their pictorial
or iconic representations in graphical user interfaces may be animated at appropriate
times. Just as online help presentations should incorporate the pictorial elements found
in graphical user interfaces, these elements should be appropriately animated to provide
a more effective representation of the computer state and function (Rieber, 1990).
Individual Characteristics and Performance in HCI
Prior research has documented significant variability of performance when users
are first introduced to unfamiliar computer systems and applications (Pocius, 1991;
Whiteside et al., 1985). Although many potential sources exist for this variance, and
while most sources have not been reliably measured, human-computer interaction
research has become increasingly focused on identifying the factors involved. For
example, HCI researchers have reported effects of general intelligence, prior computer
experience, cognitive style, academic background, and age on computer-based learning
outcomes (Pocius, 1991). Among the individual characteristics found to influence per-
formance during computer-based instruction, two were selected for further examination
in this study: (a) cognitive style and (b) computer experience.
Cognitive style. In a review of field dependence-independence research (Witkin
et al., 1977), four essential characteristics of cognitive style were described. Cognitive
style (a) refers to individual differences in how people perceive, solve problems, and
learn; (b) is a pervasive dimension that influences one's personality, not only one's
cognitive processes; (c) is stable over time, although it may be manipulated or altered
over time; and (d) is a bipolar dimension, where "...each pole has adaptive value under
specified circumstances, and so may be judged positively in relation to those circum-
stances" (p. 16).
Witkin's research has identified field dependence-independence as the cognitive
style dimension most widely investigated and systematically applied to educational
problems. His theory of cognitive style was not intended to narrowly categorize indi-
viduals as "field-dependent" or "field-independent." Rather, these terms have been used
as convenient, if somewhat misleading, labels for extremes of performance on percep-
tion tests (e.g., the Group Embedded Figures Test). In defining field dependence-
independence, Witkin et al. (1977) made the following assertion:
Because scores from any test of field dependence-independence form a
continuous distribution, these labels reflect a tendency, in varying degrees
of strength, toward one mode of perception or the other. There is no
implication that there exist two distinct types of human beings. (p. 7)
In describing individuals' cognitive styles, Witkin et al. consistently referred to
relative, rather than absolute, characteristics. For example, "relatively field-independent
persons have been found more likely to impose structure spontaneously on stimulus
material which lacks it" (1977, p. 9), and "the relatively field-dependent person tends to
adhere to the organization of the field as given" (p. 9). An individual's ability to per-
ceive structure in a complex graphical computer application and the ability to correctly
interact with that application could be expected to correlate positively with that
individual's degree of field independence.
Computer experience. A user's prior computer experience can dramatically
influence performance in a new, unfamiliar human-computer interface. In comparative
expertise research, performance differences between novices and experts have been
analyzed with regard to how they performed in problem solving situations (Lesgold,
Gabrys, & Magone, 1990), or on other cognitive operations such as memory and per-
ception (Aster & Clark, 1985). Whiteside et al. (1985) demonstrated consistent
differences between expert and novice computer users' performance on tasks in familiar
and unfamiliar user interfaces. They observed that as users' familiarity with the type of
interface increased, the higher their performance tended to be. User knowledge of
computers, in most cases directly derived from experiences using them, has been cited
as the most significant factor affecting performance on computer tasks (Moran, 1981).
Applving ATI Research to Online Help Design
Aptitude x Treatment interaction research is appropriate where the instructional
design problem involves determining how the elements of an instructional message
might affect learning for certain individuals under certain task conditions (Clark &
Salomon, 1986). In the design of online help messages, for users differing along
dimensions of cognitive style and computer experience, an instructional designer must
determine the level of information abstraction and the combination of media attributes
to apply to maximize user performance.
The general and specific effects that aptitudes such as field independence have
across a variety of instructional treatments need to be better understood. ATI research
techniques may be usefully applied to investigate such effects. Snow and Lohman
(1984) described the goal of theories of instructional treatment design: "There was a
clear prescriptive goal for such a theory. It was the design of an adaptive instructional
system [providing] alternative instructional treatments to fit the major differences in
aptitude profiles among students" (p. 350).
One of the advantages of computer-based instruction, including online help
information, is its potential to adapt each presentation to the aptitude and ability char-
acteristics of the individual learner. Where these characteristics can be reliably
measured and are clearly understood, instructional design principles may be applied to
adjust the presentation to optimize the fit between the learner and the lesson. Before
proceeding to design and validate adaptive instructional systems; however, principles
defining relationships between learner characteristics and specific instructional treatment
variables must first be identified and their reliability established. This has been a fun-
damental research objective of instructional designers developing intelligent tutoring
systems (Perez & Seidel, 1990; Wiggs & Perez, 1988). This study was designed to
identify and measure specific ATI effects and to contribute toward the design of future
adaptive instructional systems.
Another goal of this research was to test the validity of instructional message
design principles based on Paivio's dual-coding theory applied to the design of online
help in a GUI. In addition, this research attempted to determine whether Aptitude x
Treatment interaction effects exist between an individual user's cognitive style or level
of computer experience and the use of pictorial messages in online help. Evidence
relating to such interaction effects has been systematically collected and examined in
Definition of Terms
The term graphical user interface implies more than the use of a graphics display
terminal to present a human-computer interface. There are four key aspects to the
design and operation of a GUI: (a) the technique of direct-manipulation is broadly
applied; (b) the state of data and program objects is consistently represented using
nonverbal, pictorial (iconic) symbols; (c) changes in system states of interest to the user
are visually perceivable when they occur; and (d) functions that cannot be controlled
using pictorial symbols are accessible using textual menus of consistent structure and
organization (Shneiderman, 1983).
The term field dependence-independence has been applied by Witkin et al.
(1977) and many other researchers to designate a dimension of cognitive style. Witkin
et al. described this trait as "the extent to which the person perceives part of a field as
discrete from the surrounding field as a whole, rather than embedded in the field ... or,
to put it in everyday terminology, the extent to which the person perceives analytically"
(p. 7). Further, they described field dependence-independence as "a broad dimension of
individual differences that extends across both perceptual and intellectual activities"
In general terms, experience has been defined as knowledge or skill gained
through activity or practice. The term computer experience has been used in this study
to refer to both the extent of prior computer interaction activities and the types of prior
computer interaction experience (e.g., interaction with different types of user interfaces,
computer applications, and systems). Furthermore, computer experience is operationally
defined in this study as the score obtained on the experience scale of the Computer
Experience and Competence Survey.
Pictorial message elements are the structured components of an information
display, comprised of organized visual-iconic symbols, and designed to convey specific
meaning. Pictorial (visual-iconic) symbols also are differentiated from textual
(verbal-digital) symbols. Pictorial message elements may be either static or dynamic.
Dynamic pictorial elements periodically or continuously change in appearance, whereas
static pictorial elements have fixed visual appearance. In this study, digital motion
video playback sequences were used to operationalize dynamic pictorial message ele-
ments within the instructional treatment.
This study was designed to answer several research questions relating to a
learning situation in which computer-based instruction was implemented using online
help displayed in a GUI, where the GUI was unfamiliar to the users, and where the
instruction was systematically varied by adding dynamic pictorial elements to text-based
help displays. The research questions were stated in the form of the following null
1. No significant differences in application task performance result from a
three-way interaction among field dependence-independence, prior computer experience,
and the presence of dynamic pictorial message content in online help.
2. No significant differences in application task performance result from an
interaction between prior computer experience and the presence of dynamic pictorial
message content in online help.
3. No significant differences in application task performance result from an
interaction between field dependence-independence and the presence of dynamic picto-
rial message content in online help.
4. No significant differences in performance on computer application tasks exist
between subjects viewing text-only online help and subjects viewing online help con-
taining text and dynamic pictorial elements.
5. No significant relationship exists between prior computer experience and a
computer user's performance on computer application tasks in an unfamiliar GUI.
6. No significant relationship exists between field dependence-independence and
a computer user's performance on computer application tasks in an unfamiliar GUI.
Assumptions and Limitations
The hypotheses given above state the core questions of this research. In
attempting to find answers to these research questions, a number of assumptions were
made and certain limitations were accepted which constrained the research problem and
the generalizations which might be made regarding the results. These assumptions and
limitations are discussed in detail below.
Variance of Treatment Duration
The time that subjects spent using online help messages was another, potentially
confounding variable in this study. The use of online help, and the selection and dis-
play of dynamic pictorial elements, was entirely subject to individual user control and
discretion during task completion. The use of online help by subjects participating in
this study was assumed to be representative of their use of help in similar computer
application learning tasks. The between subjects variance of the use of online help was
statistically controlled by treating time in help as a concomitant variable in the analysis
The subjects for the experiment were sampled from an adult population of pri-
marily undergraduate university students enrolled as business college majors. This
population was expected to exhibit a unique and characteristic distribution of cognitive
style and computer expertise. The generalization of results in this study has therefore
been restricted to this population. Caution should be used in generalizing any results
from this study to other populations.
The computer application and the nature of application tasks were selected to be
meaningful and relevant to the sample population. Individuals from the sampled popu-
lation (business majors at a university) were required to demonstrate competencies in
computer operations, specifically spreadsheet applications. In addition, tasks were
arranged in a sequence such that the completion of each task was one step toward a
project goal (e.g., creating and printing a graphical representation of a small company's
annual balance sheet data). The tasks, therefore, had intrinsic incentives that were
expected to increase subjects' motivation to learn the operations of the computer system
and to complete application tasks.
The computer-based instructional treatments (online help messages) were
assumed to involve a degree of novelty because the sample was comprised of students
with varying computer experience, but who had no prior exposure to the computer
system used--IBM Operating System/2 I version 2.0 (OS/2)--and its direct-manipulation
GUI. At the time of this study, this version of OS/2 was a new product with many new
features, particularly with respect to the design and operation of its GUI. Novelty
effects may have also derived from the use of digital motion video technology to
present the dynamic pictorial message elements in online help. The presence of these
novelty effects was considered when describing and characterizing the results of this
Principles of instructional message design should be carefully applied to the
design of online help in human-computer interfaces. Where these interfaces employ
direct-manipulation techniques and rely on the use of pictorial (visual-iconic) symbols
to represent computer state and function, online help should also employ similar pre-
sentation symbologies. For some individuals, learning should improve as the
instructional conditions more closely resemble the criterion task performance condi-
tions. These learning benefits, however, may be altered or limited by individual
Computer user characteristics, particularly cognitive style and computer experi-
ence, may influence learning and performance in the human-computer interface. The
combination of specific online help designs with these user characteristics may result in
detectable Aptitude x Treatment interactions. Detailed understanding of such ATI
effects may prove to be useful in the design and development of adaptive, computer-
based instructional systems. This study was designed to identify and measure
relationships that exist among the online help message design variables and the user
aptitude variables (computer experience and field dependence-independence), and to
I Operating System/2 and OS/2 are registered trademarks of International Business
recommend directions for future research on related problems of instructional design for
REVIEW OF LITERATURE
This study was designed to examine the effects of field independence and com-
puter experience on application task performance in an unfamiliar graphical user
interface (GUI). This study also was designed to investigate whether Aptitude x
Treatment interaction effects occur between computer users' cognitive style (field
dependence-independence) or prior computer experience and the use of dynamic picto-
rial message elements in online help messages. The research questions addressed by
this study were derived from an examination of theories contributing to research on
instructional message design, human-computer interaction, cognitive styles, and com-
puter expertise. This chapter describes these theories and where they intersect,
identifies issues raised in previous research, and summarizes the body of literature rel-
evant to the research questions addressed by this study.
Instructional Message Design for Visual Learing
Modality in message presentation refers to the sensory modes utilized to convey
meaning. One principle of instructional message design states that "the modality used
in the final testing or application situation should be the modality employed during
instruction" (Fleming & Levie, 1978, p. 106). This principle is relevant to the design of
online help where the criterion tasks require performance in a highly pictorial or
graphical human-computer interface. It suggests that instruction (e.g., information pre-
sented in online help) should incorporate supporting pictorial message elements together
with textual elements.
The symbol systems used to convey information during instruction differ in their
capacities to support the extraction of meaning (Salomon, 1978). The better a symbol
system can convey the critical features of an idea or event, the more appropriate it
should be for instruction. In a direct-manipulation user interface, where tasks involve
the manipulation of iconic visual symbols, the online help should directly incorporate
those iconic symbols, rather than simply refer to them with verbal descriptions.
Dual-mode theories suggest, and evidence supports the argument, that repeating
verbal information with visuals results in significantly greater learning over verbal
messages alone (Fleming, 1987). Dual-coding theory contends that two independent
information encoding mechanisms exist. One stores and processes information as ver-
bal codes while the other stores and processes information as visual images. These two
modes also are referred to as analytic and analogic modes, respectively (Clark &
Paivio's dual-coding theory of visual learning clearly suggests that to support
construction of an adequate mental model of the computer system and its operations, the
online information should attempt to present messages describing those operations using
visual, nonverbal stimuli in addition to textual, verbal stimuli (Rieber & Kini, 1991).
Both static and dynamic visual stimuli may be incorporated into the online messages.
Because a direct-manipulation GUI is inherently a dynamic pictorial display, it follows
that adding dynamic pictorial message elements to online help would enhance its effec-
tiveness. This study was designed in part to examine certain effects of using dynamic
pictorial message elements in online help.
Dynamic Pictorial Message Elements
Dynamic pictorial message elements may be incorporated into online help using
various techniques, such as animated graphics or motion video windows. The use of
animated graphics in computer-based instruction is becoming increasingly common
(Horton, 1990; Rieber, 1990). Instructional message design principles suggest that
where temporal or directional concepts are being taught, dynamic pictorials may be
used to visually portray these concepts and this will improve learning (Rieber & Kini,
1991). Operating a computer system with a GUI requires understanding the visible
motion of dynamic pictorial symbols in the user interface. It follows that the use of
dynamic pictorials in online help may improve task performance in the GUI. This
study was designed to measure the effects of dynamic pictorial elements by incorporat-
ing digital motion video segments into online help messages.
Instructional message design principles suggest that instruction should incorpo-
rate dynamic pictorial symbols wherever such symbols are employed in a task
environment, to support the extraction of meaningful information relevant to that task
environment. These principles have not been tested, however, with respect to the design
of online help in direct-manipulation interfaces. One goal of this study was to evaluate
the utility of the dual-coding theory as applied to learning application operations in a
graphical user interface.
Human-Computer Interface Design
Direct-Manipulation and Nonverbal Literacy
Graphical user interfaces were developed to allow the computer user to more
directly manipulate an interactive computer system's state, instead of relying on com-
mand language interpreters (Shneiderman, 1983). The manipulation of visual iconic
symbols displayed by the computer, using visual tools (e.g., an arrow pointer), results in
an immediately visible change in system state. This is the central principle of direct-
manipulation interface design. It requires the real-time animation of iconic display
elements that are mnemonics for system or application states and functions. These
visual iconic symbols appear to the user to represent controls that operate functions of
the computer which otherwise would be invisible and more difficult to understand and
Learning to control computer functions in a GUI involves different cognitive
processes than are required of a user learning the same functions in a textual, command
language user interface. This follows directly from research on verbal and nonverbal
literacy (Sinatra, 1986). A learner's perception and memory of the syntax and semantics
of verbally encoded messages depends on verbal language skills, which are sequential
and analytical in nature. On the other hand, the perception and memory of spatial-
temporal manipulation of pictorial elements depends on visual and kinesthetic processes,
which are holistic and analogical in nature. This contrast between analytical (verbal)
and holistic (nonverbal) processes also has been presented as the fundamental determi-
nant of cognitive style differences (Miller, 1987). Thus, a theoretical link can be
proposed between dual-coding theory and cognitive style theories. This link provides a
basis for this study.
Symbol Systems and Mental Models
An instructional message design that is appropriate for teaching the skills
required in a verbal command language interface may be inadequate or inefficient for
teaching the skills required in a predominantly iconic interface. This derives from
Salomon's theory of media attributes. "The closer the match between the communica-
tional symbol system and the content and task-specific mental representations, the easier
the instructional message is to recode and comprehend" (Clark & Salomon, 1986,
Learning to manipulate a computer system's functions requires the user to
develop an internal representation, or mental model, of the system (van der Veer, 1990).
A mental model may be based largely upon propositions encoded verbally, as would be
expected for users of a command language interface, or it may be based predominantly
on analogical images. An effective mental model should parallel the organizational
metaphors depicted in a GUI. Factors that influence the development of these mental
models are significant determinants in the design of user interfaces. The ease with
which a user creates an adequate mental model of a computer system or application
largely determines the productivity that user will be able to achieve.
The formation of mental models while learning computer functions in a graphi-
cal user interface depends more heavily on analogical, rather than analytical,
information processing. Individual differences in cognitive style, particularly field
dependence-independence, are characterized by differing tendencies to exercise analyti-
cal information processing. This theoretical link between cognitive style and
construction of mental models provides a basis for a deeper understanding of how field
independence may influence performance on tasks in a GUI.
Direct-manipulation user interfaces support formation of visual as well as verbal
mental models, which computer users may construct to help them manipulate system
and application functions. The ability of users to form correct mental models has been
demonstrated (van der Veer & Wijk, 1990). Performance on application tasks in a GUI
is believed to depend on the user's ability to form effective mental models. To the
extent that online help can be designed to facilitate this ability, performance should
Cognitive Style Effects
The Dimensions of Cognitive Style
Cognitive styles are psychological dimensions that represent consistent tenden-
cies in an individual's manner of acquiring and processing information. Gregorc (1984)
indicated that "stylistic characteristics are powerful indicators of deep underlying psy-
chological forces that help guide a person's interactions with existential realities"
(p. 54). Many dimensions of cognitive style have been reported. Canelos, Taylor,
Dwyer, and Belland (1988) summarized nine different cognitive style dimensions.
Miller (1987) employed an information processing model of cognition in his analysis of
eight dimensions of cognitive style. The most compelling and authoritative research on
cognitive styles, however, has been conducted regarding the dimension of field
dependence-independence, which has been extensively studied by Herman A. Witkin
and his associates.
Field Dependence-Indenendence (FDI)
Begun in 1941, Witkin's research into the human phenomenon of field indepen-
dence has been extensively reviewed, extended, and broadly applied. Witkin's early
research detected significant individual differences in perceptual abilities, particularly
the ability to perceive an upright object embedded in a tilted frame (Witkin et al.,
1977). The concept of field dependence-independence was first described by Witkin in
1954 (Canelos et al., 1988).
The process of attention is the selective focusing of conscious mental activities.
Research shows that there are both deliberate and automatic forms of attending to
stimuli, and that there is evidence for individual biases toward relying on one form or
the other. Individual differences in selective attention have been found using various
tests to measure field dependence-independence. As measured with the Embedded
Figures Test (EFT), relatively field-independent persons exhibit deliberate attention
focusing and an ability to disembed an item from an organized context of distracting
cues. Relatively field-dependent persons, on the other hand, exhibit a deficit in this
regard or a tendency toward relying on more automatic attention processes (Miller,
When compared to more field-independent learners, relatively field-dependent
learners are less able to identify discrete objects in complex visual fields, but are better
able to perceive and identify patterns in complex visual fields (Witkin et al., 1977). In
their work defining this dimension of cognitive style, Witkin et al. elucidated the bipo-
lar, process-oriented, enduring, and pervasive characteristics of cognitive style.
Although other dimensions of cognitive style, such as impulsivity-reflectivity, also are
being investigated with respect to performance on tasks involving computers (van Mer-
rienboer, 1988, 1990), the FDI dimension is overwhelmingly the most frequently
Cognitive Style and Computer-Based Learning
Many researchers have investigated relationships between the level of field
dependence-independence of computer users and various aspects of their performance
when learning with or from computers (Burwell, 1991; Canelos et al., 1988; Canino &
Cicchelli, 1988; Cathcart, 1990; Cavaiani, 1989; MacGregor, Shapiro & Niemiec, 1988;
Martin, 1983; Mykytyn, 1989; Post, 1987). Despite sometimes inconsistent findings,
the frequency and recency of these studies indicate that compelling purposes motivate
this research. Some of these studies were designed to detect Aptitude x Treatment
interaction effects between learner cognitive style and instructional treatments. One
objective of this study was to systematically measure relationships between cognitive
style and instructional message design variables.
Learners' cognitive style can significantly affect their ability to perceive,
remember, and apply declarative and procedural knowledge. Due to their greater ana-
lytic capacity, relatively field-independent learners exhibit greater skill disembedding
simple visual stimuli embedded within a complex field. More field-dependent learners
perform relatively poorly at such tasks, due to their greater tendency to process infor-
mation in a holistic manner. By identifying an individual learner's cognitive style,
computer-based instruction may adapt the presentation to the individual by appropri-
ately varying certain instructional message design parameters under software control.
Researchers believe that this may significantly improve learning from computer-based
instruction (Canelos et al., 1988).
Comparative Expertise Effects on Mental Models
Experts and novices approach problem solving in different ways (Aster & Clark,
1985; Lesgold et al., 1990). Some researchers have suggested that the underlying con-
ceptual model in a GUI, evident to expert users, is undetected or misinterpreted by
novice users. These expert-novice differences refer specifically to differing levels of
experience, not to different levels of general intelligence (Aster & Clark, 1985; Mestre
& Touger, 1989).
HCI research also suggests that cognitive processes employed by computer users
early in the use of a new user interface differ from those used later. This may result
from changes in the nature of tasks presented (e.g., tasks become more complex and,
therefore, more difficult) or because users develop new task completion strategies over
time as their expertise increases (Chiesi, Spilich, & Voss, 1979). Another explanation is
that the user's mental models gradually become more complete and accurate (van der
Veer & Wijk, 1990).
Whiteside et al. (1985) found that the performance of users who differ in prior
computer experience was consistent across several different user interfaces. Regardless
of the user interface style presented, novice users with little or no prior computer expe-
rience performed at the lowest levels. Those users with prior computer experience, but
not with the particular user interface tested, scored higher than novices regardless of the
interface. Those users who had extensive prior experience with the user interface being
tested performed consistently better than either novice users or those with experience in
a different user interface. This study was designed to examine the effects of prior
computer experience on learning application function when an unfamiliar user interface
Computer expertise is not, however, a single, monolithic dimension. At a
minimum, prior experience must be evaluated both in terms of extent (depth) and range
(breadth). An expert user may have considerable depth of experience, yet be limited to
a single type of user interface or system and thus have limited breadth of experience.
An expert user with both depth and breadth of experience has worked on a variety of
systems with differing interface styles and has sufficient exposure to the operation of
each to be adept at manipulating and controlling many of its functions.
An expert user's experience is, however, rarely global or comprehensive. No
matter how extensive a user's experience, there are aspects of system function that may
never be explored. This is partially due to the complexity of modern computer systems
and partially due to limited user needs. Systems are designed to fulfill the requirements
of a wide variety of users, but each individual user requires only a subset of that
system's functions to perform the tasks at hand. As a computer user's tasks become
more varied and complex, the range of the user's experience with an interface will
In addition, user experience level is not a stable factor. Whenever a user
engages in a new type of task, that user becomes more experienced (Aster & Clark,
1985; Federico, 1983). The changes that occur in the user during this progression
include becoming more consistent and precise, engaging in more complex forms of task
abstraction, using more automatized and internalized strategies for performance, and
becoming increasingly skillful in the application and interpretation of the rules for per-
Mental Models and HCI Metaphors
A user's mental models of a computer system are continually being modified,
extended, and tested against the actual behaviors of the system (van der Veer & Wijk,
1990). Users differ in the style of representation that they apply to construct mental
models, either verbal, visual, or dual-mode styles. "The learning process that leads to
the mental model will be based on analogies to known situations and systems. The
learning process may be facilitated by providing adequate metaphors" (p. 195). In the
case of direct-manipulation interfaces, visual metaphors are used in the interface design
and appear as pictorial elements which may be visually manipulated by the user. A
visual metaphor that extends a conceptual model assists with perceptual processing and
the formation of a correct mental model.
Comparative expertise research has demonstrated that experts and novices differ
in the way they perceive and solve problems. Research on expert-novice differences
has also shown that these differences are related to experience in a specific task domain
and are not a measure of general intelligence. The mental models possessed by novices
are marked by their simplicity and incorporation of surface features, while those pos-
sessed by experts reflect greater abstraction and organization according to fundamental
principles related to the task domain. Visual metaphors presented using pictorial sym-
bols can assist novice computer users with learning system function by supporting
formation of mental models.
Measuring Computer Expertise
One of this study's objectives was to examine the relationship between computer
users' prior experience and their ability to learn application functions in an unfamiliar
GUI. The selection or construction of an instrument to reliably measure computer
experience was therefore of primary importance.
Instruments for assessing computer literacy, experience, and knowledge have
been the subject of much research. During the past decade, computers have become
essential tools applied in nearly every occupation. As a result, public schools, colleges
and businesses large and small have had to determine the computer skills of their stu-
dents and employees. However, development of instruments to measure computer
skills, knowledge, and expertise has lagged behind concurrent rapid changes in com-
puter technology. Such tests are difficult to design and validate since the subject matter
and the skills to be evaluated are continually changing as computers rapidly evolve
(LaLomia & Sidowski, 1990).
For this study, computer expertise was defined as the combination of an
individual's computer experience with that individual's computer knowledge or compe-
tence. Experience with computers is acquired over time, is cumulative and incremental.
It relates the extent and type of computer usage the individual has engaged in, and is
typically measured using a self-reporting, survey-type instrument. Knowledge of, or
competence with, using computers is highly dependent on the specific computer systems
and programs involved, although this knowledge may transfer more or less well from
one system or program to another. Computer competence may be measured either by
administration of an objective test composed of cognitive knowledge items, or it may be
measured directly within the context of task-specific computer usage.
Published computer literacy, competency, and knowledge assessment instruments
have become outdated by the significant and rapid changes in computers and computer
applications. One area of computer skills and knowledge particularly exposed to rapid
change is the use of application programs such as word processors, spreadsheets, data-
base systems, and graphical or drawing editors. Assessing expertise in the use of such
computer application programs requires measurement of performance on those aspects
of application software which impinge directly on the user's ability to control the
computer's function. Such control is exercised through an interaction dialog with the
computer via the human-computer interface. An instrument designed to measure com-
puter expertise must have a component that measures the ability to interact with
software through a variety of user interfaces, both visual-iconic and verbal-digital in
Components of Expertise
The measurement of expertise--the quality or aptitude of being an expert within
a particular domain--must comprise the measurement of both experience and knowl-
edge. Experience can be expressed in terms of frequency and duration of practice
within the domain. For computer user expertise, this may be expressed as a numerical
value ranging from zero, indicating no experience, to some arbitrarily high value, indi-
cating the highest experience level among the particular sample of individual users.
An equally important aspect of expertise is cognitive knowledge regarding the
functions of computer systems and applications and how the software may be operated.
Knowledge about computer systems and operations is frequently referred to as computer
literacy in the research literature on educational computing. In a review of six com-
puter literacy and aptitude scales, LaLomia and Sidowski (1990) found all six had
defined computer literacy and operationalized their definitions in different ways. Items
testing knowledge of computer operations or applications appeared in most of these
Both experience and knowledge were measured in the Computer Competence
Test developed for the 1986 National Assessment of Educational Progress (NAEP)
(Educational Testing Service, 1988). In creating the NAEP Computer Competence
Test, Martinez and his colleagues at ETS extensively and systematically developed 228
objective, multiple-choice items to measure computer experience and knowledge (Mar-
tinez & Mead, 1988). These included items covering general knowledge of computer
systems, knowledge of four common types of computer applications, and knowledge of
two computer programming languages. Items designed to assess student attitudes
towards computers were also included. The test was administered to students in the
third, seventh, and eleventh grades during the 1985 to 1986 school year.
All of the published and privately available computer experience and computer
literacy instruments reviewed had serious content validity problems due to being pub-
lished more than five years previous to this study. Thorough editing and revision would
have been required to improve the validity of these instruments. The Computer Com-
petence Test developed for the NAEP was selected for use in this study as the most
comprehensive instrument. All the original NAEP test items were obtained from ETS.
The selection of items used, the editing of item text, and other modifications to the
NAEP items made for this study are described in Chapter 3.
The measurement of computer literacy and expertise has become a subject for
much research in the past decade. The availability of valid and reliable instruments for
measuring computer expertise has been limited by the rapid changes in computer tech-
nology and applications. Computer competence tests developed prior to the widespread
availability of graphical user interfaces would require substantive changes to accurately
measure expertise with current computer systems. The instrument used in this study to
measure computer experience was derived from the Computer Competence Test devel-
oped for the 1986 National Assessment of Educational Progress.
This study was designed to investigate an apparent link between the theory of
field dependence-independence and the dual-coding theory of visual learning. This link
was predicated on a characterization of the mental processes for visual learning as being
more holistic-analogical than those for verbal learning. Research into the formation of
mental models by users of direct-manipulation computer interfaces provided a paradigm
for this investigation. Researchers have found that relatively field-independent com-
puter users learn the operations of some computer interfaces more readily than do
field-dependent users. This suggests that field-independent users may develop mental
models more efficiently than field-dependent users. By manipulating the pictorial con-
tent of online help messages, this study attempted to test whether users with different
levels of field independence would respond differently to varying levels of visual-iconic
Prior computer experience has been identified as the most significant factor
contributing to successful performance in learning new human-computer interfaces.
This study examined the influence of computer experience on performance in an appli-
cation where information about application operations was presented using online help.
Instruments designed to measure computer experience, knowledge, literacy, and compe-
tence have not kept pace with the rapid changes in human-computer interfaces. In order
to reliably determine the effects of computer experience on performance, instruments
that accurately measure experience must be developed. This study employed one such
instrument and attempted to demonstrate its validity and reliability.
This study was conducted to examine the effects of field independence and
computer experience on learning computer application functions in an unfamiliar
human-computer interface. Students who were enrolled in a university-level business
curriculum completed a computer training session during which their use of online help
was observed and their performance on computer tasks was measured. In the training,
students were randomly assigned to one of two different online help formats. Online
help provided information on how to use the computer system and a spreadsheet appli-
cation. One online help format displayed text-only information, while the other format
displayed dynamic pictorial elements in addition to text. The display of help messages
in both online help formats was under individual student control at all times. The
dependent variable was performance on application tasks in the training posttest.
A multiple covariance analysis was used to examine effects of the online help
format, prior computer experience, cognitive style (field dependence-independence), and
the amount of time in help on application task performance. This analysis also tested
for potential Aptitude x Treatment interaction (ATI) effects among field dependence-
independence, computer experience, and the display of dynamic pictorial content in
Before proceeding with this study a pilot study was conducted (Tyler, 1993).
The objective of the pilot was to validate the instructional materials and performance
measurement procedures and to collect data to support methodology decisions which
could not be made based solely on existing literature. Results of the pilot study that
directly influenced the design of this study are described where appropriate. This
chapter presents the experimental design, the population and sample, the assessment
instruments, the instructional treatments and materials employed, and the data collection
methods used in this study.
A fully randomized experimental design with pretest-treatment-posttest sequence
was employed in this study. A random sample of 38 subjects was obtained from a
population of university students enrolled in an undergraduate business management
course. The subjects completed assessments for field independence and prior computer
experience. Each student in the sample was then randomly assigned to one of two
treatment conditions consisting of different online help formats. Both help formats
displayed identical textual information, while one displayed dynamic pictorial elements
in addition to the text. Each student completed individual computer training in the use
of a personal computer equipped with a direct-manipulation graphical user interface
(GUI) and a graphical spreadsheet application. The training consisted of a pretest, 12
training lessons, and a posttest. During the training, online help was the primary
instructional resource available to students. For the students to obtain information on
operating the computer system or the spreadsheet application, they had to activate
online help displays. At the completion of training, students' posttest performance
scores were analyzed using an analysis of covariance.
Analysis of Covariance
Data collected in this study were interpreted using a one-way analysis of
covariance (ANCOVA) with multiple covariates. This analysis was performed using an
ANCOVA hierarchical regression analysis technique (Cliff, 1987). This technique is
equivalent to multiple covariance analysis through multiple regression (Huitema, 1980).
In this multiple covariance analysis, the effects of the treatment factor (a categorical
variable) and multiple covariates (interval scale concomitant variables) on the dependent
variable were determined using linear regression models. The use of ANCOVA in this
study had three purposes: (a) determine whether significant interactions had occurred
between the online help format and any of the covariates, (b) test the effects of the two
different online help formats on the dependent measure, and (c) examine the regression
relationships between the concomitant variables (covariates) and the dependent measure.
This approach for ANCOVA emphasized a linear regression analysis between the
interval dependent and independent variables, within each level of the treatment factor.
Independent variables. There were four independent variables in this analysis.
The two aptitude variables, field dependence-independence (FDI) and computer experi-
ence (CEXP), were measured as interval scale random variables. FDI was measured
using the Group Embedded Figures Test (GEFT) (Witkin, Oltman, Raskin, & Karp,
1971). CEXP was operationalized as the score on the experience scale of the Computer
Experience and Competence Survey, which was developed for this study using items
selected from the Computer Competence Test of the 1986 National Assessment of
Educational Progress (Educational Testing Service, 1988).
Time in help (THELP), a third interval scale independent variable, was mea-
sured using a computer software logging program. THELP reflected the accumulated
time, in minutes, that a subject displayed online help messages during the 12 training
lessons. The final independent variable was the online help treatment factor (TREAT).
TREAT was a categorical variable with two levels representing the two online help
formats to which subjects were randomly assigned. The first treatment level was text-
only help (TOH), wherein online help displays contained only textual information. The
second treatment level was text-with-motion-video help (TMVH), which displayed the
same text as TOH, but also displayed--when initiated by the subject--dynamic pictorial
message elements in the form of motion video windows.
Dependent variable. The dependent variable in this analysis measured the
subject's performance on computer operation tasks in the direct-manipulation GUI.
Application task performance was computed as the training posttest performance score
(POST). The training posttest, administered at the completion of the training lessons,
was composed of three application tasks involving operation of a graphical spreadsheet
program. This performance measure accounted for both task completion accuracy and
Despite its computational complexity, ANCOVA was the most appropriate ana-
lytical procedure for this study. This was due primarily to the continuous nature of
measurements used to assess both field independence and computer experience (Cliff,
1987). Since interval measures on these aptitude variables were used, rather than con-
verting the interval scores into levels of blocking variables, greater precision was
attained in determining the relationships between each aptitude variable and the depen-
dent measure. Also, in evaluating the effect of the online help format, ANCOVA
reduced any unintentional bias in the comparison of treatment groups that might have
occurred as a result of experimental mortality or sampling fluctuations. Comparison of
treatment groups was done on the basis of adjusted group means rather than actual
group means. Adjusted means removed the within-group variance accounted for by the
regression of the dependent variable on each of the covariates. The adjusted treatment
mean differences could therefore be interpreted as independent of all variables used as
covariates (Huitema, 1980).
Assumptions for ANCOVA
There are several assumptions for the analysis of covariance, including those
which apply to analysis of variance. First, the observations must be independent. In
this study, treatments were individually administered, so this assumption was met.
Second, the dependent measure (POST) was normally distributed within each treatment
group, satisfying the assumption of normality of Y scores. Additional assumptions
apply in a unique manner to ANCOVA and are discussed in detail in the following
Homogeneity of regression slopes. This assumption for ANCOVA states that the
regression slopes of the dependent variable on any covariate must not be significantly
different between treatment groups. If the regression slopes associated with the various
treatment groups were not the same, it would mean that the ANCOVA model did not fit
the data. In that case, an alternative method such as the Johnson-Neyman technique
would have been required (Stevens, 1990). Testing this assumption was the first step in
the analysis technique applied in this study.
Homogeneity of variance. In ANCOVA, there should be no significant differ-
ences in the distributions for each of the covariates between the different levels of the
treatment variable, particularly when group sizes are unequal (Stevens, 1990). In this
study, subjects were randomly assigned to one of the two online help formats. After
accounting for mortality effects, 18 subjects completed training in the TOH group while
20 subjects completed training in the TMVH group. Because this resulted in an unbal-
anced design, a test of the homogeneity of variance was required.
Independence of treatment and covariates. In ANCOVA, the treatment should
not directly influence the covariate scores obtained. In this study, the aptitude variables
CEXP and FDI were obtained prior to, and therefore independent of, the instructional
treatment. The third covariate, THELP, was measured during training and could have
been indirectly influenced by the treatment. An ANOVA on THELP was performed to
determine that this covariate was independent of treatment (Huitema, 1980).
Linearity of regression. ANCOVA assumes a linear relationship between each
covariate and the dependent measure. For this data set, a visual inspection of scatter-
plots generated when POST was plotted against CEXP, FDI, and THELP demonstrated
that the assumption of linear relationships was tenable.
Selection of Covariates.
To conduct ANCOVA for this data set, each of the proposed covariates was
scrutinized for appropriate use in the analysis following two recommendations by
Stevens (1990). First, independent variables that were correlated with the dependent
variable could be used as covariates. For this study, the relevant Pearson correlations
were: (a) CEXP with POST, r = 0.63; (b) FDI with POST, r = 0.42; and (c) THELP
with POST, I = -0.51. Given these correlations, CEXP, FDI, and THELP were appro-
priate for use as covariates with POST as the dependent variable.
Second, when multiple covariates are used they should not be strongly correlated
with each other, so that each may contribute a unique component to the analysis. For
this analysis the relevant Pearson correlations were: (a) CEXP with FDI, r = 0.21; (b)
CEXP with THELP, r = -0.40; and (c) FDI with THELP, r = -0.19. Except for the
moderate correlation between CEXP and THELP, these covariates contribute separately
to the analysis.
As a final consideration for selecting covariates, Huitema (1980) recommended
limiting the number of covariates used, in relation to the sample size, to prevent the
adjusted means from becoming unstable. Following this recommendation, the
ANCOVA model in this study included only three covariates.
Computing ANCOVA with Three Covariates
Performing analysis of covariance with three covariates is not a common statis-
tical procedure. In general, it is an extension of a one-way ANCOVA, but with
multiple covariates. Huitema (1980) referred to this as multiple covariance analysis.
This method proceeded in three steps. First, a test for the assumption of homogeneous
regression slopes was performed. This test determined whether there were significant
interactions between online help format and any of the three covariates. Second, after
confirming that no significant interactions had occurred, a test was performed to
determine whether the online help format had a significant effect. Third, the analysis
determined whether there were significant regression effects on the dependent measure
for any of the three covariates.
For all computations described below, the sources of variance for each
ANCOVA model were computed using the Statistical Analysis System (SAS)', release
6.07. Procedures for programming the SAS General Linear Models (GLM) procedure
to perform the ANCOVA hierarchical regression analysis were derived from Cliff
(1987) and from SAS User's Guide: Statistics (SAS Institute, 1985). Certain statistical
tests not supported in the SAS GLM procedure were also required. The formulas for
calculating these test statistics are described below.
Test for homogeneous regression slopes. In this study, ATI effects were antici-
pated by the research questions. The test of the ANCOVA assumption of homogeneous
regression slopes was a key first step in the analysis, because it tests for interactions
between the covariates (aptitude variables) and the treatment factor. Had significant
interactions been found, continuing with the remainder of the multiple covariance
analysis would not have been appropriate.
Testing the assumption of homogeneous regression slopes required computing
the sum of squares for two linear regression models, referred to as the complete model
and the reduce model (Agresti & Agresti, 1979). These two regression models were
used to calculate the between-groups regression sum of squares and the within-groups
regression sum of squares. An F test statistic was then computed as the ratio of the
mean squares between over the mean squares within.
The general form of the complete ANCOVA model with 3 covariates is shown
in Equation 1. This linear regression model yields predicted I scores (,~) and
includes the X-intercept (a), X-intercept difference parameter (6) for the effect of
SSAS is a registered trademark of SAS Institute, Inc.
treatment (A), regression slope parameters (Pi) for each covariate (X), and cross-
product term coefficients (y,).
rd = a + 5A + PX, + PAX2 + P3X + y,(XA) + y,(XA) + y,(X3A) (1)
The reduced ANCOVA model, shown in Equation 2, follows the form of the complete
model but eliminates the interaction terms. The reduced model calculates predicted Y
scores assuming there were no interactions between treatment and the covariates.
rNA = a + 8A+ p1X + PX + P3X3 (2)
The regression sum of squares and error sum of squares were computed for the
complete and reduced models shown above. These values were then used to compute
the F statistic to test for significant interactions between the covariates and the treat-
ment. This F statistic, calculated using Equation 3, tests the assumption of
homogeneous regression slopes (Cliff, 1987).
(SS, S,c) / p(g. 1)
F = (3)
SSE,~ / (N- pg-g)
In Equation 3, SER is the error sum of squares for the reduced model, SSE, is
the error sum of squares for the complete model, p is the number of covariates, g is the
number of groups, and N is the total number of subjects. If the resulting value of F did
not exceed the critical value, F4_.o.05, .p I_ the ANCOVA assumption of homoge-
neous regression slopes would have been met.
Test for significant treatment effect. Where the assumption of homogeneous
regression slopes was valid (i.e., no significant interactions between treatment and
covariates had occurred), the analysis next tested the treatment effect. For this test,
another F statistic was calculated to determine whether the regression lines representing
the two treatment groups had different Y-intercepts.
Similar to the test for homogeneous regression slopes, this test statistic was
computed using the error sums of squares for a complete model and a reduced model.
To compute this F statistic, the complete model was the ANCOVA regression equation
without interaction terms--the reduced model from the previous test--shown in Equation
2. The reduced model for this test was the regression equation with only terms for the
covariates, shown in Equation 4.
Lp, = a + P,X, + X2 + 3X3 (4)
The formula for calculating the F statistic for this test is shown in Equation 5. As in
the previous test, this equation compares the error sum of squares for the complete
model (SSc) with the error sum of squares for the reduced model (SS,,R). Also, p is
the number of covariates, g is the number of groups, and N is the total number of sub-
jects (Cliff, 1987).
(SR -SSEC) / (g- 1)
F = (5)
SSErC (N -g -p)
This test determined whether there were significant differences between the two online
help treatment groups on the dependent variable POST. Where the resulting value ofF
did not exceed the critical value, EF sgigp, the null hypothesis of no significant
treatment effect was not rejected.
Test for significant regression effects. The third and final step in this analysis
was to test whether there were significant regression effects on task performance for
each covariate. In this step, the ANCOVA resembled a multiple regression analysis and
the covariates became predictor variables (X,) in a prediction equation, as shown in
For each covariate, an F statistic was tested to determine whether a significant
relationship existed between the covariate and the dependent variable, posttest perfor-
mance. In other words, this tested whether the regression slope coefficients (P,) in this
prediction equation were nonzero. These F statistics were computed using a standard
linear regression analysis procedure.
The analysis of covariance method described above was used to test all the
hypotheses for this study. ANCOVA provided greater power than alternative methods,
but involved greater computational complexity. Given that this study was primarily
focused on examining the regression effects of the aptitude variables (covariates) on
application task performance, and determining whether Aptitude x Treatment interac-
tions had occurred, ANCOVA was the most appropriate analytical method for the data
gathered in this study.
Population and Sample
The population from which subjects were drawn for this study consisted of stu-
dents majoring in business curricula at a state university in southern Florida. Subjects
were randomly sampled from all students enrolled in four sections of Management
Information Systems, an undergraduate course required of all students majoring in a
business college program at the university. A total of 129 students comprised the
available population pool. In this student population, 85% were upper class under-
graduates. After identifying the population and conducting initial screening, individual
computer training sessions were held at a nearby corporate product evaluation center.
At the initial screening, all students were asked to volunteer for the study. In
return they received free training on an advanced personal computer operating system
and a graphical spreadsheet application. No other form of compensation or course
credit was given for participation. Because the subjects were business majors who were
required to learn computer operations, and because they were volunteers, a high level of
motivation to complete the training was anticipated. Experience with similar students
during the pilot study had confirmed this.
From the four course sections, all 129 students were screened using the Sign-un
Form for OS/2 Training and the Computer Experience and Competence Survey. These
instruments were administered during regular class sessions. The sign-up form was
designed to collect demographic and computer experience data. It is included here as
Appendix A. This form was also designed to check whether students had prior experi-
ence using the computer operating system (IBM Operating System/2 version 2.0).
Individuals in the population who indicated prior experience using this system were
eliminated prior to selection. Because the graphical spreadsheet application, PM Chart2,
was included as a feature of the OS/2 operating system product, experience with this
application was also determined in the initial screening.
Power estimate and sample size. Based on data gathered during the pilot study, a
relatively large effect size (f > 0.80) was estimated for the effect of computer experi-
ence on task performance. A small effect size (f < 0.50) was anticipated for the effect
of field independence on task performance. Using these effect size estimates, the
sample size for the study was set at 60 subjects using Cohen's power tables (Stevens,
1990). This value was based on setting a = .05, group size to 30, and achieving power
of 0.87 for an estimated effect size off = 0.40. This effect size corresponds to that
observed in the pilot study for the effect of field independence on posttest performance.
Experimental mortality. From the population pool, 60 student volunteers were
randomly selected and were scheduled for training appointments. Experimental
mortality resulted from appointment cancellations, no-shows, and incomplete training.
Of the 60 volunteers, 18 students scheduled training appointments and later either can-
celled the appointments or failed to appear for their training sessions. The remaining 42
students attended training sessions as scheduled. This sample was composed of 2
Sophomores (5%), 16 Juniors (38%), 20 Seniors (48%), and 4 students (10%) who
reported other academic status. There were 26 males (62%) and 16 females (38%) in
Incomplete training also contributed to mortality. Of the 42 subjects who
attended training, 32 subjects completed all 12 training lessons plus the pretest and
2 PM Chart is copyright 1991 by Micrografx, Inc.
posttest. Of the ten remaining subjects, four completed fewer than eight lessons and
were dropped from the analysis due to incomplete data. The loss of subjects due to
these mortality effects contributed to power problems in this analysis. The loss of four
low-experience subjects shifted the computer experience distribution slightly toward
higher scores. However, the distribution of computer experience scores for the resulting
sample of 38 was not significantly different from the distribution for the initial sample
of 60. Two of the four subjects dropped had scored above the mean on the GEFT,
while two had scored below the mean. Finally, these four subjects were evenly divided
between the two online help formats. The balanced loss of subjects indicates that mor-
tality did not bias the results.
Summary of the sample. The resulting sample of 38 subjects exhibited diversity
on both computer experience and field independence measures. All subjects completed
at least 8 of the 12 lessons during the training, covering all aspects of skills required for
the pretest and posttest. GEFT scores for this sample ranged from 2 to 18. These 38
subjects had normally distributed scores on the computer experience subscale (CEXP)
of the CECS instrument. CEXP scores for the sample ranged from 9 to 58. Although
mortality effects reduced the size of the sample, it remained representative of the target
population with regard to these aptitude measures.
Random Selection and Assignment
Using the CEXP scores obtained from initial screening, a stratified random
sampling technique was used. The resulting sample's distribution of computer
experience was similar to the distribution in the population pool. Using this approach,
60 students were initially selected from the volunteer student population to form a
representative sample to participate in the study. These students were contacted directly
to confirm their willingness to participate and to schedule their training appointments.
After having scheduled their training sessions, each student was randomly
assigned to one of the two online help formats, text-only help or text-with-motion-video
help. From a randomized list of volunteers, each was alternately assigned to TOH or
TMVH to ensure equal treatment group sizes. Random assignment to treatment mini-
mized initial differences between treatment groups. Mortality effects (students who
scheduled training sessions but failed to appear) during the study caused treatment
group sizes to become unequal. In this sample (N = 38), 18 were assigned to the TOH
treatment, while 20 were assigned to the TMVH treatment. The analyses performed
accommodated for this unbalanced design.
Instruments and methods used to measure cognitive style (field dependence-
independence), prior computer experience, and computer task performance were central
aspects of the design of this study. The instruments chosen from published sources or
developed for this study to assess these individual characteristics are described in this
Volunteer Sign-up Form
129 students were initially screened using the Sign-up Form for OS/2 Training
(Appendix A). Responses to questions on this form provided student identification data
such as name, phone number and academic status. It also included eight questions
designed to estimate the students' computer experience. This data was used when stu-
dents were contacted to schedule their individual training sessions. Students who
indicated very little experience were scheduled for longer training appointments.
Finally, the form was signed by students as an indication of their interest in voluntarily
participating in the computer training and study.
Computer Experience and Competence Survey (CECS)
During initial screening, subjects completed the CECS instrument, a multiple-
choice, self-reporting survey of prior computer use. This survey was composed of three
scales: user demographics (CDEM, 8 items), experience survey (CEXP, 39 items), and
cognitive items (CCOG, 47 items). The experience scale items were designed to mea-
sure the amount and types of prior computer experience. The competence scale
includes items designed to measure knowledge of computer systems and applications
and how they operate. This instrument, like the GEFT, was designed to be conducted
in a group setting such as a classroom. The 94-item CECS instrument was timed to be
completed within 30 minutes. Sample items from the CECS are included here in
The CECS items were derived primarily from items developed for the Computer
Competency test of the 1986 National Assessment of Educational Progress (NAEP)
(Educational Testing Service, 1988). For many items, the original NAEP versions were
used verbatim. Some items were edited to account for differences in the target popula-
tion (university students versus secondary school students). Other items were modified
to accommodate the significant changes in personal computers during the six years
between the development of the NAEP items and this study. NAEP items designed to
measure computer programming knowledge were not included because those items did
not appear to relate closely to knowledge of or ability to operate computer applications.
The resulting CECS instrument was designed to determine the extent of a subject's prior
computer experience and knowledge of computer systems, applications, and their
Due to reliability and validity problems found in the CCOG (cognitive knowl-
edge) scale during the pilot study, this scale was not used in the data analysis. Only
scores from the CEXP (experience) scale were used in the study.
CEXP ecological validity estimate. The subjects' initial computer experience
was also measured using a three-task computer operations pretest. The scores on the
pretest were expected to correlate strongly in a positive direction with the subjects'
CEXP scores. The pilot study had confirmed this strong positive correlation (r = 0.85),
which establishes a reasonable measure of ecological validity for the CEXP scale.
CEXP reliability estimate. An analysis of the computer experience scale
revealed problems with several items. First, items 17 and 18 were not included in the
reliability estimate because they were not calculated into the CEXP score (these two
items provide qualitative data only). In addition, items 43 through 47 formed another
qualitatively scored component of CEXP and were also excluded from the reliability
computation. The resulting CEXP scale included 32 items, but initially showed only
moderate reliability (a = 0.61).
A procedure to increase the reliability of the CEXP scale was used, based on a
routine incorporated into the reliability procedure of the Statistical Package for the
Social Sciences (SPSS), version 4.0. This routine used an iterative approach to increase
reliability by removing individual items. As each item was removed, alpha was recal-
culated for the remaining items. All items were finally ranked in the order of
decreasing negative influence on alpha. Using this approach, Chronbach's alpha for the
CEXP scale was improved to 0.85 by removing nine items (listed in order of their
removal: 42, 30, 12, 11, 40, 38, 10, 31, and 39). These items had the greatest negative
influence on alpha. The final CEXP scale therefore had 23 scored items with improved
reliability. The CEXP scores with improved reliability are used throughout the
remaining data analysis and discussion.
Improving the reliability of the CEXP subscale increased the power of the
analyses of covariance where CEXP was used as a covariate. More consistent determi-
nation of computer experience among subjects within the sample allowed more accurate
analysis of regression slopes. Removal of the nine items did not significantly affect the
correlation between CEXP and posttest scores (r = 0.63). The initially low reliability of
the CEXP scale was partially explained by the variety of item types within the scale.
The improved reliability CEXP scores adjusted for response inconsistencies by remov-
ing those items that least contributed to the instrument's reliability.
CEXP scores obtained. The population mean CEXP score was 30.0 (SD = 10.9).
CEXP scores appeared normally distributed, ranging from 9 to 62. For the sample
(N = 38) attending training sessions, CEXP scores ranged from 9 to 58, with a group
mean of 31.4 (SD = 11.0).
Group Embedded Figures Test (GEFT)
All subjects who attended training sessions completed the Group Embedded
Figures Test. GEFT scores were used to measure the subjects' cognitive style (degree
of field independence). Scores on the group-administered scale ranged from 0 (extreme
field dependence) to 18 (extreme field independence).
The GEFT is a widely used and extensively normed instrument that measures
field dependence-independence, one dimension of an individual's cognitive style. It is a
20 minute test consisting of 18 items presented in three individually timed sections.
Scores for college-aged men have a mean of 12.0 (SD = 4.1) and 10.8 for women
(SD = 4.2) (Cavaiani, 1989). Because the range of scores is continuous between 0 and
18, it is considered incorrect to label a subject "field-independent" or "field-dependent"
(Witkin et al., 1977). Using two parallel forms of the GEFT with identical time limits,
an internal consistency reliability estimate of 0.82 (for both males and females) was
obtained (Mykytyn, 1989). Validity measures based on comparing results on the GEFT
to results on the EFT (an individually administered form of the GEFT) show correla-
tions of 0.62 for females and 0.82 for males (Cavaiani, 1989).
GEFT scores obtained for this sample (N = 38) ranged from 2 to 18, with a
group mean of 12.4 (SD = 4.1). As expected, this sample distribution was skewed
revealing a larger percentage of more field-independent individuals than would be
found in a general public sample of same-aged subjects (Witkin et al., 1977). This
sample distribution is similar to that reported for other college student populations
(Cavaiani, 1989; Witkin et al., 1971).
Task Performance Assessment
Measurements of computer task performance were taken for each subject in the
training pretest and posttest. Each test was comprised of three tasks. A performance
score was calculated for each task and the sum of the scores for the three tasks in each
test formed the total test performance score. The computer task performance score
reflected the accuracy and rate of task completion and was calculated using Equation 6.
PS =------ P DW (6)
PS = Performance score
ET = Expected time to task completion
AT = Actual time to task completion
P = Percentage of task completed
DW = Difficulty weighting factor
This method of scoring performance approximately followed the calculation
used by Whiteside et al. (1985). This score quantified users' performance on tasks
taking into account the rate of task completion and the competence demonstrated on the
task. This formula normalized time across all tasks by dividing expected task
completion time (10 minutes) by the actual task completion time. In addition, task dif-
ficulty was taken into account by using a difficulty weighting factor. This factor
accounts for increasing task difficulty across a series of tasks.
Expected time to task completion (ET). This was a constant (10 minutes),
determined by taking the average time subjects required for completing the tasks, then
adding 20%. This value was initially based on the researcher's estimate. The final
value was established based on task performance data gathered during the pilot study.
This also served as the time limit allowed for attempting to complete a task. This time
limit prevented subjects from spending too much time attempting a task and helped
limit their frustration when no progress was being made, particularly during the pretest.
Actual time to task completion (AT). This value was computed for each task by
measuring the total task completion time, beginning to end. Subjects were observed
during task completion and their computer interactions were logged by an observer.
Regardless whether the subject was continually on-task or not, total time to task
completion was used. Subjects remained seated at the computer work station during
tasks. The work station environment was controlled to minimize distractions and sub-
jects were not allowed to take breaks between tasks during the pretest and posttest.
Also, subjects were not allowed to discuss their performance or other aspects of the
study until after they had completed all lessons and tests.
Percentage of task completion (P). This value reflected the degree of success on
the task with respect to predetermined mastery criteria. Each task was composed of five
subtasks. For every subtask completed correctly, P was increased 20%. For example, if
a subject completed only four of five subtasks, that task was scored 80% complete.
Subjects received P = 1.0 for every fully completed task.
Task difficulty weighting factor (DW). This value represented approximate task
difficulty, which was operationally defined as the minimum number of discrete opera-
tions in the user interface required to complete the task. Examples of discrete
operations included moving the mouse pointer (cursor) to an object, clicking or
double-clicking with the mouse button on an object, or entering keystrokes (multiple
keystrokes for a single entry were counted as one discrete operation). Three DW levels
were identified to organize the 12 lessons and to design representative tasks for the pre-
test and posttest. Low difficulty tasks required 1 to 3 discrete operations per subtask,
with a total of 8 to 9 per task. Medium difficulty tasks required 2 to 5 discrete opera-
tions per subtask, with a total of 15 to 20 per task. High difficulty tasks required 3 to 7
discrete operations per subtask, with a total of 20 to 25 per task.
Three test tasks, one task from each difficulty level, comprised the training pre-
test and posttest. The 12 training lessons were designed with gradually increasing
difficulty, with four lessons at each of these three difficulty levels. The pretest and
posttest tasks were designed as equivalent forms, so each task in the pretest had the
same DW value as the corresponding task in the posttest.
By adjusting for task difficulty when computing the task performance score, a
subject's competence was more accurately recorded. Subjects were expected to gain
competence as they proceeded from easy to more difficult tasks. Therefore, more dif-
ficult tasks were weighted more heavily than the easier tasks.
The instruction provided each subject was individually delivered as a short
computer-based training course. After screening and aptitude assessment, the instruc-
tion consisted of an introductory videotape, followed by a computer-based interactive
tutorial, followed by a pretest, 12 training lessons, and a posttest. The experimental
treatment (online help format) was administered during the 12 training lessons. Sub-
jects were randomly assigned to one of two treatment groups: (a) text-only help (TOH)
or (b) text-with-motion-video help (TMVH). These two online help formats were simi-
lar in all but one respect. In the TMVH group, dynamic pictorial message elements
(digital motion video windows) were added to text messages in online help, while the
TOH group had identical text help messages without the dynamic pictorial elements.
The computer training sessions were conducted at an International Business
Machines Corporation facility in Boca Raton, Florida. The setting was a product
usability evaluation center that was ideal for this type of study. Individual subjects
were seated in simulated offices--rooms equipped with computer systems, desks, tables,
and other accessories typically found in corporate offices. Each subject completed the
training in a single session, scheduled during normal business hours or at night, accord-
ing to the student's preference. During the training each subject was situated beyond
sight and hearing of others, so that he or she would not be distracted while completing
Subjects completed the training using a microcomputer equipped with a high-
resolution color display and dot-matrix printer. A standard keyboard and mouse were
also attached as user input devices. The software installed consisted of IBM Operating
System/2 version 2.0 (OS/2), an advanced personal computer operating system with a
graphical user interface (GUI). The OS/2 GUI incorporated direct-manipulation,
object-oriented control features whereby application functions were controlled by
manipulating graphical features with the mouse pointer, rather than relying solely on
verbal menus or command-line interfaces for function selection and activation. User
interaction with the system and the graphical spreadsheet application required manipu-
lation of appropriate icons, buttons, dialog panels, windows, pull-down menus, and
other user interface controls.
During the training, subjects were monitored by an observer situated immedi-
ately outside the simulated office. The observer could view the office interior through a
one-way mirror or through remote-controlled color video cameras. The display of the
computer used by the subject was attached to a second color display at the observer's
console, allowing the observer to closely follow the subject's progress on each task. To
help minimize the level of anxiety a subject might experience from being observed,
subjects were not shown the observer's console until after the training session was
One of two observers was randomly assigned to monitor each subject. Both
observers had conducted observations during the pilot study so they were familiar with
the training protocol and comfortable with the monitoring procedures. As the observers'
role included rating subjects on task performance, inter-rater reliability was a concern.
A series oft tests were run, grouping subjects by observer, to determine whether dif-
ferences existed that were related to the observer assigned. Dependent variables
examined included pretest and posttest scores, and time in help. No significant differ-
ences attributable to observer assignment were detected.
The computer training protocol was delivered in five stages: orientation, intro-
ductory video tape and tutorial, pretest, lessons, and posttest. A sample of 42 subjects
attended individual training sessions at the corporate product evaluation center. Most
training sessions were held during normal business hours although several were held in
the evening. The training periods were scheduled to last three to six hours. Actual
training times ranged from 2.6 to 6.8 hours, with a mean training time of 4.1 hours.
Orientation. First, subjects were escorted into the product evaluation facility
housing the simulated office where they remained throughout the training. The training
observer then verbally presented an overview of the purpose of the facility and the
computer training lessons. Subjects then reviewed and signed informed consent forms
and nondisclosure agreements.
Introductory video tape and tutorial. The introductory video tape, Working With
OS/2 Version 2 3, provided the students with a general overview of features in the
operating system and its graphical user interface. This 40-minute video tape demon-
strated use of system features and defined the several different object types in the
system GUI. It provided illustrative scenarios for using OS/2 applications, gave non-
interactive instruction to the subjects, and augmented the instruction provided by the
OS/2 system tutorial.
3 Working with OS/2 Version 2 is copyright 1992 by Comsell, Inc.
Each subject was then seated at the microcomputer to complete the interactive
tutorial. The system tutorial program was started and ready to use when the subject was
seated. Subjects were allowed as much time as needed to complete the tutorial. This
typically required 30 to 40 minutes. The system tutorial provided practice using basic
and slightly more advanced operations and functions of the system, focused on using
the mouse to directly manipulate objects in the GUI. Observer assistance to subjects
was not provided after beginning the tutorial, except in a few cases of software failures
that required observer intervention.
Training pretest. A training pretest was given immediately following completion
of the system tutorial, with the subject operating the computer. The pretest included
three pretest tasks, designed with increasing levels of difficulty, as described previously.
The five subtasks in each pretest task were selected from subtasks found in the training
lessons. Thus the pretest accurately reflects the content and design of the lessons, mea-
suring performance on operations the subject was expected to learn during training.
Subjects were allowed up to 10 minutes to complete each pretest task. Many subjects
completed the tasks in less time while others were unable to complete the tasks in the
allowed time. Tasks in the pretest were sequenced so that subsequent tasks could be
started without requiring the preceding task to be completed. The pretest tasks were
scored using the task performance formula (Equation 6) and the sum of the three pretest
task scores was used as the pretest performance score.
Training lessons. After the pretest, subjects completed a series of 12 lessons,
during which one of the two online help formats, TOH or TMVH, was encountered.
The subjects were given a tersely worded task description as they began each lesson and
were encouraged to use online help whenever they experienced difficulty or were
unable to proceed. Although discouraged from requesting assistance from the observer,
when such assistance was requested the subject was directed to use online help to learn
more about that particular operation. The instructional materials included the printed
task description for each lesson, plus the online help messages. Because each subject
determined the extent to which he or she would use online help in a given lesson, the
amount of instructional information viewed during training varied considerably between
subjects. Use of online help was tracked automatically by system software.
Subjects were given as much time as necessary to complete each lesson. How-
ever, if an impasse was reached where no progress was made for five minutes, the
subject was prompted by the observer to access help for a specified help topic. If the
subject completed a lesson satisfactorily, the next lesson description was immediately
provided. If an entire lesson or portions of the lesson were not completed correctly, the
errors were logged and the subject was allowed to proceed to the next task. Lessons
were structured to facilitate smooth transitions from one lesson to the next.
When subjects reached an obvious impasse, the observer redirected the subject to
first reread the task and subtask directions. Occasionally subjects required help inter-
preting the directions. If the subject was still unable to proceed, the observer then
prompted the subject to open the online help facility. When the subject could not locate
an appropriate topic, the observer directed the subject to the specific help topic for that
situation. Observers did not directly instruct subjects with procedures for task comple-
The 12 training lessons were designed with gradually increasing difficulty (four
lessons each in the low difficulty, medium difficulty, and high difficulty groups). The
series of 12 lessons had three objectives: First, it provided practice in elementary skills
for operating the system and its graphical user interface. Second, it provided practice
using the online help facilities. Third, it provided training in the use of the PM Chart
graphical spreadsheet application.
Each lesson included a task description, composed of five subtasks or steps.
Lessons 1 to 3 covered system tasks in the user interface, such as creating a new folder,
copying several files from a diskette into a folder, and moving the PM Chart program
icon into the folder. Application tasks (Lessons 4 to 12) included modifying an existing
file using PM Chart features, loading spreadsheet data, and creating several presentation
The lessons were also arranged as a series of tasks within a meta-task, so there
was a clear project goal the subject could perceive as each lesson was completed. The
directions for the 12 computer training lessons are included in Appendix C.
Training posttest. The posttest was administered immediately following
completion of the 12 training lessons. The three posttest tasks were equivalent, but not
identical, to the three pretest tasks. The posttest scores were used as the primary mea-
sure of learning outcomes in this study.
Gain scores (the difference between a subject's performance on the pretest and
posttest) were calculated for each subject. Although commonly applied in educational
research, use of gain scores has been criticized on the basis of generally poor reliability
(Stevens, 1990). Specifically, when the correlation of pretest and posttest scores
approaches the reliability of the test, the reliability of gain scores goes to zero. For this
reason, gain scores were not used as dependent measures in the analysis.
Experimental Instructional Variable
The two treatment conditions (online help formats) were identical except for one
variable: the presence of dynamic pictorial elements in the spreadsheet application
online help messages. Online help provided instruction for subjects as they attempted
to learn application functions during lessons 4 to 12. The two online help formats are
described below, along with characteristics of online help common to both treatments.
General online help characteristics. The online help facility in the system, the
information presentation facility, presented help messages displayed within windows
adjacent to or overlapping the application windows. The help messages consisted of
text formatted as paragraphs, sentences, and lists. Dynamic pictorial elements (digital
motion video sequences) could be displayed, in addition to the text content, in the
experimental treatment condition. Direct-manipulation window controls were provided
so the help window could be moved, resized, and closed at any time at the discretion of
the user. Figure 3-1 illustrates the OS/2 application help window interface for the
graphical spreadsheet application used in this study.
All help windows incorporated standard controls that allowed the user to access
additional functions, such as printing a help topic, viewing the help index, and moving
forward or backward through selected help topics. The sequence of information dis-
played at any time in the help window was controlled in an interactive manner. The
user could select a help topic and then change the topic at any time. Help messages
often displayed related topic labels, called links. Links appeared as text displayed in a
different color (green) than standard help text (blue). The related topics could thus
?! S--?_ __ Ii '
Select the Worksheet tool button on the o
display Ihe worknseel window Use Ihe worksheel
window to enter or change aalda i credlting charls
Worksneet data may De Imported from Iles created
by PM Chart and various olher spreadsneel
O Iw' ll ,i r .1 'liil j i ..
0 N ;. -1 111, i ll -1,
0 1, .1 1iC *I- 1 i i 1
1Pr lWtFi I rJ I'I r index I LIt j
Figure 3-1. OS/2 help window interface used in PM Chart, showing video button.
easily be accessed by double-clicking on the desired link. The selected help topic
would then be displayed in the help window, overlaying the previous help topic in the
help window. The user could also interact with the application at any time without
closing the help window. When in use, the entire help window remained visible, occa-
sionally covering a small portion of the application window.
The information presentation facility in this system provided a hypertext
implementation that supported selective access to information structured in a nonlinear
manner. Subjects were able to select a series of related help topics, when desired, sim-
ply by repeatedly selecting links in the displayed help windows. When the desired help
information had been viewed, the subjects closed the help window and returned to the
application to complete the task at hand.
Text-only help (TOH) treatment. This treatment provided instruction in online
help messages that included only text (verbal-digital) message elements relating to the
selected help topic. No graphical or pictorial message elements were included in these
help messages. The text contained in the help topics was carefully edited to fully
describe application functions. In many help messages, the text extended beyond the
window borders. For these messages, the user had to scroll within the help window to
read the complete help topic text. Subjects' use of online help was automatically
tracked using system software to record the total time in help and the topics displayed.
Text-with-motion-video help (TMVH) treatment. This online help format
incorporated the same text messages as the TOH treatment. In addition, a Video button
was added to the help window controls. An example of a help window showing the
location of the Video button is shown in Figure 3-1. When a subject selected the Video
button, a motion video sequence was displayed in a window overlaying the previously
visible help text. This overlay technique was identical to how related help topic text
windows appeared whenever text links were selected. When a motion video sequence
had ended, the final video frame remained visible as a static image until that window
was closed. Display of these video sequences by subjects was tracked in the same
manner as the display of text help topics.
The digital motion video playback facilities employed in this study included
Digital Video Interactive 4 (DVI) hardware and software features. Additional custom-
ized software was developed to provide an interface between the OS/2 information
presentation facility and DVI. DVI playback at 12 frames-per-second was used to
present the dynamic pictorial sequences that visually portrayed application functions
described in the help text. Each video sequence, lasting from 15 to 40 seconds, was
designed to match a specific help topic's text message. Each video sequence was pro-
duced using an 8mm video camera aimed directly at an active high resolution computer
display. The camera's S-video output signal was connected directly into the DVI cap-
ture adapter input, so that the camera output could be captured without loss of signal
quality. The camera remained stationary during each sequence, usually tightly cropping
the PM Chart application window being manipulated according to the accompanying
help text. This approach follows from instructional design principles based on dual-
coding theory (Fleming & Levie, 1978).
Although professional grade video equipment was used to develop the video
sequences, conversion to the digital display format in this system induced some loss of
visual detail in the image. Several subjects commented that the images were "blurry"
but maintained they could understand and follow the video sequences. Visual quality of
the digital video sequences also was reduced by limiting the size of the playback win-
dow to the help topic window size. The video images thus displayed showed
application features smaller than they appeared in the application interface itself.
Despite these visual quality issues, subjects assigned to the TMVH treatment frequently
stated their preference for the video help format.
4 Digital Video Interactive and DVI are trademarks of Intel Corporation.
Data was gathered in several ways for this study. Scores on the aptitude mea-
sures of interest were obtained by hand-scoring the test answer forms for the GEFT and
CECS instruments. During training sessions observers manually logged the actions of
subjects, while at the same time the computer automatically logged online help activity.
These techniques are described below.
Automated Online Help Tracking
Subjects were instructed to use online help whenever they were uncertain of how
to proceed with a task. Instructions for using online help were repeated several times
throughout the training lessons. Custom computer software was used to automatically
collect data on the use of online help. This software consisted of a help tracking pro-
gram that automatically created a log file for each subject containing accurate timing
data on the subject's use of online help. Each time a help topic was opened, the time in
help for that topic was recorded in the log file. The total time in help (THELP) was
calculated for each subject and used as a dependent variable in the analyses of
covariance described below.
The help tracking program ran in the background (not visible to the student) as
the subject completed the training lessons. For each help topic opened, the log con-
tained the topic name, the time the topic window was opened and closed, and the
elapsed time for each topic. Total time in help was computed as the sum of the elapsed
times. For students in the TMVH treatment, the use of video help segments was also
recorded. The elapsed time for the video segments opened by the student was included
in the total time in help. A sample help tracking log is included in Appendix D.
Because help windows could remain open while a subject was interacting with
the application, the observer also kept records of help usage. The observer was respon-
sible to log any occasion where a subject left a help window open when interacting with
the application. Only two of the 38 subjects had used help in this manner. For these
subjects, their time in help scores were adjusted to accurately reflect when they were
using help and when they were interacting with the application.
Observer Event Logging
Subjects were observed by the researcher, who sat at a control panel in an adja-
cent room and recorded significant events in a computer database. During the training,
the observer continually updated the database by adding records of actions taken by the
subject and, occasionally, by the observer. Each entry in the log was automatically
time-stamped to facilitate accurate timing of the subject's actions. For the three tasks in
the pretest and posttest, log entries indicated success or failure for each subtask and how
long it took to complete each task. The log files were later examined to calculate the
pretest and posttest performance scores. A sample event log is included in Appendix E.
This experimental study of learning computer operations in graphical human-
computer interfaces was designed to test the effects of university students' cognitive
styles (field dependence-independence) and prior computer experience on their perfor-
mance on tasks in a direct-manipulation, graphical user interface. The population pool
completed assessments of field dependence-independence and prior computer experi-
ence. Volunteer subjects randomly selected from the population were scheduled for
individual training sessions. The subjects were randomly assigned to one of two treat-
ment conditions that differed only with regard to presence of dynamic pictorial elements
(digital motion video) in the online help messages.
After an introductory videotape and an interactive computer-based tutorial, all
subjects then completed an initial computer operations pretest to establish a perfor-
mance baseline. The subjects completed a series of 12 lessons comprising tasks using
the computer system's graphical user interface and a graphical spreadsheet application.
Task performance was again measured in a posttest at the completion of the training
lessons. A completely randomized design with pretest-treatment-posttest sequence was
employed in this study. A multiple covariance analysis was used to determine whether
ATI effects occurred between field independence and treatment, or between computer
experience and treatment. The ANCOVA was also used to detect effects of the two
aptitude measures on performance. The ANCOVA techniques were further employed to
control for between subjects variance on time in help. The results of the study are
described in the following chapter.
RESULTS AND ANALYSIS
This study was conducted to determine whether field dependence-independence
or level of computer experience influenced computer users' performance on application
tasks in a direct-manipulation graphical user interface (GUI). Other research questions
addressed in this research concerned whether Aptitude x Treatment interactions
occurred between field independence or computer experience and the presence of
dynamic pictorial message elements displayed in online help. A random sample of 38
university student volunteers attended individual computer-based training sessions. The
students were randomly assigned to one of two treatment groups using different online
help formats. Both treatment conditions required subjects to use online help to obtain
instruction for learning application functions. Text-only help was provided in one
treatment level, while in the other level dynamic pictorial content (digital motion video)
was displayed in addition to help text.
A fully randomized design with pretest-treatment-posttest sequence was used in
the study. Data were analyzed using a multiple covariance analysis (Huitema, 1980).
Field independence and computer expertise were employed as covariates in the
ANCOVA. Time in help was also included as a covariate to control for between sub-
jects variance on exposure to help messages. The dependent measure was performance
on application tasks in the training posttest.
No significant interaction effects were found between any of the three covariates
and the online help format. In addition, no significant effect was found for the online
help format. Significant regression effects were found for both computer experience
and field independence on application task performance. Increased prior computer
experience and increased field independence were significantly related to improvements
in application task performance on the posttest. No significant regression effect was
found for time in help. These results are described in detail in this chapter.
Data were collected during the experiment and analyzed as described in the
preceding chapter. This section presents the ANCOVA results obtained using the Sta-
tistical Analysis System (SAS), release 6.07.
The one-way ANCOVA model for this analysis included the treatment variable
and three covariates. The treatment factor (TREAT) consisted of two levels, text-only
help (TOH) and text-with-motion-video help (TMVH). The covariates included com-
puter experience (CEXP), field dependence-independence (FDI), and time in help
(THELP). The single dependent variable was application task posttest performance
(POST). The 38 students in the sample were randomly assigned to one of the two
treatment groups. There were 18 subjects in the TOH group and 20 subjects in the
TMVH group. The results of the ANCOVA are described below, beginning with a
review of the null hypotheses tested.
Treatment x Covariate Interaction Effects
First, the ANCOVA assumption of homogeneous regression slopes was tested.
This was also a test for interactions between the covariates and the treatment factor.
Therefore, the test for homogeneous slopes tested the following null hypotheses
regarding interactions: Hypothesis 1, that no significant differences in application task
performance would result from a three-way interaction among field dependence-
independence, prior computer experience, and the presence of dynamic pictorial
message content in online help; Hypothesis 2, that no significant differences in
application task performance would result from an interaction between prior computer
experience and the presence of dynamic pictorial message content in online help; and
Hypothesis 3, that no significant differences in application task performance would
result from an interaction between field dependence-independence and the presence of
dynamic pictorial message content in online help.
As described in Chapter 3, the test for homogeneous slopes required computing
the error sums of squares for two linear regression models, referred to as the complete
and reduced ANCOVA models. Summary source tables for the complete and reduced
ANCOVA models are given in Tables 4-1 and 4-2. The F statistic to test the assump-
tion of homogeneous regression slopes was calculated using the error sums of squares
for these two models. The resulting test statistic, F (3, 30) = 1.14,2 > .05, did not
reach significance. The assumption of homogeneous regression slopes had been met.
Therefore, the null hypotheses regarding interactions between the covariates and the
treatment factor (Hypotheses 1, 2 and 3) were not rejected. No significant interaction
effects on posttest performance were detected between online help format and computer
experience, field dependence-independence, or time in help.
Since the assumption of homogeneous slopes was valid for this analysis, the next
step in the ANCOVA was to determine whether treatment differences had a significant
effect on posttest performance. This tested Hypothesis 4, that no significant differences
in performance on computer application tasks would exist between subjects viewing
text-only online help and subjects viewing online help containing text and dynamic pic-
Testing the treatment effect required computation of a third regression model,
the reduced ANCOVA model without the treatment effect. The summary table shown
in Table 4-3 identifies the sources of variance for the reduced model with the treatment
effect removed. This model determined the regression effects for the three covariates,
assuming the treatment variable had no effect.
Table 4-1 Summary Table for Complete ANCOVA Model Effects on Posttest Scores
Source df Type III S F Pr >
TREAT 1 12.02 0.01 0.9394
CEXP 1 26451.12 12.94 0.0011
FDI 1 13391.26 6.55 0.0158
THELP 1 6608.32 3.23 0.0822
CEXP*TREAT 1 4896.28 2.40 0.1322
FDI*TREAT 1 2816.04 1.38 0.2497
THELP*TREAT 1 509.35 0.25 0.6213
Table 4-2 Summary Table for Reduced ANCOVA Model With Treatment Factor
Source df SS E Pr>F
Model 4 82449.42 9.96 0.0001
Error 33 68306.72
Corrected Total 37 150756.13
Source df Type IIISS F Pr > F
CEXP 1 27063.47 13.07 0.0010
FDI 1 11437.64 5.53 0.0249
THELP 1 6682.56 3.23 0.0815
TREAT 1 1266.57 0.61 0.4397
The test of significant treatment effect was performed by calculating the appro-
priate F statistic, supplying the error sum of squares for the reduced model without the
treatment effect (from Table 4-3), and the error sum of squares for the reduced model
including the treatment effect (from Table 4-2).
Table 4-3 Summary Table for Reduced Model Without Treatment Factor
Source df SS F Pr > F
Model 3 81182.84 13.22 0.0001
Error 34 69573.29
Corrected Total 37 150756.13
Source df Type III SS F Pr > F
CEXP 1 26849.40 13.12 0.0009
FDI 1 10192.35 4.98 0.0323
THELP 1 8541.05 4.17 0.0489
The resulting test statistic, F (1, 33) = 0.61, 2 > .05, did not reach significance.
Therefore, Hypothesis 4 was not rejected. This test statistically controlled for the
between-subjects variance on the covariates and determined that the adjusted group
means on POST between the two online help treatments were not significantly different.
The addition of dynamic pictorial elements to textual online help in this study did not
significantly effect performance on application tasks in the unfamiliar GUI.
After determining that the interaction and treatment effects were not significant,
the analysis proceeded to test for significance of regression effects for each of the
covariates. These effects were determined using the Type III sums of squares found in
Table 4-2. This table shows the corresponding F statistics and probabilities for the
regression effects of the covariates CEXP, FDI, and THELP.
First, the a priori assumption that time in help would have no significant effect
on performance after controlling for differences on FDI and CEXP was upheld. The
test statistic for regression of POST on THELP, F (1, 37) = 3.23, 2 > .05, did not reach
significance. Increasing use of help, measured as the total time a user displayed help
messages during training, was not significantly related to performance on posttest tasks.
Next, the regression effects of the aptitude variables, CEXP and FDI, were
examined. The appropriate E statistics were examined to test the research hypotheses.
Hypothesis 5 stated that no significant relationship would exist between prior computer
experience and a computer user's performance on computer application tasks in an
unfamiliar GUI. The test for regression effect of POST on CEXP, E (1,37) = 13.07,
p = 0.001, revealed a significant effect. Therefore, the null hypothesis was rejected.
Prior computer experience, as measured using the Computer Experience and Compe-
tence Survey, was significantly related to performance on the posttest tasks. As the
level of prior computer experience increased, performance on application tasks was
found to improve.
Hypothesis 6 stated that no significant relationship would exist between field
dependence-independence and a computer user's performance on computer application
tasks in an unfamiliar GUI. The test statistic for regression of POST on FDI,
F (1, 37) = 5.53, g = 0.025, showed a significant effect. Therefore the null hypothesis
was rejected. Field dependence-independence, as measured using the Group Embedded
Figures Test, was significantly related to performance on the application posttest tasks.
As field independence increased, there was a significant tendency for performance on
application tasks to improve.
In this analysis, significant regression effects were found for computer experi-
ence and field dependence-independence. No significant treatment effect was found for
adding dynamic pictorial elements to online help displays. In addition, no significant
Aptitude x Treatment interactions were found. The following analysis of these results
examines the significant regression effects as well as the character of the nonsignificant
interaction and treatment effects.
The multiple covariance analysis found no significant interaction effects between
online help format and any of the covariates. Also, no significant main effect was
found for online help format. Significant regression effects were found for both
computer experience and field dependence-independence. This section presents an
analysis of the regression slopes for the ANCOVA model as they were examined at
each step in the procedure. This analysis begins with a review of the tests performed to
verify that assumptions for ANCOVA had been met.
Testing ANCOVA Assumptions
Homogeneity of variance. To test this assumption, I tests were performed to
determine whether the mean scores and within-group variance for CEXP, FDI, and
THELP were significantly different between the two treatment levels. No significant
differences (p > .05) were found between the treatment group means. Tests of unequal
group variance for CEXP and THELP did not reach significance. A test for unequal
group variance did reach significance (2 = .04) for the FDI scores. However, homoge-
neity of variance is not required when the covariate is statistically independent of the
treatment (Huitema, 1980). Since FDI scores were obtained prior to the training, this
assumption was met for this analysis.
Independence of treatment and covariates. Since computer experience and field
dependence-independence were measured prior to the instructional treatment, these
covariates were measured independently. Time in help (THELP) was measured during
the 12 training lessons when the online help displays were being used. An ANOVA on
THELP was computed to verify that it was independent of the online help format. The
resulting test statistic, F = 1.21, p = 0.2796, did not reach significance. There was no
significant effect of treatment level on time in help, so this assumption was also met.
Homogeneity of regression slopes. This assumption was tested in conjunction
with the test of hypotheses concerning interaction between the treatment factor and the
covariates. No significant Covariate x Treatment interaction effects were found, so this
assumption was valid. Additional analysis was performed concerning between-group
regression slope differences, as described in the following section.
Covariate x Treatment Interactions
There were no significant Covariate x Treatment interactions. Although these
interaction effects did not reach significance, the regression slopes for the two treatment
groups were plotted. Cronbach and Snow (1977) recommended that even nonsignifi-
cant interactions should be examined, particularly when the number of subjects in each
treatment group is much smaller than 100. In taking this position they stated: "Consis-
tent nonsigificant results are at least as valuable to a science as are incoherent
significant results" (p. 53).
The complete ANCOVA model (see Equation 1 in Chapter 3) includes a
Y-intercept parameter (a), regression slope parameters (Pi) for each covariate (X,),
cross-product term coefficients (y), and a Y-intercept difference parameter (8) for the
effect of treatment (A). Since there were only two treatment groups, only one 6 is
required for this model. The estimated values of these regression equation parameters
as computed by the SAS GLM procedure are shown in Table 4-4. These regression
equation parameters were used to plot regression lines to illustrate the nonsignificant
ANCOVA interactions. Regression line pairs for each covariate were plotted in three
separate two-dimensional graphs. These illustrations allow visual inspection of the
nonsignificant two-way interaction effects. The nonsignificant three-way interaction
(CEXP x FDI x TREAT) was illustrated by plotting two regression planes in a three-
The nonsignificant interaction effects between treatment (online help format)
and the three covariates are depicted in Figures 4-1 to 4-3. The regression of POST on
computer experience for both levels of online help treatment is shown in Figure 4-1.
The regression of POST on field dependence-independence for the two treatment con-
ditions is depicted in Figure 4-2. The regression of POST on time in help for both
treatment levels is shown in Figure 4-3. The regression planes formed by the intersec-
tion of CEXP and FDI regression slopes are shown in a three-dimensional graph
Table 4-4 Parameter Estimates for Complete ANCOVA Model Regression Effects
Y-Intercept (a) 21.81
FDI*TREAT (7,) -5.03
FDI (P1) 8.00
CEXP*TREAT (y,) 2.32
CEXP (12) 1.54
THELP*TREAT (y,) 0.93
THELP (03) -2.14
TREAT (6) -6.02
in Figure 4-4. This three-dimensional illustration permits a visual inspection of the
nonsignificant three-way interaction between computer experience, field dependence-
independence, and online help format.
As illustrated in Figure 4-1, the nonsignificant interaction between computer
experience and online help format shows a trend for subjects in the text-only help con-
dition to perform better than subjects in the text-with-motion-video help condition. The
between-groups performance difference tends to increase with increasing prior experi-
ence. This nonsignificant ATI effect reflects that in this study text help with dynamic
pictorial elements were somewhat less helpful than text-only help, particularly for indi-
viduals who had more extensive computer experience.
The nonsignificant trend toward an interaction between field dependence-
independence and online help format is shown in Figure 4-2. Visual inspection of the
regression lines for the two treatment groups reveals that as field independence
increased, subjects in the text-with-motion-video online help format tended to score
250 I--- -------------.----.-i^ | -TO H-
1..0-.---------------- -- -
0 10 20 30 40 50 60
Figure 4-1. Nonsignificant CEXP x TREAT interaction.
higher on the posttest than highly field-independent subjects in the text-only group.
While this interaction effect did not achieve significance, the trend indicates that in this
study the performance of extremely field-independent subjects was higher when
dynamic pictorial presentations appeared in the online help messages. The task per-
formance of the most field-dependent individuals, on the other hand, appeared to be the
same regardless of the online help format used.
As shown in Figure 4-3, the effect of time in help was very small (the slopes of
the regression lines are approximately zero) and the slopes of the regression lines for
the two treatment levels were nearly identical. There was no clear indication of a trend
toward an interaction between time in help and online help format. Regardless of how
much time subjects spent using help, the posttest performance score difference between
the two treatment groups remained very small.
Figure 4-2. Nonsignificant FDI x TREAT interaction.
200 -- --- ---------- ..... -...-..
1 5 0 ......... ... .........
-50 ...-........................... ................
0 10 20 30 40
Figure 4-3. Nonsignificant THELP x TREAT interaction.
0 2 4 6 8 10 12 14 16 18
The nonsignificant three-way interaction between computer experience, field
dependence-independence, and online help format is depicted in Figure 4-4. This
three-dimensional plot of two regression planes reveals a trend towards an interaction.
The lightly shaded plane, representing the predicted posttest scores for the text-with-
motion-video help (TMVH) treatment group, shows the combined effects of increasing
computer experience and increasing field independence. Among the very field-
independent computer experts (FDI = 18, CEXP = 60) in this group, the predicted
posttest score was 279.8. For extremely field-dependent computer experts (FDI = 0,
CEXP = 60) in the TMVH group, the predicted posttest score was 114.0.
The more heavily shaded plane in Figure 4-4 represents the predicted posttest
scores for the text-only help (TOH) treatment group. In this treatment group, very
field-independent computer experts (FDI = 18, CEXP = 60) had predicted posttest
scores of 300.6, only slightly higher than in the TMVH group. However, for extremely
field-dependent computer experts (FDI = 0, CEXP = 60) the predicted posttest score for
the TOH group was 247.1, much higher than the predicted score for the TMVH group,
114.0. This contrast shows that the presence of dynamic pictorials in online help was
related to a negative effect on performance for highly field-dependent computer experts.
For field-independent computer experts, however, there appeared to be no significant
application task performance difference between the two online help treatment groups.
The two regression planes depicted in Figure 4-4 reveal another aspect of the
trend towards a CEXP x FDI x TREAT interaction. For computer novices with high
field-independence (CEXP = 0, FDI = 18), the predicted posttest score for the TMVH
group was 165.8, while for similar individuals in the TOH group, the predicted score
was 69.3. For field-independent novices, therefore, the presence of dynamic pictorials
was associated with an increase in performance on GUI application tasks.
In the regression plots shown in Figures 4-1 and 4-2, there is visible evidence of
trends towards both a CEXP x TREAT interaction and a FDI x TREAT interaction.
Figure 4-4. Three-dimensional plot of two regression planes showing nonsignificant
CEXP x FDI x TREAT interaction.
In addition, as shown in Figure 4-4, there was visible evidence of a trend towards a
CEXP x FDI x TREAT interaction. However, these Aptitude x Treatment interactions
were nonsignificant. Therefore, these interaction effects were ignored in the remaining
steps of the multiple covariance analysis. In the following discussion of treatment
effects, the regression slopes for the two treatment groups were assumed to be equal.
Since there were no significant interactions and the assumption of homogeneous
regression slopes was valid, the analysis of treatment effects examined the adjusted
posttest performance means for the two treatment groups. The between-groups differ-
ence on POST adjusted means was characterized by the difference on the Y-intercept
(6), calculated using the reduced ANCOVA model. This reduced model included the
covariates and the treatment factor but eliminates the interaction terms.
The regression coefficient estimates calculated using the reduced ANCOVA
model are given in Table 4-5. The between-groups POST adjusted means difference
attributable to the online help treatment is 12.31. The range of scores on POST was
from a minimum of 53.2 to a maximum of 273.7 (D = 63.83). Given this distribution,
the between-groups difference on POST--independent of all covariates--was remarkably
Controlling for the effects of the covariates, the reduced ANCOVA model pro-
duced an estimate of the treatment effect on POST that was nonsignificant at the .05
alpha level. The very small between-groups difference on POST (6 = 12.31, 0.19 SD)
indicated that--independent of computer experience, field dependence-independence,
and time in help--the effect of online help format on application task performance in
this study was negligible.
Covariate Regression Effects
The final step in the ANCOVA hierarchical regression analysis was to examine
the regression of the dependent variable on the covariates. In the prior steps, interaction
and treatment effects were found to be nonsignificant. Removing the interaction and
treatment terms from the ANCOVA model resulted in a multiple regression prediction
equation. The regression coefficient estimates for this equation are given in Table 4-6.
Table 4-5 Parameter Estimates for Reduced ANCOVA Model Regression Effects
with Nonsignificant Treatment
Y-Intercept (a) 23.92
FDI (PI) 4.67
CEXP (P2) 2.72
THELP (P3) -1.67
TREAT (6) 12.31
Using these estimates, predicted posttest performance scores were obtained. The
coefficient of multiple correlation for this regression equation, R = .733, was reasonably
high. About 53.9 percent of the variance on posttest performance was accounted for by
these three covariates.
The regression effects of POST on CEXP, and of POST on FDI, were found to
be significant. The regression slope of POST on CEXP is shown in Figure 4-5.
Examining only the effect of computer experience, the predicted posttest performance
scores ranged from 37.40 (CEXP = 0) to 194.71 (CEXP = 58). The regression effect of
CEXP, expressed in terms of the sample variance on POST, was 2.46 SD.
Table 4-6 Regression Equation Parameter Estimates for Reduced ANCOVA Model
Y-Intercept (a) 37.40
FDI (IP) 4.21
CEXP (p1) 2.71
THELP (13) -1.84
Figure 4-5. Regression effect of posttest task performance on computer experience.
The regression slope of POST on FDI is shown in Figure 4-6. The predicted
POST scores ranged from 37.40 (FDI = 0) to 113.24 (FDI = 18). The regression effect
of FDI, again standardized on the sample variance on POST, was 1.19 SD.
By comparing these regression contributions, or beta weights as they are often
referred to in multiple regression analyses, prior computer experience accounted for
more than twice the posttest variance that was accounted for by field dependence-
independence. Time in help accounted for slightly less posttest score variance, with a
beta weight of only 1.16 SD. With either increasing computer experience or higher
field independence, subjects' posttest task performance improved significantly, regard-
less of online help format. Increased time in help was related to a performance decline
on the posttest. Computer experience accounted for more than twice the effect of field
40 50 60
0 10 20 30
0 --I---I I I I-----------
0 3 6 9 12 15 18
Figure 4-6. Regression effect of posttest task performance on field dependence-
This study was conducted using a fully randomized experimental design with
pretest-treatment-posttest sequence. The sample of 38 subjects were randomly assigned
to one of two online help treatment groups. A multiple covariance analysis was applied
with computer experience, field dependence-independence, and time in help as three
covariates and posttest task performance as the dependent measure.
No significant Covariate x Treatment interactions were detected. Although the
interaction between field independence and online help format was not significant, a
trend towards an interaction was revealed. Highly field-independent subjects had higher
task performance in the text-with-motion-video help treatment. A trend towards a
Computer Experience x Treatment interaction was also observed. Subjects with very
high computer experience performed better in the text-only help treatment. Finally, a
trend towards a three-way interaction was also revealed. Very field-independent com-
puter novices had higher task performance in the text-with-motion-video help format,
while highly field-dependent computer experts performed better in the text-only help
There was no significant effect resulting solely from the online help format.
When considered independent of computer experience, field dependence-independence,
and time in help, the use of dynamic pictorial elements in online help messages had a
negligible effect on application task performance.
Increased computer experience and greater field independence were significantly
and independently related to improved performance on application tasks in the unfamil-
iar graphical user interface. Time in help was not significantly related to application
task performance, when controlling for differences in computer experience and field
The results of this study may contribute to our understanding of how computer
users learn to operate advanced graphical applications. When such learning involves
presentation of information via online help, knowledge of these aptitude effects can help
designers improve the design of online help information, particularly with respect to the
use of pictorial message elements. The implications of this study for the design of
online help systems, and for future research in this area, are discussed in the next
DISCUSSION AND RECOMMENDATIONS
The purpose of this study was to examine the effects of field independence and
computer experience on computer users learning application functions in a graphical
user interface (GUI). Data were analyzed using a multiple covariance analysis. Analy-
sis of application task performance detected significant regression effects for field
independence and for computer experience. No significant Aptitude x Treatment inter-
action (ATI) effects were detected, although trends towards such interactions were
evident. Specific results of this study are discussed below with respect to the research
questions addressed. The significance of the results are then described within the con-
text of the theories upon which this study was founded.
The implications of these results for future research and development of online
help systems and other aspects of human-computer interface design are also presented
here. Emphasis has been placed on recommendations for improving the design of
online help systems to accommodate differences in users' cognitive styles and prior
computer experience. The application of these findings to the design of adaptive
computer-based instructional systems is discussed, and several recommendations for
additional studies are presented.
Discussion of Findings
Each of the research questions posed for this study are examined in this section
based on the results and analysis presented in the preceding chapter, and subject to the
limitations within which this study was conducted. The limitations for this study
included the composition of the target population, the sample size obtained, the nature
of the computer system and spreadsheet application involved, characteristics of the
dynamic pictorial elements used in the experimental treatment, and the instructional
The null hypotheses tested in this study are examined below citing the signifi-
cant findings of the analysis and relevant aspects of the data collected. These
hypotheses were stated pertaining to a learning situation where computer application
online help messages were displayed in a GUI, where the GUI was unfamiliar to the
users, and where the instruction was systematically varied by adding dynamic pictorial
elements to text-based help displays.
Hvyothesis 1. No significant differences in application task performance result
from a three-way interaction among field dependence-independence, prior computer
experience, and the presence of dynamic pictorial message content in online help. This
hypothesis was not rejected.
The ANCOVA did not detect a significant three-way interaction effect. A trend
towards an interaction was evident, however, from a visual inspection of a plot of the
regression planes (see Figure 4-4). The presence of dynamic pictorials in online help
was related to a decline in application task performance for the most field-dependent
computer experts, while for field-independent computer novices the presence of
dynamic pictorials was associated with an increase in performance.
Limited power in the analysis inhibited detection of significant interaction
effects. The sample size obtained for this study was small (N = 38). Cronbach and
Snow (1977) recommended that ATI studies incorporate samples with at least 100 sub-
jects per group. The lack of significance of the interaction effects in this study should
be viewed in light of the small sample size. The trends toward interactions observed in
this analysis are evidence of effects that require further study.
Hypothesis 2. No significant differences in application task performance result
from an interaction between prior computer experience and the presence of dynamic
pictorial message content in online help. This null hypothesis was not rejected.
A test for an interaction effect between computer experience and the instruc-
tional treatment on task performance--when controlling for individual differences on
field independence and time in help--did not reach significance. However, when the
predicted posttest performance scores were plotted for the two treatment groups against
computer experience, a trend towards an interaction effect was evident (see Figure 4-1).
An examination of this interaction plot revealed that students with the most computer
experience tended to perform better in the text-only help treatment than in the text-
with-motion-video help treatment. For students with low levels of computer experience
there was negligible performance difference between the treatment groups.
This interaction trend, although not significant, indicates that highly experienced
users may perform better on GUI application tasks when provided with text-only online
help than when help also includes dynamic pictorials. This assertion, however, should
be viewed in the most tentative manner. This trend towards an interaction effect
requires further investigation.
Hywothesis 3. No significant differences in application task performance result
from an interaction between field dependence-independence and the presence of
dynamic pictorial message content in online help. This hypothesis was not rejected.
The ANCOVA failed to detect a significant interaction effect between field
independence and treatment on posttest performance scores. An illustration of this
nonsignificant interaction effect, depicted in Figure 4-2, revealed different regression
slopes on posttest performance for the two treatment groups as field dependence-
independence varied. First, performance increased as field independence increased,
regardless of treatment. The increase in performance was greater when help messages
included dynamic pictorial elements than when help messages were text-only. This
interaction trend indicated that as field independence increased, greater benefit was
gained from the motion video content. Although nonsignificant, this trend toward an
interaction was noteworthy in this study, particularly in light of the very small sample
This interaction trend is consistent with Witkin's field independence theory.
Individuals with higher field independence are better able to internalize and comprehend
the structure of visually complex stimuli (Witkin et al., 1971). Since the motion video
help displays used in this study were visually complex, the students who were most
field-independent were able to benefit most from them, while the more field-dependent
students benefited less. This interaction trend, while not significant, warrants further
Hvyothesis 4. No significant differences in performance on computer applica-
tion tasks exist between subjects viewing text-only online help and subjects viewing
online help containing text and dynamic pictorial elements. This hypothesis was not
There was no significant difference between treatment groups on posttest per-
formance when controlling for between-subjects variance on computer experience, field
dependence-independence and time in help. The addition of dynamic pictorial message
elements to online help had no detectable effect on learning the spreadsheet application
The absence of a treatment main effect was not unexpected in this study. The
lack of a treatment effect appeared to contradict Paivio's dual-coding theory applied to
online help message design. There appeared to have been no positive impact on appli-
cation task performance resulting from the addition of motion video displays in help.
There were, however, several mitigating experimental design factors that may have
diminished the instructional benefits of dynamic pictorial elements. First, a primary
instructional design characteristic of online help is that it is learner-controlled.
Computer-based instructional designs that are learner-controlled have been found to be
less effective than program-controlled designs (McNeil & Nelson, 1991). Because the
subjects in this study controlled their viewing of online help and dynamic pictorials,
control of exposure to the instructional treatments was limited. This is a problem
inherent to all studies involving learner-controlled computer-based instruction. Second,
subjects in both treatment groups selected and viewed an average of about nine help
topics. On average, students displayed help messages for less than five percent of the
total training time. The exposure to the experimental treatment was therefore relatively
brief. Third, the clarity of the motion video images was reduced by the compression-
decompression methods inherent in the digital video technology employed. This may
also have reduced the effectiveness of the pictorial sequences. Finally, since the GUI
and the application were unfamiliar, and the video segments presented images captured
from that interface, the pictorial displays contained visually unfamiliar--and perhaps
unrecognizable--interface features. These four methodology factors may have dimin-
ished the potential instructional benefits of the dynamic pictorial message content. The
potential for dynamic pictorial elements to contribute to learning from online help
should not be entirely disregarded on the basis of this nonsigificant finding.
Hvyothesis 5. No significant relationship exists between prior computer experi-
ence and a computer user's performance on computer application tasks in an unfamiliar
GUI. This null hypothesis was rejected.
A significant regression effect for computer experience (CEXP) on posttest per-
formance (POST) was detected. This effect demonstrated the significant relationship
between computer experience and application task performance when individual differ-
ences of field dependence-independence and time in help were controlled by using these
variables as covariates.
Computer experience, as measured using the Computer Experience and Compe-
tence Survey, proved to be a useful predictor of success for the computer-based training
implemented in this study. When learning graphical application functions in an unfa-
miliar GUI, where instruction was provided in an online help environment, individuals
with extensive computer experience would be expected to perform significantly better
on application tasks than would individuals who had little prior computer experience.
Importantly, the precise nature of prior computer experience need not be determined.
The CEXP score, a general measure of prior experience, was found to be significantly
related to application task performance in this study.
Hvyothesis 6. No significant relationship exists between field dependence-
independence and a computer user's performance on computer application tasks in an
unfamiliar GUI. This hypothesis was rejected.
A significant regression effect was detected for field dependence-independence
(FDI) on posttest performance (POST), when differences in computer experience
(CEXP) and time in help (THELP) were controlled by using these concomitant vari-
ables as additional covariates. This relationship indicated that students with higher field
independence performed more accurately, more rapidly, or both on task-based perfor-
mance measures when learning application functions in the unfamiliar direct-
Individuals who were highly field-independent, who had demonstrated strong
visual disembedding skills on the Group Embedded Figures Test (GEFT), were better
able to interpret and manipulate the complex visual environment of the graphical
spreadsheet application in this study. Because the GUI for the PM Chart application
was relatively complex--compared with other direct-manipulation application
interfaces--this effect may have been amplified. The relationship between field inde-
pendence and application task performance might not be detected in studies of
applications having simpler user interfaces.
Effects on application task performance. Significant effects were found for both
computer experience and cognitive style on application task performance. Performance
improved as computer experience increased and as field independence increased. The
relationship between field independence and performance was weaker than that found
between prior experience and performance. No significant Aptitude x Treatment inter-
actions were detected related to performance on application tasks, although trends
toward such interactions were observed. The small sample size obtained for this study
reduced the power of this experimental design to detect significant ATI effects. Future
studies examining these questions should be conducted with much larger samples.
Caution regarding generalizations. These results are interpreted here only with
respect to the population sampled for this study. Caution must be exercised when
attempting to generalize these findings to other populations. In addition, the effects of
cognitive style and computer experience on application performance and online help
usage must be understood in relation to how these variables were operationally defined
and measured in this study. In particular (a) cognitive style referred to field indepen-
dence as measured with the Group Embedded Figures Test; (b) computer experience
was measured using the Computer Experience and Competence Survey; (c) time in help
was measured as the total time that application help was displayed during the training
lessons; and (d) application task performance was measured in the PM Chart graphical
spreadsheet application. Generalization of these results to different populations or other
instructional conditions is not recommended.
Recommendations for Future Research
The results of this study may be applied to improving the design of the
human-computer interface (HCI), particularly with respect to online help systems. In
addition, these findings may influence the design of future intelligent tutoring systems;
computer-based instructional systems that can automatically sense and immediately
adjust to salient characteristics of learners while they are learning. Finally, this study
can be used as an example of applied research where theoretical problems of instruc-
tional design may be investigated while significant progress is also made in the
development of advanced instructional systems software. These recommendations are
presented to prompt other researchers to conduct additional research regarding similar
instructional design problems.
Improving HCI Design
The results of this study may lead to improvements in the design of online help
and other interface features. Both computer experience and field independence were
found to be significantly related to performance on graphical application tasks. There
was a trend toward an interaction between field independence and the use of motion
video affected task performance. Similarly, there was a trend toward an interaction
between computer experience and the presentation of dynamic pictorials in online help
that also affected task performance. Each of these results should be considered when
designing features of graphical user interfaces.
Sensitivity to user experience. As other research on learning in human-computer
interfaces has shown, students' prior experience with computers had a significant effect
on their performance in the OS/2 graphical spreadsheet application. Understanding this
effect, and developing advanced interface features to accommodate different levels of
user expertise, should be a high priority for those engaged in human-computer interac-
tion research and development. Novice users should find the features of a GUI
intuitively obvious and easy to learn. Expert users should also find these features intui-
tive, consistent, and efficient to manipulate. A key goal for HCI designers must be to
not place either experts or novices at a disadvantage by incorporating complex or inef-
ficient features into a GUI. Moreover, interface features that might significantly
influence the operation of the system or application should first be examined in
prototype form and then be evaluated in realistic work settings with groups of potential
users who vary considerably in their prior computer experience.
One example of GUI features that created difficulty for novice users was appar-
ent in this study. Several subtle marking techniques--often small or marginally visible
graphical symbols--were used in this GUI that indicated changes in status for icons,
windows, and other controls. These subtle visual cues were difficult for novice users to
recognize. Novices appeared to learn to recognize and identify these markings less
readily than experts. Also, icons that appeared nearly identical (e.g., OS/2 icons for
folders and folder templates) were often misidentified by novice users. More experi-
enced users required less practice to correctly identify and manipulate such similarly
appearing objects. HCI designers should carefully evaluate instances of minimally cued
interface changes, and the use of similarly appearing visual symbols, to determine
whether novices can correctly identify and manipulate them.
Sensitivity to cognitive style. Design problems similar to those discussed above
also relate to designing graphical interfaces that are as usable for field-dependent users
as they are for field-independent users. There was no data from this study to suggest
that expert users were also highly field-independent. Designers therefore cannot assume
that features of a graphical interface that are more usable for novice users will auto-
matically be usable by those with low field independence. Different design issues arise.
For example, would field-dependent users find a tree-structured file management inter-
face more usable than a flowed-icon interface? Would field-dependent users find a
series of graphical function buttons more efficient to manipulate than pull-down menus?
How would the performance of field-independent users be influenced by these different
interface structures? These design questions can best be resolved if further research into
these phenomena is conducted.
Appropriate use of dynamic pictorials in help. One important instructional
design issue prompting this research was the appropriate use of pictorial message
content in online help. The objective of this study was to examine what relationships
exist between the presentation of dynamic pictorials in help, the users' computer
experience and cognitive style, and their performance on application tasks in a GUI.
One inference that may be drawn from these results is that motion video images did not
appeal to or did not benefit the most experienced computer users. Using text-only help,
expert users were somewhat better able to understand and control the application inter-
face. When motion video images were added to online help, expert users' performance
did not increase as much as with text-only help. Also, the addition of motion video
images appeared to increase the performance of the most field-independent users while
there was no such benefit for field-dependent users.
If these results can be replicated, designers of online help systems might utilize
these findings in designing online help and other computer-based tutorial environments.
The design of online help for novice users may make greater use of dynamic pictorial
content than would be used in help designed for expert users. In addition, alternative
visualization techniques might be incorporated to support field-dependent users who
would not benefit from the type of dynamic pictorials used in this study. Interface
designers, whether focused on computer application or operating system interface fea-
tures, should systematically evaluate the range of cognitive and affective responses
elicited by the online information in their products.
Shneiderman (1986) identified sensitivity to individual differences as one of the
most important issues in HCI design. He urged researchers to develop "design guide-
lines to support individuals with differing gender, age, education, ethnic background,
cultural heritage, linguistic background, cognitive styles, [and] learning styles" (p. 346).
Advances in HCI design must rely more heavily on the result of rigorous, theoretically
motivated studies of user behavior. Studies that concentrate on the effects of individual
differences, and the ATI effects between these differences and features of the user
interface, will help improve the quality of human-computer interaction.
Intelligent Tutoring Systems
The effects of computer experience and field independence on the use of online
help identified in this study can be applied to improving the design of adaptive instruc-
tional systems. Online help is one class of computer-based instructional system that
typically has very limited ability to adapt to users' individual characteristics. An intel-
ligent tutoring system (ITS) is a computer-based instructional environment that
incorporates heuristic decision-making capabilities which allow it to appropriately adapt
instructional presentations to best fit certain characteristics of each individual learner.
The results of this and similar studies may be incorporated into the design of an
ITS through the development of heuristics relating aptitude variables to parameters of
instructional design. This would allow an ITS, for example, to appropriately adapt
aspects of an instructional presentation to users with differing levels of computer expe-
rience or field independence. This can be done through interactive determination of
individual aptitudes, tracking user interface actions, and providing for user control and
customization of information presentation parameters.
Interactive assessment of individual differences. In this study, group-
administered assessment instruments were used to determine the level of students'
computer experience and field independence. Ideally, an ITS could measure these traits
using an interactive, online assessment. Methods for performing a variety of interactive
assessments are being developed by researchers (Perez & Seidel, 1990). Interactive
techniques to measure field independence, such as an online form of the Group
Embedded Figures Test, might be developed. Alternative techniques to assess field
independence could be incorporated into the interface such that the individual would not
become aware that an aptitude test, per se, was being administered. One advantage of
this approach would be that aptitudes could be measured without requiring separate
testing and data entry procedures. The major benefit, however, would be the capacity to
individualize presentation of information by matching presentation attributes to learner
Tracking interface activity. In this study, the use of online help was tracked
automatically by software that logged all help topic display activity without the
student's awareness. The data collected included time in help, the number of help topics
opened, the names of the help topics opened, time in help per topic, and the frequency
of playing motion video sequences. Further collection and analysis of such data might
provide information useful to both the online help designers and the application devel-
opers. In an ITS, tracking user actions in this way would provide a continuously
updated source of information containing patterns of user response to instructional
messages. Decisions regarding instructional presentation may then be made on the basis
of that information. In addition, similar tracking logic could be incorporated into any
graphical application to construct a profile of a user's manipulation of interface objects.
This profile could be examined periodically, and if the pattern of manipulation fell out-
side certain parameters, the interface might automatically present an explanation of that
object, or change the object so it would be easier for that user to understand.
User customization of interface features. Most graphical user interfaces devel-
oped for wide use, such as the workplace model incorporated into IBM Operating
System/2, provide features that support interface customization by individual users.
This customization includes how icons are arranged, how different mouse buttons affect
objects in the interface, the colors used to highlight various interface controls, the type
and degree of confirmations required for actions on certain objects, and many other
features. Online help systems should provide for similar customization capabilities.
One user may wish to have information presented with audio-only or audio-visual con-
tent, while other users may prefer a text-only display. Once users have determined
what information formats and features best suit their needs, they would be able to cus-
tomize the help environment accordingly.
Related Research Issues
Beyond the scope of this study are many related issues that future research
should address. Core issues raised by this study concerned the design of online help
messages, appropriate use of dynamic pictorials in online help, and the relationship
between cognitive style, computer experience, and performance on tasks in graphical
user interfaces. Related issues raised by this study that require further investigation are:
(a) the effects of alternative visualization techniques; (b) measuring performance sensi-
tive to field independence; (c) potentially negative effects of using dynamic pictorial
elements; and (d) the relationship between computer experience and field independence.
Alternative visualization techniques. The trend toward an interaction between
cognitive style and use of dynamic pictorials in online help that influenced application
task performance indicated that for some users, in certain applications, such visuals may
have a desirable effect. Would a similar effect have occurred if the dynamic pictorial
content had been presented using a different technique? For example, would animated
bitmaps that precisely matched the application interface have been more effective in
improving performance across all levels of field dependence-independence? Would a
similar ATI effect be observed with an animated bitmap sequence, or would the effect
be modified? Did the visual blur effect in the digital motion video images in this study
have a negative influence on students' task performance? Future studies comparing
motion video with other visualization techniques may answer these and other related
Performance measurements sensitive to field independence. Another research
question is related to the small effect size detected for field independence in this study.
Did the manner in which performance was measured, whereby each subtask was scored
as either success or failure, overlook subtle ability differences related to field indepen-
dence? Would a more fine-grained measure of task completion have been more
sensitive to the effects of cognitive style? Would a different approach to performance
testing yield a more sensitive measure for this type of study? The disembedding skill
attributed to highly field-independent users might not be measured in certain user inter-
face tasks. Further research in this area should focus on identifying the categories of
interface actions or objects that field-dependent users find most difficult to master.
Such studies would lead toward a more complete understanding of the nature of cogni-
tive style influences on human-computer interaction.
Negative effects of dynamic pictorial elements. Although not significant, the
trend toward an interaction effect between computer experience and use of motion
video on application task performance is indicative of an effect that should be
investigated further. Specifically, does the use of dynamic pictorials (i.e., motion video)
in online help contribute to a decline in performance as an individual's computer
experience increases? If such an effect can be clearly demonstrated, online help sys-
tems may be designed to track user activity so that as a user's experience in the
application interface increases, the use of dynamic pictorials in online help would be
decreased. More conclusive evidence of such an ATI effect is required before such
implementations would be justified.
The relationship between field independence and computer experience. In this
study, no relationship was found between these aptitude measures. The data appear to
indicate that as computer users become more experienced, their level of field indepen-
dence is not affected. Would field independence remain constant through all types of
computer experience? Could prolonged, intensive experience with GUI applications
increase individuals' field independence? If such an effect could be demonstrated,
future online help systems could be designed to accommodate change in a user's cogni-
tive style, as well as change in the user's level of computer experience.
Future studies addressing these and other related questions would help develop
valuable HCI design guidelines, and contribute further to understanding the effects of
field dependence-independence and computer experience on learning to operate
computers with direct-manipulation graphical user interfaces. This promising research
direction provides an opportunity to develop and evaluate instructional message design
theory while simultaneously advancing the art of human-computer interface design.
Each of the findings reported here require further investigation. This study
demonstrated trends toward ATI effects anticipated by instructional design theory. It
also provided new evidence of a significant positive relationship between field
independence and task performance in a graphical user interface. For HCI design to
benefit from these findings, however, further studies are needed to isolate specific
features of graphical user interfaces that contribute to poor performance in
field-dependent users. Such interface features might then be eliminated for those users
by incorporating user-customization capabilities or by adding adaptive interface fea-
tures. Greater sensitivity to individual differences will help make computers more
human-literate, rather than requiring all users to become computer-literate.
This study was conducted to examine the effects of field independence and
computer experience on learning application functions in a graphical user interface
where online help was the primary instructional resource. The experimental instruc-
tional treatment consisted of online help incorporating dynamic pictorial message
elements. From a university student population in an undergraduate business manage-
ment course, 38 subjects volunteered for computer-based training. The subjects were
randomly assigned to one of two treatment groups that varied only the online help
format: text-only help and text-with-motion-video help. In both treatment groups, the
display of help information was controlled by the subjects.
For this study, a fully randomized design with pretest-treatment-posttest
sequence was used. Data were analyzed using a multiple covariance analysis. Field
dependence-independence, computer experience, and time in help were applied as the
covariates. The grouping factor was the online help treatment. The dependent measure
was performance on application tasks in the training posttest. Significant effects on task
performance were found for both field independence and computer experience. No
significant interaction effects were found between field independence and treatment or
between computer experience and treatment.
From an analysis of these results, several tentative conclusions were drawn.
First, the performance of computer users on GUI application tasks increased as either
field independence or computer experience increased. Second, there was a trend toward
a Field Independence x Treatment interaction. The positive influence of field indepen-
dence on task performance was greater when online help incorporated dynamic pictorial
message elements. There was also evidence of a trend towards a Computer Experience
x Treatment interaction. As computer experience increased, the presence of dynamic
pictorial elements in online help had an increasingly negative effect on application task
The results of this study should be independently confirmed before these tenta-
tive conclusions are applied in HCI design and development. Also, these results may
not apply to other populations or for other types of computer applications or user inter-
faces. Further research into the effects of field independence and computer experience
on human-computer interaction are warranted. The results of future studies in this area
can lead to the development of intelligent, adaptive user interfaces that are sensitive to
individual differences. As working with computers becomes an increasingly pervasive
aspect of life, the application of instructional design principles to improving human-
computer interaction must become an interdisciplinary priority.