UFDC Home  

Usability Testing: A User-Centered Approach to Improve Electronic Resource Design (workshop for ACURIL XXXVIII)

Digital Library of the Caribbean UFEDUC UFLAC
Permanent Link: http://ufdc.ufl.edu/UF00089500/00001
 Material Information
Title: Usability Testing: A User-Centered Approach to Improve Electronic Resource Design (workshop for ACURIL XXXVIII)
Physical Description: Mixed Material
Language: English
Creator: Ochoa, Marilun
Wooldridge, Brooke
Publisher: George A. Smathers Libraries
Publication Date: 2008
 Subjects
Subjects / Keywords: Caribbean   ( lcsh )
dLOC Presentation
Spatial Coverage: Caribbean
 Record Information
Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UF00089500:00001

Permanent Link: http://ufdc.ufl.edu/UF00089500/00001
 Material Information
Title: Usability Testing: A User-Centered Approach to Improve Electronic Resource Design (workshop for ACURIL XXXVIII)
Physical Description: Mixed Material
Language: English
Creator: Ochoa, Marilun
Wooldridge, Brooke
Publisher: George A. Smathers Libraries
Publication Date: 2008
 Subjects
Subjects / Keywords: Caribbean   ( lcsh )
dLOC Presentation
Spatial Coverage: Caribbean
 Record Information
Source Institution: University of Florida
Holding Location: University of Florida
Rights Management: All rights reserved by the source institution and holding location.
System ID: UF00089500:00001


This item has the following downloads:

Woolridge_Ochoa ( PDF )


Full Text




Usability Testing:
A User-Centered Approach to Improve Electronic Resource Design

ACURIL XXXVIII

The e-librarian: ideas, innovation and inspiration
June 1-6, Montego Bay, Jamaica


Presenters:

Marilyn Ochoa
Assistant Head, Education Library
University of Florida
Email: mochoa@uflib.ufl.edu

Brooke Wooldridge
Project Coordinator, Digital Library of the Caribbean
Florida International University
Email: brooke.wooldridge(ifiu.edu

Abstract
Some academic and research libraries have expanded their roles in providing access to
information. No longer are they simply the information gatekeepers, but they are making
available old and hidden knowledge by creating new resources such as digital library collections
and by customizing vendor produced resources. Although there is growing use of digital libraries
by researchers to discover information, researchers also have access to more and more content
online from which to search. Users may not understand the importance of using library provided
resources and instead use a site solely based on how easy it is to search and navigate. Pitted
against the multitude of web resources available, the institutions that produce digital libraries
then must be cognizant of user expectations in their design process and provide a useful and
appealing resource.

User centered research and development are important methods to discover the expectations and
required functions of a resource during the design. Generally identified as usability testing, the
evaluation and design of a user interface based on data collected from real users of a system
focuses on maximizing the success of an interface.

This workshop will increase awareness of the importance of designing products and services
with a high degree of usability to increase the use of library provided electronic resources.
Topics include the justification, planning and implementation process for usability testing.
Presenters will provide participants with a framework for the evaluation of electronic resources,
including the methods to effectively develop and conduct testing, as well as a chance to
participate in the usability testing of the Digital Library of the Caribbean.

The Digital Library of the Caribbean (dLOC), a joint project of the University of Florida,
University of Virgin Islands and Florida International University in partnership with institutions
in the Caribbean and circum-Caribbean, serves as the access point for scholars, students and
citizens of interdisciplinary Caribbean and circum-Caribbean research, gathers together a critical








mass of cultural, historical and research materials originally held in archives, libraries and
private collections. This unique digital library provides content submitted directly from dLOC
partners and members and allows users to browse materials or search the text through
multilingual interfaces. The structure of dLOC presents a challenge for designers to ensure that
the end product is easy to use and navigate and meets a diverse body of researchers'
expectations.

During this workshop participants will learn about both the theory and application of usability
testing. By evaluating dLOC themselves, participants will help to identify opportunities to
improve this resource while learning how to analyze tasks, identify issues and provide
recommendations for any resource.








Introduction
Some academic and research libraries have expanded their roles in providing access to
information. No longer are they simply the information gatekeepers, but they are making
available old and hidden knowledge by creating new resources such as digital library collections
and by customizing vendor produced resources. Although there is growing use of digital libraries
by researchers to discover information, researchers also have access to more and more content
online from which to search. Users may not understand the importance of using library provided
resources and instead use a site solely based on how easy it is to search and navigate. Pitted
against the multitude of web resources available, the institutions that produce digital libraries
then must be cognizant of user expectations in their design process and provide a useful and
appealing resource.

Conceptualizing usability: What is usability?
Although usability has been discussed since the 1970s, in the early 1990s, human computer
interaction research and instruction made it clear that usability has an important role in design
and implementation of computers, interfaces and other technology products (Nielsen, 15). The
usability component of HCI is now often taught to software engineers and training departments.
According to the world's largest developer and publisher of International Standards, International
Organization for Standardization, "Human-centered design is characterized by: the active
involvement of users and a clear understanding of user and task requirements; an appropriate
allocation of function between users and technology; the iteration of design solutions; multi-
disciplinary design"(ISO 13407).

In practice, usability testing, a part of human, or user-centered design, consists of the methods
used by organizations and their designers, to obtain information from end users to discover how
they think and work through a product to obtain what they need. According to the U.S.
Department of Health & Human Services Usability.gov site, usability refers to how well users
can learn and use a product to achieve their goals and how satisfied they are with that process;
usability means that people who use the product can do so quickly and easily to accomplish their
tasks (Dumas and Redish 885). Overall, usability measures the quality of a user's experience
when interacting with a product or system, a website or software application--"the extent to
which a product can be used by specified users to achieve specified goals with effectiveness,
efficiency and satisfaction in a specified context of user" (ISO 9241-11).

Usability is a combination of many factors. Usability considers such factors as cost-effectiveness
and user satisfaction. The quality of documentation can affect a user's perception of the usability
of the interface. However, although help information may be provided, it is important to make
the resource as learnable and easy to use as possible. Ease of learning addresses how quickly a
user who has never seen the user interface before can learn it well enough to accomplish basic
tasks. Efficiency of use measures how fast experienced users can accomplish desired tasks and
signifies that productivity using the resource will be high. Memorability assesses whether a
casual user who has used the system before can remember enough to use it effectively the next
time, without having to learn the system all over again. Error frequency and severity determines
how often users make errors while using the system, the severity of the errors, and how users
recover from these errors; the expectation is that errors will be low with users making few
mistakes that they can recover from easily. An important component that is difficult to quantify
is the subjective satisfaction of the users. These aspects of usability, and many others, together
provide an overall assessment of a user interface.








Formative and summative evaluations make up the two purposes for usability testing. Formative
testing seeks to improve interface as part of iterative design process, while summative tests
consider the overall quality for use in deciding between 2 interfaces, etc.

Why is usability important?
Technological change is stressful for many users and often developers focus on the newest
developments rather than on what is most useful and important from the user perspective. End
users are often not part of the project development process which creates difficulties for them to
understand the resource and therefore have their expectations met with it. The goal for
developers should be to create user-centered products to help users achieve their goals/
expectations from a particular resource. Since the users are usually far removed from the
development stage, developers must find strategies to actively engage them in the development.
Users offer a fresh perspective on the resource tested. With users and developers having
differing priorities and expertise with the resource/interface, each should contribute to the final
product. Usability testing is important to ensure that the interface or resource meets the goals of
the end user and the organization.

In addition in the future it is anticipated that more accessibility rules and regulations may be
mandated. Compliance to these standards for developed resources will be necessary; testing
should be conducted.

Managerial Buy-in: Managerial by-in can sometimes be difficult to obtain because testing is
perceived as a time consuming and expensive step in the design process. A usability specialist
can explain that testing is actually an often inexpensive solution for the organization in the long
run. With so many websites to choose from, developers need to ensure they provide users with
resources that they are satisfied with, in navigation and content, so that they will want to return
to it. With buy-in, testing can help determine how well the site, resource or interface meets user
needs based on usability factors, and designers would be encouraged to make the changes as
discovered in and recommended through testing.

Cost: Cheap methods for testing are available and payment to specialists to plan, run and
analyze tests may be the biggest cost. Users' time needs to be compensated in some way, and
space (lab or other test space) along with cost for testing materials need to be added to the final
budget. As Nielsen suggests, however, the nature of the resource would determine the actual
budget. For example answers to questions such as does the resource need broad acceptance or
will this site experience daily use may require that a larger investment in testing might need to be
made. (Nielsen 7).

What are the practical benefits?
If developers involve the user from the very beginning of a design and throughout it, this ensures
that the final implemented design improves the quality of their expected users' experience with
using the interface by making it easy to learn, easy to remember and intuitive and also meets the
desired expectations of the organization. Involving actual users in testing also increases the
value of the product by providing exactly the content that the users desire. With the
modifications resulting from testing, the cost benefits to the organization means less time by
developers making features available that might not be what the users want.

Also, by identifying and even anticipating problems before they arise, the long term costs of
support and future redesigns are reduced. Plus, usability testing creates goodwill between user








and organization by showing the users that their input matters, and even encourages users to take
ownership of the resource. Ensuring the resources meet these usability factors will translate into
more use and better reputation for the resource, although these may not be immediately tangible.

Types of Usability Testing
Many types of usability test methods exist but user testing is the fundamental usability method.
Testing should come sometime (and repeated) in the development lifecycle, not necessarily at the
final evaluative stage.

There are four techniques that are easily employed for usability testing: user and task
observation, scenarios, simplified think aloud and heuristic evaluation. Because an interface has
many components that a user may be faced with, using the scenario based testing method enables
the usability facilitators to target an area for consideration. There are many aspects to any given
resource that may need to undergo usability testing. Each element on the page can be a source of
difficulty for the user to be tested in a formative evaluation, or if a big picture view is taken, the
summative evaluation needs to be explored. Identifying the intended use of the resource and
type of user are essential in defining the purpose of the testing. Because even minor interface
details have can affect the way a participant uses the resource, a very narrow approach to the
testing can be effective to get at what might improve the resource. This honing in on an area
forces the user to follow expected paths/links to answer the question.

The think aloud method enables facilitators to learn both how and why users complete the tasks.
This type of testing provides the added benefit for developers who learn a little more about the
thought process behind using the interface and can be identify problems needing to be fixed; the
difficulty users experience when completing tasks can identify what is not intuitive about a
particular feature. As part of the managerial buy-in idea, it is noted that this type of testing can
be done cheaply; all that would be needed is someone to write down the responses and behavior
of the test taker.

Heuristic evaluation focuses on the guidelines for usability applied to a site, as determined by
developers or usability specialists. Such areas addressed in this evaluation can be whether the
dialogue used is simple, the site is consistent, or documentation is useful.

Other cheap methods exist for gathering data and supplemental data. For example, paper
prototyping when it is early in design can test using simple paper sketches; card sorting exercises
can be done using index cards or sticky notes and a pencil; and scenario based testing requires
just a computer with internet connection on live site. Focus groups enable the assessment of the
resource to obtain supplemental data about the errors that might be found and the general
strategy problems of the interface. Analysis of web site search logs gives a snapshot of how
users are already using a particular resource. Card Sorting uses cards that represent all the items
on the interface that a user will sort and name into categories. This sometime time-consuming
testing can reveal issues in terminology.

Combining various testing methods enables usability specialists to more fully capture the issues
about a resource or interface. For example, often user testing will focus on specific interface
problems while focus groups can reveal general strategy problems. In whatever format of tests
selected, the testing is not about the answer to the questions asked but about the process to find
the answer using the resource/interface. Testing aims to review the tool not the user's abilities








and seeks to answer major questions such as: does the user understand the purpose of the site and
its organization and can the user complete tasks using the resource?

Preparing to develop a Usability Test
The Why and Who
When managers agree to usability testing, the specialists should begin planning by reviewing the
interface/resource itself and determining who the intended audience is. Understanding the
culture, mission and objectives that developed the resource/interface is also critical; meeting with
an organizational representative/designer will help to define these areas. In that meeting, the
organization should explain the overall purpose of testing-to provide a formative or summative
evaluation of the resource/interface.

Further, who the intended audience is should be revealed along with the expected or perceived
needs/behavior/preferences of those users that drove the initial design. Determining user profiles
is a part of defining the purpose of testing. Identifying the background of user and level of
experience expected is the foundation for the recruitment plan for test participants. This will
include how many of each group is needed. Nielsen suggests that between 5 to 7 participants can
be in group or up to five for each user group in individual testing. These participants should be
representative users of the interface and if possible include both novice and expert users.

Specific tasks/functionality based on expected interactions of users with the interface/resource,
or the reasons why users go to it, can be defined. Tasks need to be representative of the uses of
the system and cover the most important features of the interface. The purpose for the testing
identified early in the planning stages can determine the tasks and specific testing method
ultimately selected.

Tasks are best if related to a particular overall scenario, and questions, especially if in talk aloud
testing, should yield precise results and be answerable within the time limit defined prior to
testing. The test method selected should encourage collection of the necessary data to analyze.

The Where, When and How
Once testing type is determined, logistical issues about the type of testing need to be addressed.
Where, when and how long each session should be is among the issues to consider. The testing
environment-either formal (in a laboratory) or informal (at a public workstation or an office)-
computer support and software needs need to be decided. Tests can be conducted in a space that
is comfortable but it can be done in a formal laboratory setting.

Choosing and training experimenters/facilitators is relatively easy and can affect when testing
will occur. They should have some familiarity with resource, enough to answer questions if they
should arise but be able to politely explain that in most testing situations s/he cannot assist in
navigating through the resource/interface. Facilitators must not lead participant and do not
interfere with user but observe and record participant activities. In infrequent cases where a user
gets stuck on a question and is no longer productive can a facilitators to give a hint to move the
test forward or stop the user. Overall, the facilitator must be objective, patient, observant, and be
able to record what is done.

The type of data is collected had already been decided but a developing the actual test questions
and other instruments and deciding how data will be analyzed will be finalized.








Once the questions have been drafted, it is good practice to pilot the test, or test the test. This
enables the specialist to determine if time allotted is correct. In addition specialists can make
improvements to the wording of test, questionnaires or other survey instruments if necessary.

Conducting the usability testing
Usability testing as part of user-centered design involves users throughout all stages of site
development, in order to create a site that meets users' needs. Individual sessions are often best
so that developers can actually see where real users are having difficulty using the resource. It is
often commented that developers themselves, not simply the usability specialists/ facilitators
should be there to watch how users navigate through what they think is an easy interface; this
allows developers to truly understand and accept what changes need to be made to make the
resource usable.

Because testing requires direct contact with end users, most institutions require that a human
testing protocol be developed and approved prior to testing. At the University of Florida the test
protocol includes an informed consent and all survey instruments. Once this protocol is
approved, recruitment and scheduling of test participants can begin.

During the testing, all questions should be given to the participant in writing, preventing
misinterpretation of what is sought. Facilitators observe user behavior, identifying glaring errors
that the user had, etc. A record of the test is done via note-taking, screen capture, and exit or
post test interview or questionnaire. After the test is complete, often compensation such as
refreshments, gift cards, etc. is provided (but this needs to be budgeted in). After the participant
leaves, it is best that a preliminary report on individual tests is drafted while the session remains
fresh in the mind.

Generally how is usability testing conducted?
1. Introduce purpose of test: improve interface, voluntary participation, results confidential,
recording method, facilitator role of not answering questions, time for questions before
testing
2. Run test: facilitator refrains from answering questions/interacting, 1 observer to remain
quiet
3. Debrief: Fill in satisfaction questionnaire before discussion, ask for comments or
suggestions
4. Prep analysis: label test materials by code, write up report: well organized notes and
preliminary report on individual tests make the work easier later on

After all testing, a full analysis of the collected data is conducted and is reported to the
organization managers and developer. The data is often analyzed by finding patterns; specialists
interpret and suggest recommendations while also documenting mistakes for future redesign.
The recommended changes are reviewed and possibly implemented. Normally this becomes an
iterative process where the usability testing is a systematic part of the development lifecycle;
usability experts and developers retest the modified interface, revisiting the process. Depending
on the type of test conducted, the time for testing itself may be short. In addition, only a
relatively small number of usability participants are usually necessary to get effective feedback
about the resource.

Generally, steps to designing/completing a usability plan are:
1. Planning: Defining the purpose (goals and objectives) of testing








2. Identifying the intended use of the resource and type of user
3. Determining user profiles is a part of defining the purpose of testing
4. Define top tasks completed using the tool/site:
5. Define method of testing
6. Select evaluation measurements and draft test
7. Pilot test/rewrite test
8. Select participants and facilitators
9. Test and reward
10. Analyze and report
11. Implement
12. Retest

How does an actual test work? Digital Library of the Caribbean as a model
The Digital Library of the Caribbean (dLOC), a joint project of the University of Florida,
University of Virgin Islands and Florida International University in partnership with institutions
in the Caribbean and circum-Caribbean, serves as the access point for scholars, students and
citizens of interdisciplinary Caribbean and circum-Caribbean research, gathers together a critical
mass of cultural, historical and research materials originally held in archives, libraries and
private collections. This unique digital library provides content submitted directly from dLOC
partners and members and allows users to browse materials or search the text through
multilingual interfaces. The structure of dLOC presents a challenge for designers to ensure that
the end product is easy to use and navigate and meets a diverse body of researchers' expectations.

Currently the library houses over 3,000 titles from ten of our thirteen current partners. Topics
include historical photographs, current social science publications, institutional historical
memory, maps, books and more. This project has facilitated the development of the website,
online content submission tools, purchase of high quality digitization stations in the Caribbean
and the training of our Caribbean partners in state of the art digitization for preservation. The
decentralized organization structure of dLOC where our partners choose the content for
submission has resulted in a variety of information being housed together under one dLOC
banner. Our goal as we move forward is to continue to support our partners as we develop more
defined topical collections through both human and technology aided selection.

It is precisely at this crossroads that we are implementing a usability study to guide our
development process. The dLOC has already benefited from previous testing of the University
of Florida Digital Library Center. This test will allow us to analyze dLOC and to make
adjustments to the current site and move forward with new designs developed with user input.
The examples and exercises in this workshop will be modified and implemented in our upcoming
multi-site Usability Assessment. The dLOC Usability team is currently seeking volunteers to
conduct testing at various sites.

Conclusion
This workshop addresses the major steps necessary to conduct usability testing of an electronic
resource and provides concrete examples of how such testing is being approached in the case of
the Digital Library of the Caribbean. The investment of time and resources in such testing serves
to guide the development process to ensure that the resource meets the expectations of the end
user and organization.








Post Usability Testing Questionnaire
Thank you for participating in the usability test of the Digital Library of the Caribbean. Please
complete this short questionnaire about the resource.

1. How would you rate the overall ease of use of dLOC resource?
5 4 3 2 1
very somewhat difficult
easy easy

2. How would you rate the overall navigability of the resource?
5 4 3 2 1
very somewhat difficult
easy easy

3. How would you rate the overall learnability of the resource?
5 4 3 2 1
satisfying somewhat frustrating

4. How likely would you use this resource for online research?
5 4 3 2 1
somewhat likely not
likely likely

5. How likely would you recommend this resource to someone for online research?
5 4 3 2 1
very somewhat difficult
likely likely likely

6. How would you rate your overall experience with dLOC resource?

5 4 3 2 1 5 4 3 2 1
Excellent Good Poor Satisfying Frustrating

7. Which areas did you find easiest to navigate:


Collection pages result list pages help pages
Search pages individual item records

8. What were your impressions of the Help information?

5 4 3 2 1 5 4 3 2 1
Clear Confusing Adequate Inadequate
Information Information


9. What did you find to be the most exciting feature of the UFDC?








10. Please indicate which areas could be improved (provide more information if necessary):

Site design navigation within result pages page loading
Search interface (general) content format output options
SBasic terminology other, please specify
SAdvanced help pages
_ navigation throughout site result pages

Focus Group
The purpose of this study is to examine information discovery and retrieval using the digital
Library of the Caribbean resource. We now wish to ask questions about your experience with
and impressions of the resource. We hope to determine how well dLOC matches your
expectations and needs and allows for ease of learning and use.

1. After continued use of dLOC, what features of the resource would you definitely use? Why?
2. After continued use of dLOC, what features of the resource would you definitely NOT use?
Why?
3. Why would you want to use this resource?
4. Would you use this resource?
5. How important is it for you to find scholarly or academic articles? Does dLOC provide you
the types of resources you want to find?
6. What other databases do you use for finding information for Caribbean and circum-
Caribbean research?
7. Question about type of formats needed
8. Were results pulled up in a reasonable timeframe (seconds versus minutes?)? Did you have
any trouble retrieving results due to connection speed?
9. Was it difficult to figure out how to search from within dLOC? What could make this
easier?
10. Should there be more ways to search database content?
11. When performing searches in the library catalog or an online search engine, do you care how
the results are sorted when displayed (e.g., by relevancy, by title, by author)? Explain why.
12. Question about what kind of documentation they might want/need
13. Question about trilingual interface...Abstracts and subject headings are translated to another
language...
14. Does it matter to you that you cannot print or E-mail your saved citations/records within the
e-shelf section?








Resources
Dumas, Joseph and Janice (Ginny) Redish. 2002. Usability in practice: formative usability
evaluations evolution and revolution Conference on Human Factors in Computing Systems
Minneapolis, Minnesota, USA 885 890 Usability in Practice Session.
International Standards Organization, "Ergonomic requirements for office work with visual
display terminals (VDTs) -- Part 11: Guidance on usability," ISO 9241-11:1998, March 13,
1998

(February 2, 2005).
International Standards Organization, "Human-centred design processes for interactive systems,"
ISO 13407:1999, July 1999

(February 2, 2005).
Krug, Steve. 2006. Don't make me think!: a common sense approach to Web usability. Berkeley,
Calif: New Riders Pub.
Nielsen, Jakob. 1993. Usability engineering. Boston: Academic Press.
U.S. Department of Health & Human Services, "Usability.gov,"
(February 16, 2005).




  Home | About dLOC | Collections | Governance | Digitization | Outreach | Contact  
  Powered by SobekCM
Acceptable Use, Copyright, and Disclaimer Statement  
© All rights reserved