• TABLE OF CONTENTS
HIDE
 Title Page
 Table of Contents
 Executive summary
 Methodology
 University of Florida Digital Collections,...
 Findings and recommendations
 Continuing review/study closure...
 Usability test participant...
 University of Florida Digital Collections...
 Post usability testing questio...
 Popup survey






Title: Usability report for the University of Florida Digital Collections (UFDC)
CITATION PAGE IMAGE ZOOMABLE PAGE TEXT
Full Citation
STANDARD VIEW MARC VIEW
Permanent Link: http://ufdc.ufl.edu/UF00073771/00001
 Material Information
Title: Usability report for the University of Florida Digital Collections (UFDC)
Physical Description: Book
Language: English
Creator: Ochoa, Marilyn
Affiliation: University of Florida -- George A. Smathers Libraries
Publisher: George A. Smathers Libraries, University of Florida
Publication Date: 2007
 Subjects
Subject: University of Florida   ( lcsh )
Library science   ( lcsh )
Surveys   ( lcsh )
University of Florida.   ( lcsh )
Spatial Coverage: North America -- United States of America -- Florida
 Record Information
Bibliographic ID: UF00073771
Volume ID: VID00001
Source Institution: University of Florida
Holding Location: George A. Smathers Libraries, University of Florida
Rights Management: All rights reserved, Board of Trustees of the University of Florida

Table of Contents
    Title Page
        Page 1
    Table of Contents
        Page 2
        Page 3
    Executive summary
        Page 4
    Methodology
        Page 5
        Page 6
    University of Florida Digital Collections, scenario-based testing
        Page 7
        Page 8
        Page 9
        Page 10
        Page 11
        Page 12
        Page 13
        Page 14
    Findings and recommendations
        Page 15
        Page 16
        Page 17
        Page 18
    Continuing review/study closure report
        Page 19
        Page 20
    Usability test participant questionnaire
        Page 21
        Page 22
    University of Florida Digital Collections questionnaire
        Page 23
        Page 24
    Post usability testing questionnaire
        Page 25
        Page 26
    Popup survey
        Page 27
Full Text






Usability Report for the University of Florida Digital Collections (UFDC)


Date of Report: March 13, 2007


Period of Testing: February 9- February 19, 2007


Date of Test Session/Focus Group: February 27, 2007


Prepared for: Erich Kesse
Digital Library Center


Prepared by: Marilyn Ochoa
Digital Services Librarian












Table of Contents


E executive Sum m ary .............................. ............ ... ........................................ ................... ... ........ 4
M eth o d o log y ...................................................................................... ................. ................... ..... ... 5
What happened during the usability test ................... .........................................5
W h o w e tested ................................................................................................5
W here w e tested ............................................. .. .............. .... ........................ ...........
W hat data w as collected ................................................................................ .........................7
University of Florida Digital Collections (UFDC), Scenario-Based Testing
Initial R source Im pressions ....................................................... ............................... 7
What are your initial impressions of this resource? .................................. ................
W hat did you like about this resource? ................................ ..............................................
W hat did you dislike about this resource? ............................. ............................................
What type of information would you expect to find on this resouce? ...............................
Scenrio-B ased T testing ............................................................................ ................... 8
Overall N aviation and Ease of U se .............................................................................. 10
S ea rch in g ......................................................................................... ................. 11
R e su lts ............................................................................................................... 1 1
Item V iew er.................................. ................ ........ 12
University of Florida Digital Collections (UFDC), Post-Test Reactions/Focus Group Discussion ........... 13
Findings & R ecom m endations .................................................................................................................. 15
Overall Navigation and Ease of Use/Help Issues........................................................... 15
Finding [#1]: More information about UFDC is needed on the homepage .......................
Finding [#2]: The current homepage layout dies not adequately emphasize collections ........
Finding [#3]: Collection descriptions are not entirely accessible .......................................
Finding [#4]: Links to Help, UFDC homepage and external resources are not obvious to the
u ser....................................... ............................................ ..............................................
Finding [#5]: Customization and other output options are desured to improve user
ex p erien ce ........................................................ .................................
Finding [#6]: Marketing tools can be used to help explain the resource ..........................
Searching Issues ......................................................................... ................... .. 16
Finding [#1]: All Collections and Advanced search interfaces only should be available........
Finding [#2]: It is difficult to search multiple collections or specific subcollections ..............










Finding [#3]: More advanced search features should be clearly explained from search
interface an d h elp p ag es ................................................................................... ...................
Results Issues ................................................. ............. 17
Finding [#1]: All serial titles and volumes appearing when a keyword search is conducted
m u st b e im p rov ed ..... ............ ............................................ .............. ........... ...........
Finding [#2]: More information needs to appear in the result list table view ...................
Item V iew er Issues ......... ..... ........................................................... .................. ..... ..... 17
Finding [#1]: The left navigation pane needs to be more visible.......................................
Finding [#1]: Display of thumbnail, table of contents and contributors features need to be
im proved ............................... ............ .. ................... .................
Finding [#1]: Modify navigational features on the Result pages ........................................
A p p en d ix : ................................................................. .................................... ................ .............. 18












Executive Summary

The University of Florida Digital Collections (UFDC) is a research tool that enables a user to find
resources held at the University of Florida (UF), learn about the physical collection it is held in, and use
the online full content of the resource. The University of Florida Digital Library Center (DLC) developed
this digital library to house the digital resources (e.g. monographs and serials, maps and other images) and
provide open access to remote and local users.

The Digital Services Librarian, who serves as the liaison to public services and the Digital Library
Center/Systems team, assisted with development of the interface by recommending common functionality
and design found in similar resources. The librarian considered participant preferences based on
experience with user behavior in class instruction and reference desk transactions. The recommendations
were accepted and implemented with consideration to the technical capability of the open source
Greenstone Digital Library System used for its metadata storage, retrieval, and search engine.

At the time of testing, collections were still being digitized and processed. The UFDC offers at least
thirty collections grouped into several major headings including: Arts, Humanities and Social
Sciences; Science and Technology; Maps; Florida; and World. In addition, the UFDC links to external
collections that the DLC assisted in digitizing.

Two collections that represented much of the functionality desired to test were selected for the usability
testing. The resulting UFDC interface, result pages and item viewer would need to be evaluated based on
how well participants could accomplish the retrieval of UFDC results prior to the full launch in April
2007.

To accomplish the task, the librarian developed and administered usability testing to a group of university
participants. The testing examined how well the UFDC allows for ease of use, navigability and
leamability. In addition, the testing would indicate how well the UFDC matched participant needs and
expectations. Key areas of concern were identified prior to testing:

o Are the UFDC search pages-all collections, basic and advance-intuitive?
o Do the result pages provide the type of information a participant needs?
o Are the item viewer navigation functions appropriate to ensure easy navigation and ease of use?

Although the collections offered several major groupings of collections, only a few collections were
tested. Representative of most collections, the feedback from the testing resulted in the following overall
findings:

o UFDC as a resource should be explained on the homepage.
o Selecting separate collections to search is not intuitive.
o Collection information should appear in All Collection result lists.
o Result pages for individual terms found in serials does not allow for ease of access to appropriate
resources.
o Item viewer navigation functions need to be more visible.
o More instruction to use item viewer is necessary.

The following report will provide a detailed summary of the test procedures and findings.










Methodology

What happened during the usability test

The evaluation of the UFDC was conducted at the University of Florida in Gainesville, Florida. The
research methodology involved scenario-based testing and a test session-focus group session to evaluate
online participant search behavior. Three sets of test questions were used. All participants were given a
set of questions that considered the UFDC as a whole and two separate tests focused on either the
Baldwin Historical Children's Book collection or the Florida Digital Newspaper collection. These
collections were selected because they are representative of most collections and include most of the
functionality to test.

Participants of the full usability testing completed pre- and post- test questionnaires which were used to
determine experience using online library resources and their satisfaction with using UFDC. Participants
then completed a test session consisting of structured exercises using the all collections, basic and
advanced search interfaces; result pages and item viewer. The individual test sessions focused on basic
UFDC functionality and a series of tasks using All Collections and the Baldwin Historical Children's
Book collection (Baldwin). During the individual test sessions, the librarian asked participants to talk out
loud about their process to complete the question and what they expected would happen; responses were
written down by the librarian.

The test session-focus group discussion consisted of participants taking the pre-test questionnaire and
then completing the tasks which focused on All Collections and the Florida Digital Newspaper collection
(Florida Newspapers). The librarian and one student assistant asked participants working in groups to
talk out loud as they completed each task. The students then served as a focus group to discuss their
experience with UFDC, guided by the post-test questions that individual usability participants answered.


Who we tested

The individual usability testing took place between February 9 and February 19, 2007 and the test
session-focus group occurred on February 27, 2007.

Participants of the individual UFDC usability testing were recruited prior to testing. Participants were
selected from three categories of academic users who represented the subject areas the librarian felt would
be interested in the UFDC content tested.

Faculty participants were recruited by email to personal contacts of subject specialists in Mass
Communications and Education. Graduate and undergraduate participants were frequent library users who
responded to an open invitation by the librarian. Test session-focus group participants were graduate
students (in Education, Journalism, Educational Technology, Telecommunications and Law, etc.) who
volunteered along with a faculty member in Mass Communications.












Participants who evaluated UFDC have the following profile characteristics.

Individual Participants


Academic Status


Faculty 3
Graduate Student 1
Undergraduate Student 2
TOTAL 6


Test session-focus group Participants

Academic Status

Faculty 1
Graduate Student 16
Undergraduate Student 0
TOTAL 17


Area of Academic Interest


Natural Sciences 0
Social Sciences 3
Arts and Humanities 3
Law 0
Health Professions 0
TOTAL 6


Area of Academic Interest


Natural Sciences 1
Social Sciences 13
Arts and Humanities 1
Law 2
Health Professions 0
TOTAL 17


Resource use and searching skill levels are all self-reported values gathered from pre-test questionnaires.
All six of the individual participants have used the library catalog while only half have used digital
collections such as PALMM or the Library of Congress American Memory Project. Most often used
online resources used include Google, journal databases (e.g. JSTOR) and article databases (e.g.
EBSCOHost databases). Of the test session-focus group participants 94% have used the library catalog or
online resources, including Westlaw, LexisNexis Legal, Google, and JSTOR. Only 38% of participants
have used digital collections such as PALMM or the Library of Congress American Memory Project.
Most participants indicate that reasons for using these resources include the ease of navigation,
familiarity, speed of retrieved results and appropriateness of content covered in the resource.


Individual Participants


Online Research Levels


Test session-focus group Participants

Online Research Levels

10 9
9-
8
S7 expert
S6 m highly experienced
5 o proficient
S4- o novce
S3 no experience
2 1
1 0
Online Research Lewls
Online Research Levels


Online Research Levels


* expert
* highly experienced
o proficient
o novice
* no experience










Where we tested


In most cases, individual testing and observation was conducted in Library West or in the faculty
member's offices. The following is a summary of the participants' computing environment:


URL of tested website: [http://www.uflib.ufl.edu/UFDC/]
Computer platforms: [Dell Pentium IV with an 17" display]
Browser tested: [Mozilla Firefox]
Screen resolution: [1024 X 768]
Operating system: [Windows XP]
Connection speed: [Shared T1]


The test session-focus group usability testing was conducted in the Architecture and Fine Arts 120
computing classroom with the following:

URL of tested website: [http://www.uflib.ufl.edu/UFDC/]
Computer platforms: [450Mhz Macintosh G4]
Browser tested: [Mozilla Firefox]
Screen resolution: [1024 X 768]
Operating system: [OS X]
Connection speed: [Shared T1]

Individual test sessions responses were observed and recorded by the librarian by hand.


What data was collected

The librarian collected data that would address the major elements of the UFDC including overall
navigation and ease of use; searching features; result pages; and the item viewer. Other features not yet
implemented were also addressed.


IUniversity of Florida Digital Collections, Scenario-Based Testing


Initial Resource Impressions


At the beginning of each individual and test session-focus group session, we allowed the participants to
preview the resource for about 3 minutes. We asked participants the following four questions:

What are your initial impressions of the digital library?
What do you like about the resource? Why?
What do you dislike about the resource? Why?
What do you think you will find in this resource?
Who would use this resource?











What are your initial impressions of this resource?


The overall initial impression of the UFDC is that it is graphically simple and well organized. Most
participants indicated that the UFDC was appealing because of its banners and at least one participant
commented that UFDC appeared to be easy to browse and fairly easy to follow to the desire content.

From the brief searching allowed within the allotted time for initial perusal of UFDC, most participants
believed they would find graphical materials related to various fields; images of actual documents;
photographs and materials for Florida research topics. When asked what they liked about the resource,
most participants pointed out the multidisciplinary nature of the resource. When asked what they
disliked, most participants indicated that they needed a clearer explanation of what the resource provides.
They revealed that they would only spend a short time exploring it to find out what it contains and would
abandon the resource if their needs are not readily met. One participant did not like that the search results
page sometimes provided spotty information (e.g. not all items have a publication date).

When asked who would use this resource, participants noted that content is appropriate for Florida
researchers and historians; some UF faculty and graduate students; and outside researchers. Almost 92%
of participants said that it was unlikely that the average college student would use the UFDC. They
believed that UFDC would not be appropriate or reliable for their research. This is because some
participants were looking at collections not fully populated (maps, some Florida Law collections) or
because they rely on books or journal content found in subscription databases.


Scenario-Based Tasks


During the usability evaluation, participants were asked to complete a number of scenarios or "real-life"
tasks using the resource. The tasks were presented in order on index cards. The following questions were
asked of all participants.


# Scenario-Based Tasks for All Participants Issue Addressed

1 In what grouping does the Baldwin Library of General
Historical Children's Literature Collection fall? Navigation
Indicate roughly how many volumes are in this
collection.
2 How many collections are in the Arts, General
Humanities and Social Sciences grouping? Navigation
3 How many collections are in UFDC? General
Navigation
4 How many books have the word "baby" in its Guided/Advanced
title? Searching/Results
List
5 What color is the cover of the work Drayton Item Viewer
Hall by Julia A. Mathews? Navigation
6 What is the postmark date for the postcard Item Viewer
titled "Historic University of Tampa, Tampa, Functionality
Florida" from the Florida Photograph
Collections?










7 Name one UFDC collection that has items Keyword
about biscuits. Searching/Item
Viewer

8 Send an email to the UFDC to get help. General
Navigation/Item
Viewer
9 Do a single search the Florida Law Collections, Guided/Advanced
Florida Photograph Collections, and Samuel Searching
Proctor Oral History Collections for
"Gainesville, FL." How many results do you
retrieve? Search each collection independently
for "Gainesville, FL" and name the number of
results you retrieve.
10 For advanced searching of the UFDC Guided/Advanced
collections, what are the Boolean Searching
operators/symbols used?

The six individual test participants were asked the following questions.

# Scenario-Based Tasks Using Baldwin Issue Addressed
Historic Digital Children's Collection
1 On what page does the text for the work The General
goat and her kid begin? Navigation/Item
Viewer
2 On what page does chapter IX of The widow's Item Viewer
sixpence, or, Go thou and do likewise begin?
3 What is the last word of The widow's sixpence, Guided/Advanced
or, Go thou and do likewise? Searching
4 What is the attribution statement for The Results List/Item
widow's sixpence, or, Go thou and do likewise? Viewer
5 Does the UF Library have a print copy of The Item Viewer
widow's sixpence, or, Go thou and do likewise. Navigation
6 What image is on the title page of The widow's Item Viewer
sixpence, or, Go thou and do likewise? Functionality
7 What is on page 8 of Puss in boots? Guided/Advanced
Searching
8 Describe the back cover of Puss in boots? Item Viewer
Navigation
9 What is the series title for Puss in boots? Item Viewer
Navigation
10 How many results do you retrieve when you Item Viewer
search for "children"? How many when you Functionality
truncate the word to the root "child"?










The test session-focus group participants were broken into four small, collaborative groups of 4 or 5 to
answer the questions for all participants (above) and the Florida Digital Newspaper Library questions
(below). Once completed, each group had to clearly indicate on paper how they retrieved their results.
This part of the test-focus group session revealed how students approached the questions and how well
they were able to retrieve their results-either expected or not.


The following discussion uses percentages based on 10 test sessions of the six individual test participants
and four test session-focus groups.

Overall Navigation and Ease of Use/Help

Most participants thought the homepage for the UFDC contained banners that were visually appealing
and made them want to explore the resource.

Categories. Most participants or groups (90%) could determine what collection group the Baldwin
Library of Historical Children's Literature collection (Baldwin collection) was in. When asked to
indicate how many collections were in UFDC, 30% of participants entered into each collection to find
sub-collections and counted that number. Most participants were able to count the number of icons to


# Scenario-Based Tasks Using Florida Digital Issue Addressed
Newspaper Library
1 Browse all items in the Florida Digital General
Newspaper Library Navigation
2 How many Gainesville newspapers are in the Keyword
collection? Searching
3 How many pages are in the June 12, 1878 Guided/Advanced
Gainesville Times? Searching
4 Find the W. J. Epperson advertisement. He is a Item Viewer
dealer in what kinds of goods? Where is the
shop located?

5 On what pages does the word "Sparkman" Item Viewer
appear? Navigation
6 Zoom in to find the Volume and number for the Item Viewer
June 12, 1878 Gainesville Times. Functionality
7 Do a search of the term alligator in all of the Keyword
Florida Digital Newspaper Library. How many Searching
newspapers are returned?

8 Do a search of the term alligator as a title Advanced
search. How many newspapers are returned? Searching
9 Find the January 6, 2005 Baker County Press. Guided/Advanced
On what page do the obituaries begin? Searching/Item
Viewer
10 Find the December 6, 1947 Key West citizen. Guided/Advanced
To what position was Charles Taylor just Searching/Item
elected? Viewer










identify the number of collections per group from the UFDC homepage. Some participants thought that a
quick link function or smaller icons should be used so that more of the collections in individual groupings
can be seen immediately when they enter into UFDC.

Information about the UFDC. When asked how many volumes are in the Baldwin Collection, all
participants clicked on the collection homepage and read the description for it. 50% of participants test
session-focus group participants could determine the number of titles in the Newspaper collections.
Despite indicating that they would probably not read the description unless they were asked to, they
thought that the description for each collection is useful. No participant used the Statistics link at the
bottom of the page. At least one participant wanted to know what percentage of Baldwin has been
digitized and thought the information in the collection paragraph could be clearer.

General Navigation between collections. 40% of participants did not find it easy to navigate back to the
UFDC from the result pages or item viewer. Although a homepage link is in the upper right hand comer,
participants tended to miss it and returned to the Libraries' homepage using the browser's back button.

Specific Navigation. Navigation from the Item Viewer to collections or the Item Viewer back to results
page was a surprisingly difficult task for participants to complete. Only 60% used the link in the upper
right hand comer to return to collections or used the Last Search Results link from the item viewer
navigation pane to return to results. Most participants returned to the pages using the browser's back
button.

Help Pages. 90% of participants were able to send a help message. Participants found the Contact us link
from Item Viewer (40%), the Contact link at the bottom of each page (10%) and the Help pages (40%)
themselves.

Searching

All Collections Searching. Participants found that searching the interface was generally self explanatory
when asked to search an entire collection. The UFDC homepage was assumed to provide an all collection
search. However, when asked what collection held an item about biscuits, 40% of participants did a Food
and Agricultural Sciences collection search instead of an All Collection search, which retrieved zero
results. Possible methods to retrieve the correct result would be to enter an item and look at the Full
Citation view (0%) or if available, to look at the Copyright Notice at the beginning of the item (40% did
this; 20% did not complete the task).

Collections Searching. When asked to search within specific collections, 100% of participants could
enter the collection page and begin a search in the basic collection search interface. 100% of participants
could browse all items and could identify how results are sorted.

Sub-collections or Multiple Collections Searching. All participants were asked to do a single search of
Gainesville, FL in three collections (Florida Law Collections, Florida Photograph Collections, and
Samuel Proctor Oral History Collections) that they would need to select. Only 20% of participants were
able to complete this search accurately, deselecting collections from the All Collections Advanced Search
interface. Most participants either indicated that the task could not be accomplished (40%) or only did
individual searches of the collections (40%).

Searching Features. A number of searching features are explained on the Advanced search interface of
the UFDC. However, only 40% found and used either the Basic (not to be confused with the basic
collection search interface) or Advanced Search interface for searching. Those participants normally










located the Advanced Search interface only because they were asked to complete fielded searching
(usually to locate an item by title).

The participants had to search for "Gainesville, FL." Participants found this confusing and wondered
aloud about case sensitivity, phrase searching and what to do with punctuation. 100% of participants
provided answers to this question with several variants of Gainesville, FL and provided different answers
for each variant, in and out of punctuation, including:

Gainesville fl; Gainesville, fl; Gainesville Fl; Gainesville, Fl; Gainesville FL; Gainesville, FL

Results

All results retrieved were set to the default Table view. All participants thought this view was appropriate
for quick perusal of the results.

When looking for a particular book, 100% of participants could immediately find the results they needed.
Some participants did express that they did not want just the title, author and year on the result page.
They concluded after being asked to find the name of the collection that a result came from that collection
information be added to All Collections search result pages. Most participants were uncertain how to
locate the collection (in Full Citation) and 50% went back to their search to conduct a specific collection
search to ensure they had the name of a contributing collection. No participant tried the other result views
(brief or thumbnail).

Problems specifically with serial items. Serial searching was only tested by the four test session-focus
groups. The following discussion relates only to their test experiences. A major problem encountered by
participants from the result pages dealt with the listing of serials when a keyword or fielded search was
conducted. All newspaper titles were provided with the volumes and issues listed in chronological order.
When seeking specific resources with a keyword, participants thought this list is not manageable and to
one participant "simply makes no sense". For example, when asked to search for a specific term such as
alligator within the resource, participants were uncertain why they were pulling up full newspaper item
records with no apparent method to refine the list to locate just the specific issue with the term in it.

The participants expressed that the result list is unlike other result pages that they would retrieve from
commercial databases-that the results provide only those issues having the terms searched. They raised
the question of what constituted a matching record-if the results they are getting are retrieved from full
text searching or just the record. They indicated that this needs to be more fully explained, especially
since the results retrieved are unexpected and difficult to manage. Although a Search within document
feature is available, participants need to figure out manually which of results are the ones they need to
review within serials-a time consuming task that would, for most, cause them to give up and not return
to the resource.

In addition, only 25% of the participants could find newspapers published in an area or location and used
the title field to try to refine by location. Unfortunately, this did not appear to be the most accurate
method to retrieve these results.

Item Viewer

Once participants entered into a specific item, they were impressed and excited that the full text content or
image is available to them.











Left Navigation pane. Very few participants utilized the left navigation pane while in the item viewer to
navigate to other search options within UFDC, to retrieve full citation information (40%), to review the
table of contents (50%) or to navigate within the item retrieved. When asked to find which pages had a
particular term on it, 50% of participants discovered the Search within document feature. Despite
suggesting that it is a beneficial added feature, 100% suggested that, especially in newspaper searches,
enabling highlighted text would be more beneficial.

When asked to return to the collection's search interface or to the full UFDC search interface from item
viewer, participants tended to use the back button on their browsers instead of the links that were
provided in the navigation pane. Plus, with more information available in that panel, the thumbnail and
table of contents list get moved down in the navigational panel. As a result, almost no participants (30%)
utilized the thumbnail or table of content options. Both of these useful features were underutilized by
participants.

Navigating using Zooming Features. Most participants thought that the zoom features available are
intuitive and useful. However, when asked to find the postmark date for a the "Historic University of
Tampa, Tampa, Florida" postcard, 60% of participants tended to squint hard at the monitor or move the
monitor closer to them instead of using the zoom features available. 100% of participants used the arrows
to move the item to the desired location on an item. One participant suggested that being able to drag the
resource with the mouse would greatly enhance the experience with using image collections in UFDC.

Jump to Page Features. 100% of participants successfully used the view related images and the drop
down jump to navigate to specific pages required in a task or to see front and back of images.

Thumbnail. If a zoom function is employed, thumbnail language will appear that instructs participants
that they can re-center the page image from the thumbnail. However, the language to use the thumbnail
does not appear until a participant happens to click on a zooming feature. Most participants did not find
this "after the fact" instruction useful.

University of Florida Digital Collections, Post-Test Reactions/Focus Group Discussion


At the end of the scenario-based test sessions, individual participants were asked to complete a post-test
questionnaire. Test session-focus group participants were asked identical questions and offered more
about their experience using UFDC. The questions gauged how easy the UFDC was to learn and use.

Individual Participants n=6


somewhat
easy


Overall Ease of Use


Overall Navigability


somewhat
easy


very easy


difficult


very easy


difficult











Test session-focus group participants n=17

Overall Ease of Use Overall Navgability




a. a





very easy somewhat difficult very easy somewhat difficult
easy easy



When asked what parts of the UFDC should be improved, at least 50% suggested the following:

Navigation within resource
Search interface-advanced
Navigation between result pages and other parts of UFDC
Content
Serial result lists for more accurate retrieved items
Item collection information (if found in an All Collections UFDC search)
Page load times
Folder options or printing capability
More identifiable back button to navigate within searches and individual items

Participants were also asked to provide any additional information or comments not addressed in the post-
test sessions.
Participants thought that terminology "digital collections" was not descriptive or intuitive.
Participants were impressed that the resources were full content-even though some of the pages
were blank.
Participants thought that clicking on a location in Maps collections should take you to the maps
for the area.
Participants expect that the materials would be more research oriented, but found content to
highly specialized. Unfortunately, most of the participants did not find the collections to hold the
content that they would need to use for their own research. Many were disappointed to discover
that the collections they would most be interested in were not yet populated.
Participants stressed that the way UFDC searches should be explained (if terms are typed in what
is the implied Boolean operator-is it an automatic search placing an "and" or "or" between
terms).

The major findings of the test sessions and focus group are indicated in the Findings and
Recommendation section of this report.











IFindings & Recommendations


These findings and recommendations provide information from the scenario-based testing and post-test
questionnaires/focus group discussion. While participants thought it was attractive, the UFDC was
overall found to be difficult to understand, not immediately user-friendly and requires some modification
to increase ease of use. The following are specific issues and recommendations for UFDC modification
prior to the April launch.

Overall Navigation and Ease of Use/Help

Finding [#1]: More information about UFDC is needed on the homepage.
Comments/Supporting Evidence Recommendations
* Participants indicated that the terminology Include a paragraph at the top of the homepage
"digital collections" had different meanings to to describe how content is selected, where it
them. comes from and the major intent for use
Participants did not immediately understand Offer a list of major subject areas/genres/etc.
the range of resources-content and contained in UFDC on a separate page
publication type included.__

Finding [#2]: The current homepage layout does not adequately emphasize collections.
Comments/Supporting Evidence Recommendations
Some participants did not bother with Add links to the groupings in small descriptive
scrolling down the UFDC homepage to see paragraph on UFDC homepage
groupings or collections. Use smaller icons for each collection so that
more collections can be seen immediately when
they enter into the resource


Finding [#3]: Collection descriptions are not entirely accessible.
Comments/Supporting Evidence Recommendations
Participants thought that the descriptions for Some recommended that the use of bullets or
each collection was useful but did not spend lists be used, especially for long contributor lists
the time to read the collection main pages. Provide shorter descriptive paragraphs of
Participants believed the collection collections
descriptions are too text heavy. Provide percentage of physical collection
digitized (if not here, then on Statistics page)
Include list of sub-collection links and selection
boxes instead of keeping it in paragraph format

Finding [#4]: Links to Help, UFDC homepage and external resources are not obvious to the user.
Navigation between collections, result pages and item viewer could be improved.
Comments/Supporting Evidence Recommendations
Participants were uncertain if the Contact Us Recommend changing language to Ask a
form would reach an appropriate librarian. Librarian, Ask A Specialist, or Get further
Few participants navigated using the top assistance/help to encourage use
navigational links back to the All Collections Increase size of UFDC and Library Catalog
page. home links from Results pages











Finding [#5]: Customization, and other output options are desired to improve user experience.
Comments/Supporting Evidence Recommendations
* Participants enjoy various features from other When the software is capable, add a basket
online resources. feature as well as email, print and download
output options



Finding [#6]: Marketing tools can be used to help explain the resource.
Comments/Supporting Evidence Recommendations
* Participants believe only a few academic Market particular collections to appropriate
researchers will use the resource and participants. May sell the resource as something
marketing enables more visibility and use of "fun"
UFDC. Emphasize testimonials on use by patrons

Searching

Finding [#1]: All Collections and Advanced search interfaces only should be available.
Comments/Supporting Evidence Recommendations
* Participants did not successfully conduct Make the collection search interface the "Basic"
fielded searching in questions. Most search interface
participants conduct "title" searches in basic Remove the extra Basic search interface from
collection search interface. collections pages, making Advanced searching
* Participants ignored the Basic Search links on the preferred choice for searching
all collections pages.__

Finding [#2]: It is difficult to search multiple collections or specific sub-collections.
Comments/Supporting Evidence Recommendations
* Participants struggled with idea that each Allow selection boxes for multiple collections
collection is made up of sub-collections that from UFDC homepage
can be individually searched. Keep selection boxes for collections on
* Most participants entered into the collection Advanced search interface
page and tried to use a single sub-collection
hyperlink to search instead of going to the
Basic or Advanced search interfaces to search
multiple collections.

Finding [#3]: More advanced search features should be clearly explained from search interface and help
pages.
Comments/Supporting Evidence Recommendations

* Participants tried searching by variants of Update all pages to reflect and emphasize these
Gainesville, FL only to discover that they search options
would retrieve different results for each. Include information about punctuation
* Punctuation was a concern for all participants. Increase size of help links










Result Pages


Finding [#1]: All Serial titles and volumes appearing when a keyword search is conducted must be
improved.
Comments/Supporting Evidence Recommendations

* Participants did not find the particular issue Only retrieve titles and issues that include the
needed when doing a search of a term in term searched
newspaper searches. If not possible, have some explanatory
information about returned results for user

Finding [#2]: More information needs to appear in the result list table view for serials searching.
Comments/Supporting Evidence Recommendations

* Newspaper collection result lists did not Provide publication location for newspapers
provide test participants with sufficient Provide collection information from the table
information. view
Allow sorting by location
Ensure a collapsible view of issues within a title

Item Viewer


Finding [#1]: The left navigation pane needs to be more visible.
Comments/Supporting Evidence Recommendations

* Few participants used the left navigation pane Increase the size of the top links on the page or
for revising searches or searching within use a different color in the navigation pane to
document. distinguish it
* Full citation link wwa underutilized.


Finding [#2]: Display of the thumbnail, table of contents and contributors features need to be improved.


Comments/Supporting Evidence


* Few users utilized the thumbnail feature.
* Few users utilized the table of contents
feature, especially when a thumbnail image of
the item appeared.
* Participants wondered about icons for
contributors, but tended to ignore them.


Recommendations


* Include language above thumbnail that will be
visible before zoom features are used
* Use more precise language for re-centering
thumbnail image
* Enable mouse to drag/re-center image from
viewer
* Decrease the size of the thumbnail
* Increase visibility of the "Table of Contents"
heading as well as link in '\ c'\\ s"
* Create smaller icons for contributors or have a
heading called contributor to ensure that there
isn't confusion






18


Appendix:


* Assessment of the University of Florida Digital Collections: A Usability Test Protocol accepted by
the University of Florida Institutional Review Board through 12/12/07







Please type or word process the following information.

UFIRB #_2005-U-1121 PRINCIPAL INVESTIGATOR Marilyn Ochoa

PROTOCOL TITLE Assessment of the University of Florida Digital Collections: A Usability Test

Continuing Review/Study Closure Report

To help us keep our records current, please complete the following and return it to the UFIRB Office, PO Box 112250, Gainesville,
FL 32611-2250.

If you have any questions, please call 392-0433. (This form is also available at http://www.ortge.ufl.edu/irb/irb02.)

Check all items that apply to your protocol and provide requested information. (You may attach additional sheets if necessary.)

1. On what date did data collection begin? N/A

2. Please indicate the statement that best describes the status of this protocol:
a) I have completed work on this protocol. I will answer the remaining questions on this form to
enable the IRB office to officially close the protocol.
X b) I have not begun data collection. I plan to start on _1/1/07_.
Sc) Human participants are currently being recruited.
I have attached 3 clean copies of the current informed consent. Upon approval, the IRB will
stamp an expiration date on them, and return one copy for me to duplicate.
Sd) No further recruiting will occur after the expiration date. However, the data collected during
the research shall be analyzed.
Se) No further recruiting of participants will occur, but data collection will continue on at least
one participant.
Sf) No further recruiting will occur. All interventions are completed on all participants, but
follow-up is being conducted as described in the informed consent. These follow-up activities
are described as follows:


3. If there have been any additions or deletions to the list of researchers involved with this protocol, I have described the
reason for each change below and have updated the informed consent form to include only the current researcherss:

N/A
4. If the protocol is externally funded, and the information about sponsorship is not correct in the protocol, the revised IRB
form identifies the following sponsorss:
N/A

5. Please indicate the following:
Sa) The total number of participants recruited during the past year: N/

Sb) The total number of participants to be recruited during the next year: 15

Sc) I have recruited, or plan to recruit, more participants than originally estimated because:

d) I have recruited no participants because: the resource to be tested was not yet ready for review, due to not sufficiently
populating the database or adding necessary functionality to make testing worthwhile. As of November 2006, the resource is
sufficiently populated and has new features which changed the interface and result pages.


6. Was written informed consent required by the IRB and obtained from each participant? Yes X No

If the answer is YES, attach to this form a photocopy of the last signed consent that you obtained.
If the answer is NO, please explain the circumstances under which written informed consent was not obtained:
No testing was done.

7. Were any changes, however minor, made to your protocol last year? Yes X No

If YES, were those changes reviewed and approved by the IRB prior to their implementation? Yes No








If changes were made to the protocol that were not submitted to and approved by the IRB, please describe the revision and
explain why it was not sent to the IRB for review:






8. Did any unanticipated outcomes or adverse events occur the past year? Yes XNo

If YES, please indicate which ones were previously reported to the IRB:



9. Did any participant withdraw from this research project during the past year? Yes X No

If YES, please indicate the reason for EACH participant's withdrawal:



10. The research participants in this protocol include the following that are checked:
Infants Children Adolescents
XAdults Prisoners Pregnant women
Mentally or emotionally challenged individuals



11. The IRB, the University of Florida, and the Federal regulatory agencies consider continuing reviews to be extremely
important. Research studies are approved based on an estimated ratio of potential benefits to possible risks.

Comment on how your participants responded to the study and tolerated your interventions. (This item may be omitted only
if no data were collected on research participants in the last year.)
N/A


Give your opinion about any changes in the risk-benefit ratio. Is there any new information (e.g. alternative procedures, new
information published in the literature) that might affect the risk benefit ratio?
No changes will occur in the risk benefit ratio since only a few changes are being proposed.



If data from your study have been reported, attached one copy of each published report.
N/A


Principal Investigator:



Supervisor (of student PI)



Department Chair


Date



Date



Date


Thank you for completing this information. Please return this form to the UFIRB Office, PO Box 112250, Gainesville, FL 32611-
2250. If you have any questions, please call 392-0433.









Usability Test Participant Questionnaire


The purpose of this study is to examine information discovery and retrieval using the University of
Florida Digital Collections. With this questionnaire, we will gather information about your experience
with using online library resources. All responses are kept confidential. Please return this form to
Marilyn Ochoa (Smathers Library, Room 100) when finished.


Part I. Demographic Data.
Please check the blank which applies to you.
1. College level completed:
( ) 1st year ( ) 2nd year ( ) 3rd year
( ) Bachelors ( ) Masters ( ) Doctorate


( ) 4th year


2. Which of the following best describes the general area of your academic interest?
( ) Natural Sciences ( ) Social Sciences ( ) Arts and Humanities


( ) Other


3. What is your specific major and department?


4. What is your academic status: ( ) Undergraduate ( ) Graduate ( ) Faculty

Part II. Library Resources
1. Have you ever attended a general library orientation? ( ) Yes ( ) No

2. Have you ever used the Library Catalog or another online library database (e.g. LexisNexis,
Academic Search Premier)? ( ) Yes ( ) No


2.1. How often do you use these resources?
( ) daily ( ) weekly


( ) monthly


( ) yearly


3. Have you ever used a digital collection such as the PALMM collections or the Library of Congress
American Memory Project? ( ) Yes ( ) No


3.1. How often do you use these resources?
( ) daily ( ) weekly


( ) monthly


4. What online resources do you use most often for research purposes?

4.1. Please explain why (content, navigation, etc).

5. When using the library catalog:

5.1. do you usually locate the materials you are seeking? ( ) Yes


( ) yearly


( )No


5.2. you usually locate materials you are seeking:
( ) within 1-5 minutes ( ) between 6 and 10 minutes
( ) between 11 and 20 minutes ( ) after 21 minutes or more

6. How would you rate your overall level of experience with online library research?


very
experienced


somewhat
experienced


no
experience










Usability Test Participant Questionnaire, page 2.


7. What are some of the searching features of online resources that are most important to you? Check
all that apply.
ability to browse a full collection
ability to search specific fields
use of Boolean operators
use of truncation and wildcards
other, please specify:

8. What are some of the features of online resources that are most important to you? Check all that
apply.
Help screens
Simple navigation of resource
information about resource contents
Consistent navigation
Ability to retrieve a variety of formats (e.g. image, text)
Saving items to a basket
Other, please specify:

9. What are some of the important features you need to work with the results you retrieve? Please
rank each of the following on a scale from 1 (least important) to 5 (most important).
Result sorting/relevancy ranking
Ability to customize display
review of full item record
Ability to browse the fields
Ability to save results in an account
Marking records
Sprinting or mailing records
Sgo to a matched term within the document
select output of the retrieved results (e.g. text, image)
Persistent URLs back to the resource

Part III. UFDC Initial Impressions

1. What are your initial impressions of the digital library?


2. What do you like about the resource? Why?


3. What do you dislike about the resource? Why?


4. What do you think you will find in this resource?


5. Who would use this resource?


6. Who sponsors this site?










University of Florida Digital Collections

The purpose of this study is to examine information discovery and retrieval using the University of
Florida Digital Collections (UFDC). Please read and complete the following tasks that will be monitored
using screen capture software. Please note that this session should take roughly 20 minutes to complete.
All responses are kept confidential.

Please type in http://smathersnt25.uflib.ufl.edu/UFDC/UFDC.aspx to enter the UFDC main page.

Scenario-Based Tasks for All Participants
1. In what grouping does the Baldwin Library of Historical Children's Literature Collection fall?
Indicate roughly how many volumes are in this collection.

2. How many collections are in the Arts, Humanities and Social Sciences grouping?

3. How many collections are in UFDC?

4. How many books have the word "baby" in its title?

5. What color is the cover of the work Drayton Hall by Julia A. Mathews?

6. What is the postmark date for the postcard titled "Historic University of Tampa, Tampa,
Florida" from the Florida Photograph Collections?

7. Name one UFDC collection that has items about biscuits.

8. Send an email to the UFDC to get help.

9. Do a single search the Florida Law Collections, Florida Photograph Collections, and Samuel
Proctor Oral History Collections for "Gainesville, FL." How many results do you retrieve?
Search each collection independently for "Gainesville, FL" and name the number of results you
retrieve.

10. For advanced searching of the UFDC collections, what are the Boolean operators/symbols used?
Guided/Advanced Searching


Scenario-Based Tasks Using Baldwin Historic Digital Children's Collection
1. On what page does the text for the work The goat and her kid begin?

2. On what page does chapter IX of The widow's sixpence, or, Go thou and do likewise begin?

3. What is the last word of The widow's sixpence, or, Go thou and do likewise?

4. What is the attribution statement for The widow's sixpence, or, Go thou and do likewise?

5. Does the UF Library have a print copy of The widow's sixpence, or, Go thou and do likewise.

6. What image is on the title page of The widow's sixpence, or, Go thou and do likewise?

7. What is on page 8 of Puss in boots?










8. Describe the back cover of Puss in boots?

9. What is the series title for Puss in boots?

10. How many results do you retrieve when you search for "children"? How many when you
truncate the word to the root "child"?

Scenario-Based Tasks Using Florida Digital Newspaper Library
1. Browse all items in the Florida Digital Newspaper Library.

2. How many Gainesville newspapers are in the collection?

3. How many pages are in the June 12, 1878 Gainesville Times?

4. Find the W. J. Epperson advertisement. He is a dealer in what kinds of goods? Where is the
shop located?

5. On what pages does the word "Sparkman" appear?

6. Zoom in to find the Volume and number for the June 12, 1878 Gainesville Times.

7. Do a search of the term alligator in all of the Florida Digital Newspaper Library. How many
newspapers are returned?

8. Do a search of the term alligator as a title search. How many newspapers are returned?

9. Find the January 6, 2005 Baker County Press. On what page do the obituaries begin?

10. Find the December 6, 1947 Key West citizen. To what position was Charles Taylorjust elected?










Post Usability Testing Questionnaire

Thank you for participating in the usability test of the University of Florida Digital Collections. Please
complete this short questionnaire about the resource.

1. How would you rate the overall ease of use of the UFDC resource?
5 4 3 2 1
very somewhat difficult
easy easy

2. How would you rate the overall navigability of the resource?
5 4 3 2 1
very somewhat difficult
easy easy

3. How likely would you use this resource for online research?
5 4 3 2 1
very somewhat not
likely likely likely

4. How likely would you recommend this resource to someone for online research?
5 4 3 2 1
very somewhat not
likely likely likely

5. How would you rate your overall experience with the UFDC resource?
5 4 3 2 1
Excellent/ Good Poor/
Satisfying Frustrating

6. Which areas did you find easiest to navigate:
Collection pages individual item records
Search pages help pages
Result list pages item viewer

7. What were your impressions of the Help video?
5 4 3 2 1
Clear Confusing

5 4 3 2 1
Helpful Not Helpful

5 4 3 2 1
Adequate Inadequate
Information Information










8. What did you find to be the most exciting feature of the UFDC?

9. Please indicate which areas could be improved (provide more information if necessary):
Site design _content
Search interface (general) terminology
SBasic help pages
Advanced result pages
Navigation throughout site page loading
Navigation within result pages format output options
Other, please specify










Popup Survey

Thank you for using the University of Florida Digital Collections. Please complete this short
questionnaire about your experience with the resource.


1. Which of the following best describes your general area of interest?
( ) Natural Sciences ( ) Social Sciences ( ) Arts and Humanities ( ) Other

2. What is your affiliation with the University of Florida: ( ) Undergraduate ( ) Graduate ()
Faculty ( ) Staff ( ) Public

3. How did you learn about this library resource?
( ) link on library homepage ( ) Web search engine result
( ) word of mouth ( ) other advertisement

4. How would you rate the overall ease of use of the UFDC resource?
5 4 3 2 1
very easy somewhat easy difficult

5. How would you rate the overall navigability of the resource?
5 4 3 2 1
very easy somewhat easy difficult

6. How likely would you use this resource for online research again?
5 4 3 2 1
very likely somewhat likely not likely

7. Were you able to locate the information you need from the resource?
3 2 1
Yes Mostly Not at all

8. How likely would you recommend this resource to someone for online research?
5 4 3 2 1
very likely somewhat likely not likely

9. Would you rate your overall experience with the UFDC site?
5 4 3 2 1
Excellent Good Poor

10. Which areas did you find easiest to use:
Collection pages result list pages help pages
Search pages individual item viewer

11. Which areas did you find hardest to use:

Collection pages result list pages help pages
Search pages individual item viewer

12. Which UFDC collection were you using?

13. What improvements would you like to see?




University of Florida Home Page
© 2004 - 2010 University of Florida George A. Smathers Libraries.
All rights reserved.

Acceptable Use, Copyright, and Disclaimer Statement
Last updated October 10, 2010 - - mvs