Usability Test Report for MetaLib

MISSING IMAGE

Material Information

Title:
Usability Test Report for MetaLib
Physical Description:
Report
Creator:
Ochoa, Marilyn
Jesano, Rae
Nemmers, John
Newsom, Carrie
O'Brien, Maryellen
Victor, Paul
Publisher:
George A. Smathers Libraries, University of Florida
Place of Publication:
Gainesville, Fla.
Publication Date:

Record Information

Source Institution:
University of Florida Institutional Repository
Holding Location:
University of Florida
Rights Management:
All rights reserved by the source institution and holding location.
System ID:
AA00004289:00001


This item is only available as the following downloads:


Full Text





Usability Test Report for MetaLib


Date of Report:
Period of Testing:
Period of Focus Groups:


Prepared for:


Prepared by:


April 7, 2006
February 22-March 20, 2006
March 21-24, 2006


LeiLani Freund and Tom Minton, Co-Chairs
Resource Navigation Group


Martha Hruska
Associate Director


MetaLib Implementation Task Force
Marilyn Ochoa, Chair
Rae Jesano
John Nemmers
Carrie Newsom
Maryellen O'Brien
Paul Victor













Table of Contents

E executive Sum m ary ............................................................................................... ................ ..... 4

M eth o d olo gy ....................... ............................................................. .................. .................. ......... 5
W hat happened during the usability test.................................................... ....................................... 5
W ho w e tested .............. ........... ............ ......................................................................... ........... ...........
W h ere w e tested .............. .......... .................................................................. ............................................... 7
D ata w e collected .......... ....... ......................................................................................................... ..........7

Session I: University of Florida MetaLib Library Portal, Scenario-Based Testing............................ 8
Session I, Initial R source Im pressions ................................................. ............................................... 8
W hat are your initial impressions of this resource? ................................................ ................................ 8
W hat did you like about this resource?...................................................................... ................................ 8
W hat did you dislike about this resource? ...................................................................... ........................... 8
What type of information would you expect to find on this site? ........................................ ..................9
Session I, Scenario-Based Tasks.............................................................. .............................. .... 10
F in d D atab ases....... ......... .................................................................... ...... ............... ................. .....
Find E-Journals .............................. ................. 11
Searching and R results .. .................... .......................................................................... ........... ....................... 12
C custom ization/M y Space ....................... ................................................................................. ....................... 13
Overall Navigation and Ease of Use/Intuitiveness............................................................... ................. 13
Session I, P ost-T est R actions ...................... ..................................................................... ............. ....... 15

Session II: University of Florida MetaLib Library Portal, Focus Groups .......................................21

Findings & Recommendations .......................................................... ..................... 24
Overall ,1' ,, i ii and Ease of Use/Intuitiveness Issues ............................................ ................. 24
Finding [#1]: M etaLib is not intuitive............................................................................. .................. 24
Finding [#2]: MetaLib requires training and instruction and better help pages ......................................25
Finding [#3]: MetaLib interface needs consistency. ....................................................................................25
Finding [#4]: MetaLib should be seamlessly integrated with other library resources ................................25
Finding [#5]: MetaLib icons need consistency and explanation.......................................................26
F ind D databases Issues ... .. ............................... .. ... ......................................... .... ............................... .. 27
Finding [#6]: MetaLib can be used to replace the current Database Locator, with slight modifications ......27

F ind E -Journals Issues .......... ...... ........ ..................... ... ......... ........................ .......... ........ ............... 27
Finding [#7]: MetaLib could be used to replace the current E-Journals Locator, with slight modifications.
........................... ............................................................................................................................................ 2 7

Searching Issues ........................................ ......................... .......................... .. .... .. ............ .......... 28
Finding [#8]: Categories in QuickSearch should be modified......................................... ..................28
Finding [#9]: More Advanced search options should be included in QuickSearch ...................................29
Finding [#10]: Remove MetaSearch from the initial implementation of MetaLib. ....................................29







3


Results Issues ........................... ....................................... ................. ........................... 29
Finding [#11]: Modify navigational features on the Result pages.....................................................29
C .i 'I ,:,i1,, i S I s y Space Issues ...................................................................... ............. ..................... 30
Finding [#12]: Remove My Space from the initial implementation of MetaLib .......................................30
A ppendices ....................................................... .............................................. ................... 30
Assessment of Federated Searching using MetaLib: A Usability Test Protocol accepted by the University of
Florida Institutional Review Board










Executive Summary

The University of Florida is in the midst of migrating from NOTIS library management software
to library management software developed by the ExLibris Group. As part of this migration, the
Database Locator (which is supported by NOTIS) currently in use will be replaced with the
MetaLib Library Portal (MetaLib). MetaLib is a research tool that enables simultaneous
searching of multiple electronic resources from a single interface and provides links to library
resources' native interfaces. The MetaLib Implementation Task Force (Task Force) was
commissioned by the Resources Navigation Group to evaluate the effectiveness of MetaLib's
overall interface design, with a special emphasis on database categorization, in lieu of MetaLib's
debut in the fall of 2006.

In order to accomplish its task, the Task Force developed and administered usability testing to a
cross-section of university users. The testing was intended to examine how well the design of
the resource allows for ease of use by researchers, and how effectively MetaLib matches user
expectations and needs. In developing the usability test, the Task Force outlined key areas of
concern, which the testing was designed to answer:
When researchers encounter MetaLib, do they readily know what the resource can do for
them?
How easy is it to understand and use the available search features to locate materials?
Do users retrieve the anticipated results?
Is navigation through retrieved materials intuitive?

The usability testing resulted in the following overall findings:

All users had some level of difficulty utilizing MetaLib, resulting in very low overall task
performance;
Users mistook the "General Search" default QuickSet in QuickSearch for a Google-type
"search everything" option. Often users searched for e-joumals, databases, articles and
books using the QuickSearch;
Users had difficulty locating subject specific databases, such as the Project Starter
databases, under the "Find Database" tab;
Users wanted to search the online catalog from within MetaLib from any screen;
Users were often confused about what constituted an e-joumal versus a database;
Users were confounded by many of the icons utilized;
Users were confused by the navigational and display features in the Results pages; and
Users found the MetaSearch and My Space functionalities particularly perplexing and
difficult to use.

As a result of these findings, the Task Force makes the following general recommendations:

Placement of subject-specific QuickSets on the MetaLib homepage with descriptions, i.e.
use of the "Description" field, which displays text below the title.
Seamless integration of MetaLib with other library resources.
Consistent use of navigational functions/links.









Modification of icons.
Modification of the Results pages.
Fall 2006 implementation of the QuickSearch, Find E-Journal and Find Database
functions.
Modification of the My Space and MetaSearch functions.
Delayed launch of the MetaSearch and My Space tabs based on the significant difficulty
had by users in using these functions
Creation of instructional workshops, guides and online tutorials.

The following report will provide a detailed summary of the Task Force's testing procedures and
findings.


Methodology

What happened during the usability test

The MetaLib Implementation Task Force conducted a usability evaluation of the MetaLib
Library Portal at the University of Florida in Gainesville, FL. The research methodology
involved scenario-based testing to evaluate online user search behavior.

Participants of the usability testing completed pre- and post-test questionnaires which the task
force used to evaluate participants experience using online library resources and their satisfaction
with using MetaLib. Participants then completed a test session consisting of structured exercises
using the MetaLib search interfaces and result pages During the test session we asked
participants to talk out loud about their process towards finding an answer as well as their
expectations. Test sessions were recorded using Camtasia Studio 3.0, a software program that
captures cursor movements, navigation and audio. After the test sessions participants were
compensated with a $10 gift card to Borders and invited to participate in a focus group.

Each focus group consisted of 1-3 participants, a facilitator, and at least one member of the
MetaLib Task Force. The facilitator asked a series of questions2 while the Task Force member
took notes on the session. Participants were provided a pizza lunch during the focus groups.

The scenario-based testing took place from February 22-March 14, 2006 and focus groups were
held during the week of March 20, 2006.


Who we tested

Participants of the MetaLib usability testing were recruited prior to testing by email to listservs
and through personal contact. Participants were selected from three categories of library users



1 Details of the tasks completed are in the Appendix, page 8.
2 Details of the focus group questions are in the Appendix, page 12.











who provided a representation of each major subject area (natural sciences, social sciences, arts
and humanities, law, and health).

Fifteen participants, having the following profile characteristics, evaluated MetaLib. Resource
use, computer skill level and searching skill level are all self-reported values gathered from the
pre-test questionnaires.


Academic Status

Faculty 4
Graduate Student 6
Undergraduate Student 5
TOTAL 15




Library catalog use


yearly

monthly

weekly

daily

I I I I II
0 1 2 3 4 5 6 7

# respondents

E-journal use


never
yearly
monthly
weekly
daily


Area of Academic Interest

Natural Sciences 3
Social Sciences 4
Arts and Humanities 3
Law 3
Health Professions 2
TOTAL 15


never
yearly


weekly
daily









never
yearly
monthly
weekly
daily


Database use



*


*




0 1 2 3 4 5 6 7

# respondents

Database locator use



*




*


# respondents


# respondents













expert
highly experienced
proficient
novice
no experience


Computer skill level








0 2 4 6 8 10


expert
highly experienced
proficient
novice
no experience


# respondents


Searching skill level








0 1 2 3 4 5 6 7


# respondents


Where we tested


Following is a summary of the participants' computing environment:

URL of tested website: [http://metalib.fcla.edu]
Computer platforms: [Dell Pentium IV with an 17"
display]
Browser tested: [Mozilla Firefox]
Screen resolution: [1024 X 768]
Operating system: [Windows XP]
Connection speed: [Shared T1]


In most cases, testing and observation were completed in the Marston Science Library electronic
classroom. Law and Health affiliated participants were tested in on-site conference rooms using a
Library-owned laptop with the following:


Data we collected


We sought to collect data that would address the major components of the resource gateway,
including the Overall Navigation and Ease of Use/Intuitiveness, Find Databases, Find E-
Journals, Searching, Result pages, and Customization/My Space features.


URL of tested website: [http://metalib.fcla.edu]
Computer platforms: Pentium M 713 1.1 GHz ULV
Browser tested: [Mozilla Firefox]
Screen resolution: [1024 X 768]
Operating system: [Windows XP Professional]
Connection speed: EtherNet, WiFi (wireless), and
56Kbps modem










Session I: University of Florida MetaLib Library Portal, Scenario-Based Testing


Session I, Initial Resource Impressions


At the beginning of each individual scenario-based test session, we allowed the participants to
preview the resource. We asked participants the following four questions:


What are your initial impressions of this resource?
What did you like about this resource?
What did you dislike about this resource?
What type of information would you expect to find on this site?


What are your initial impressions of this resource?
Thought the resource was organized and cleaner/aesthetically more pleasing/simpler to use than
current system
Liked the possibility of using the history information and customization features
Believed that QuickSearch would make searching easier
Was impressed by the possibility of some de-duplication of results
Believed that MetaLib needs a homepage with a description of what the resource will provide
"Some points were flawless but others made me very sad: navigation through site, QuickSearch
and finding full text"
Need to increase the wideness of the site/database areas, need consistency in the color of the links
(use of gray bars that make it difficult to see the links: seem like unavailable links)
Use of Boolean operators but addition of without
Need more information about what MetaSearch is (only a few effectively used this)


What did you like about this resource?
Use of tabs to separate different searching options: database and journals
Alphabetical list for e-Joumals and databases to navigate
Simplifies finding e-Journals than Serials Solutions
Other resources such as Google Scholar are available to search as well
One-stop shopping combining various resources to search together
Potentially very useful because a larger body of resources are available to search simultaneously


What did you dislike about this resource?
Does not understand why there is a 15 result limit per database and how they are selected
Would prefer to see the next and previous buttons on both the top and bottom of the results pages
Found that in some cases the next link will take users to next record and not next page of the
result lists
Inconsistent terminology and action icons throughout the resource (e.g. My Space, basket and e-
shelf)
Icons need a key or use of text other than the title hover and need to be consistent
Lack of an explanation of how ranking is determined
Does not understand what each component of the resource is prior to using it
Seemed that the QuickSearch Advanced features had a limited number of fields






9


Did not like to be linked to another resource/search engine
Did not know when the search was running because the "searching indicator" is small


What type of information would you expect to find on this site?
Common answers included:
Articles but not books
Journals, books, and articles
Want to find books in collection with call number available
Quick access to the SUS catalogs












Session I, Scenario-Based Tasks


During the usability evaluation, participants were then asked to complete a number of scenarios
or "real-life" tasks using the resource. The tasks, which were presented to participants in order
on index cards, addressed one or more issues.

# Scenario-Based Tasks Issue
Addressed
1 How many databases are categorized as Project Starter databases? Find Databases
2 Find the database "Academic Search Premier." How many articles are available Find Databases/
for "mouse physiology?" Find the full-text for the first article in the result set. Searching/
Results
3 What is the time-span of coverage for the database Plant Science? Find Databases
4 What is the full name of the database abbreviated EEBO? Find Databases
5 Does the Library have a print subscription to the journal Biology of the Cell? Is Find E-Journals/
the journal available electronically? Searching
6 Do a QuickSearch of the term "cargo cult." How many records are retrieved? Searching/
How many records are displayed on your screen? Did you retrieve any results Results
from the Library Catalog? How many? How are the records sorted?
6 Find the most recently published material from your result list of the search Results/
"cargo cult." What database or resource did it come from? Add the article to My Space
your My Space.
7 Perform a search on "Medicare and prescriptions" in Health Business Full Text Find Databases/
through the MetaLib interface. How many results did you yield? Go to the Searching
normal/native database interface for Health Business Full Text. Repeat the search Results
and indicate how many results you retrieved?
8 Add the databases PsycINFO and Dissertations and Theses (ProQuest) to your Find Databases/
Sets. Create the Set called "Psychology" and add those two databases to it. My Space/
Using the QuickSet "Psychology," search for "searching behavior." How many Searching/
results are found and how many are retrieved. From which database does the top Results
ranked article come?
9 Log out of MetaLib and then log back in to MetaLib. Review your e-Shelf My Space
citations. Are your QuickSets still available?
10 You need to write a paper on a criminal law topic and your professor suggests the Find E-Journals
2003 article entitled "Shame, Stigma and Crime: Evaluating the Efficacy of
Shaming Sanctions in Criminal Law." Its citation is: 116 Harvard Law Review
2186. Locate the Harvard Law Review.
10 Now locate the article cited above. Can you view the full text of the article? Results/
If you wanted to read the article later, would you be able to email it to yourself? My Space
Add the article to your my Space.
11 What are the four recommended databases to search if you are looking for articles Find Databases
about Biochemistry?
12 How can you simultaneously search these four databases for "pine beetle?" Searching









Find Databases
Most participants easily used MetaLib as a database locator, specifically when looking for
known resources. They had little/no trouble using the alphabetical list and search options in Find
Databases. However, many of the participants had difficulty locating databases by category.
When asked how many databases are categorized as Project Starter databases, 54% of the
participants could not accomplish this task, primarily because they did not see or understand the
"Category" tab. Even some of those participants who did correctly answer this question stated
that they didn't quite understand the "Project Starter" terminology and probably would never
have found it or used it on their own.

When asked to locate databases under the Biochemistry category, 85% were not able to complete
the task. Most tried the alphabetic list or title search in Find Databases-Locate to determine if
any database titles contained the term "biochemistry." When this didn't work, most gave up or
tried to conduct a QuickSearch to see if anything came up for that term. Only one participant
completed this task easily by using the Category tab, and another person eventually found the
Category tab solution but only after much searching. Even for these successful participants, they
first tried to find the subcategory in Medicine/Health or Physical Sciences, and they did not
consider first searching in Agricultural and Biological Sciences. Some participants suggested
that we consider changing "Category" to "Find by Subject/Topic".

When searching for known databases such as EEBO or ASP, the participants were particularly
successful using the alphabetic list or the Find Database search options (i.e., "starts with" or
"contains"). 100% of the participants were able to locate Academic Search Premiere, and only
one participant failed to locate EEBO.

Once the participants successfully located a database, the preference was simply to click to the
native interface rather than to remain in MetaLib to conduct searches or to learn more about the
database. When asked to locate and query Academic Search Premiere, 76% of the participants
used the native interface and only 24% used the MetaLib interface. When asked to identify the
dates of coverage for the Plant Science database, 69% went to the native interface and only 31%
used the MetaLib information icon to answer the question.

Find E-Journals
Most of the participants were able to use the E-Journals feature as easily as they used Find
Databases to locate electronic resources. When asked to locate the Harvard Law Review, for
example, 77% easily located the journal (or found it after a small amount of difficulty), and only
23% could not find it.

When asked if we have both print and electronic versions of the journal Biology of the Cell, most
of the participants (77%) were able to locate the electronic version using the E-Journals
alphabetic list or search feature. They could tell that the journal was available electronically, but
could not determine how to find the dates of coverage. In addition, many of them had difficulty
figuring out how to search the OPAC to locate print holdings and stated that they would probably
just leave MetaLib and open the Catalog to discover print holdings. Please note that the UF
Libraries Catalog failed in 3 of 15 sessions.









The participants also demonstrated uncertainty about using the SFX link; most participants stated
that they were not sure what it does. Even those participants who used SFX weren't confident in
using it (e.g., some users could not find the print version of Biology of the Cell despite seeing the
UF Library Catalog link in the SFX window for that resource).

Some participants used QuickSearch to locate a journal title. The default General Search was
usually searched, but it is not certain how many users thought about what they were using. Very
few used the Advanced title in QuickSearch, but those who did were uncertain if title could be
used for journal title (as opposed to ISSN).

Searching and Results
100% of the participants were able to easily use QuickSearch to conduct general searches, but
they had more trouble understanding and using search results. Only a handful of participants
used MetaSearch with any success but it was obvious that they preferred to use the simpler
QuickSearch feature.

After completing a QuickSearch or MetaSearch, some users thought that the initial results screen
(i.e., the Summary page that lists result totals by each resource searched) was confusing. Some
questioned the discrepancy between found and retrieved results and the lack of a total count of
resources retrieved. A couple of participants wanted to have a "View Results" link beside each
resource so that she could just look at those results, as opposed to one combined "View Results"
link for all of the resources. One participant pointed out that this was true for MetaSearch but not
for QuickSearch. Some users thought that the "View Retrieved" link was not easily noticeable at
the top of the page.

Most users could not immediately tell how the retrieved records were sorted. Although most
were able to identify the initial sort as by rank, many expressed confusion as to what this meant.
They did not think the green bar was useful to help determine how they were sorted, especially
without a numerical percentage provided. Also, they were confused when duplicate results were
not sorted together in the result list (in one case, the duplicate item was three items below the
first mention of the item; it appeared that the article was sorted that way due to its database).
62% were unable to sort by date, primarily because they did not even see the "Sort" box on the
page. Further, when sorted by year, it was uncertain if results were alphabetically sorted by title
after the initial date sort. Some participants indicated that they were surprised that sorting could
not be done using the column title.

When asked to conduct a specific search in the Health Business Full Text database using the
MetaLib interface rather than the native interface, 46% could not figure out how to search within
a specific database using MetaLib. Only 31% easily completed this task, while the other 23%
insisted on using the native interface rather than MetaLib. Participants who successfully used
MetaLib to search were asked to repeat their search of the Health Business Full Text database
using the native database interface, but not one of the participants noticed any discrepancy
between the results retrieved even though there were significant discrepancies produced by many
of the searches.









Several participants expressed a desire to have more QuickSearch options. Many proposed that
the statewide union database should be one QuickSearch option, and the UF Catalog should be
its own option as well.

Customization/My Space
By far, the My Space feature caused the most difficulty for participants. Even when successfully
saving resources to My Space or successfully locating and using those saved resources, a large
majority of the participants expressed confusion and frustration. They understood how My Space
could be useful (and many said that they would love to use the feature), but they thought it was
very difficult to use.

When asked to add an article to My Space, 54% of the participants had difficulty doing this (e.g.,
they couldn't find the icon immediately, they weren't sure it had worked once they clicked it, and
many didn't even understand what they had just done). 15% of the participants were unable to
add an article to My Space.

When asked to add two databases to My Space and save them as a QuickSet, the participants met
with quite a bit of difficulty and confusion. Only one participant easily completed the task. 46%
completed the task but had trouble doing so and many weren't quite sure that they had succeeded
(i.e., once they had clicked on the icon they wanted some sort of feedback that the databases
were successfully saved). Another 46% had considerable difficulty and were unable to complete
the task. The most common problem was that participants simply could not locate the "My
Databases" tab under My Space. Many participants even stated that they had completely missed
the menu containing My Sets, My Databases, etc., and requested that it be larger or more visible
in some way.

The problems continued when the participants were asked to use the newly created QuickSet to
conduct a search. 38% were able to search the QuickSet, but the majority of the participants had
to skip this task or was unable to complete it. Because there are no instructions on the page, most
users thought they could stay in the My Space tab to search their databases. Frustratingly, at least
4 people indicated that they were "so lost right now."

The participants expressed frustration that the terminology was inconsistent between "My
Space", adding to the "basket", e-shelf, the shopping cart icon, "My Sets", etc. Also, the minor
change that occurs to the icons for adding to My Space was not effective.

All of the participants were able to log out and log back in to review their My Space resources.
The history information seemed not to be working for some of the participants.

Overall Navigation and Ease of Use/Intuitiveness
The participants had numerous comments and concerns about navigation and usability of
MetaLib, in addition to those issues described in the sections above pertaining to specific
features of the application.

Many participants requested relatively simple changes that would make the application much
easier to use. For example, they requested that all Next/Previous page links appear at the top and









bottom of each page. Also, they requested that column headings in the results pages should be
made links that re-sort the results according to the column heading clicked. Participants wanted
more consistency in terminology, particularly related to My Space (e.g., e-shelf, basket,
clipboard, etc.). Many also suggested that icons should be explained/identified more easily,
either by using some kind of key or by using text instead.

Many participants had difficulty finding menu tabs such as the Category tab under Find Database
or the My Databases tab under My Space. Many of them simply didn't see these menu options,
and they need to be modified to make them more recognizable.

Most participants preferred to conduct searches using the database native interface even for those
questions specifically asking them to search using MetaLib. Most ignored the
Search/Information icons until a specific task forced them to figure out what they were (e.g., for
the date span coverage question, almost all of the participants went to the native interface to
answer it and simply ignored the "I" icon).

Many patrons encountered a problem when they opened a database in a new browser window
and then returned to MetaLib without closing that window. When they subsequently found
another database and tried to open it, it appeared that nothing was happening. Actually, the new
database was opening in the already-opened window but they couldn't tell because the focus
didn't change to that window. Some participants asked if we could force the focus to be on the
newly opened resource even if the browser window is already open behind the MetaLib window.

More advanced features like My Space and MetaSearch posed the most difficulty for
participants. They did not find these features to be intuitive at all, and expressed great frustration
because they really were interested in using the features but would require instruction before they
could use them with any confidence.













Session I, Post-Test Reactions


At the end of each session, we asked participants to complete a post-test questionnaire. The
questions were useful in gauging how easy the resource was to learn and to use. Overall
navigation, ease of use, intuitiveness, satisfaction, and results were considered. We asked the
participants to rank their responses to the following questions:

Overall, how easy was it to use MetaLib?


Easy to Use MetaLib


4 3
- Somewhat Easy


* How easy was it to use the My Space feature?


Easy to Use My Space


3
Somewhat Easy


1
- Difficult


Very Easy


2 1
Difficult


5
Very Easy












* How would you rate the option to create your own QuickSets?


Ability to Create QuickSets


8
7
6
c5
4
3
0. 2
1
0


5
Useful


4


* How easy was it to create your own QuickSets?


Easy to Create QuickSets


5

4

.3

S2

1

0


5 4
Very Easy


3
Somewhat Easy


2 1
Difficult


2 1
Unnecessary







17



* How easy was it to determine how many records were retrieved by searches you conducted?



Easy to Determine Records Retrieved


5 4 3
Very Easy Somewhat Easy


* How easy was it to determine how results were ranked?



Easy to Determine Results Ranking


7
6
S5


3-
. 2
1
0-


2 1
Difficult


5 4
Very Easy


-
3 2 1
Somewhat Easy Difficult







18


* How easy was it to change the sort order of the results?


Easy to Change Sort Order


6
5
S4
S3
S2


o1 0 I ,1 I!
5 4 3 2
Very Easy Somewhat Easy



* How likely are you to use this resource for online research?


Likely to Use MetaLib for Research


6
5
. 4
S3
. 2
1
0


5 4
Very Likely


3
Somewhat Likely


2 1
Unlikely


1
Difficult







19


* How likely are you to recommend this resource to someone for online research?


Likely to Recommend MetaLib to Others


7
6
S5
S4
0.
S3

1 -
0


5 4 3
Very Likely Somewhat Likely


* How would you rate your overall experience with MetaLib?


Overall Experience with MetaLib


9
8
7
S6
.5
' 4
. 3
2
1
0


2 1
Unlikely


5 4 3 2 1
Excellent/Satisfying Good Poor/Frustrating










In addition, we had participants answer the following, after providing a list of options from
which to select.

Which areas did you find most difficult to navigate?


Most Difficult Areas

Results Pages 1
Individual Results/Finding Full Text
My Space 9
MetaSearch
Find E-Journals 0
Find Databases 1
QuickSearch

0 2 4 6 8 10


Please indicate which areas could be improved (provide more information if necessary)

8 participants identified "Terminology" as the area needing most improvement and 6
participants selected "Navigation Throughout Site". 4 participants picked each of the
following: Site Design, Results Pages, and Navigation within Results Pages.

Does it concern you that you may not be retrieving all records matching your search?

8 respondents said that it did concern them, and 4 respondents said that it did not
concern them.












Session II: University of Florida MetaLib Library Portal, Focus Groups

After completing the scenario-based testing, participants were encouraged to continue to use
MetaLib and then participate in a focus group. The following questions were asked to provide
more information about various parts of the resource that were not covered by the post-test
questionnaire that was taken.

Focus Group Questions
After continued use of MetaLib, what features of the resource would you definitely use? Why?
After continued use of MetaLib, what features of the resource would you definitely NOT use? Why?
Why would you want to use this resource-to perform a QuickSearch on a single subject or if you
wanted to search several databases together?
Would you use this resource?
How important is it for you to find scholarly or academic articles? Does MetaLib provide you the
types of resources you want to find? Have you ever used Google Scholar? If you had a choice,
would you use MetaLib or Google Scholar?
The MetaLib interface can be positioned on various parts of the website. Choose your preference of
location-on the homepage or as a sub-page of the website. Explain why.
How and from where did you connect to MetaLib after the test sessions and prior to this focus
group?
Were results pulled up in a reasonable timeframe (seconds versus minutes?)? Did you have any
trouble retrieving results due to connection speed?
Was it difficult to figure out how to search from within a database or an e-joumal? What could
make this easier?
Would you use the QuickSets that we have in place now?
How easy was it to create your own personalized QuickSets? Would you recommend other
resources to add to the QuickSets, or broad areas to create a QuickSet of databases?
When you did a search of multiple databases, were you concerned that you might not have retrieved
all the records you normally would have if you searched those databases separately? Would this be
a concern to you that you would retry the searches in each individual database?
When performing searches in the library catalog or an online search engine, do you care how the
results are sorted when displayed (e.g., by relevancy, by title, by author)? Explain why.
When performing searches in the library catalog or an online search engine, do you care how the
results are ranked according to relevancy? Explain why.
Do you have experience using SFX?
Does it matter to you that you cannot print or E-mail your saved citations/records within the e-shelf
section?
When saving search results in My Space, was it intuitive or easy to use?
Did you have any strange experiences with your saved citations/records after logging off or
returning to your MetaLib account later on?

During the focus groups, most participants agreed MetaLib has many uses, but thought it was an
extremely non-intuitive library resource. Participants suggested that while they may use
MetaLib at some point in their research, they would not claim to fully understand what they were
doing. More experienced researchers suggested they might use it to get started on new or quick
research but then would move on to the native databases for more in-depth research.










MetaLib requires instruction, tutorials and better help pages. The icons used throughout the
resource require a key to explain their functions. Students suggested a welcome page or tutorial
to explain how to find articles, databases, and other resources through MetaLib. A faculty
member recommended adding instructional tips on the search interface itself, such as showing
users how to type in an author search (e.g. last name first or first name first). Further, some
participants noted that certain help sections used the term it was defining to define the term itself;
"Category" is one example of this circular and non-useful help. Rewriting the help screens to
avoid this was recommended.

Adding help screens to explain the use of MetaLib's QuickSearch and QuickSet features would
improve usability. Users did not fully understand the purpose of the Library-selected
QuickSearch QuickSets. One user believed that using the General Search QuickSet would allow
him to search all the resources to which the Libraries subscribed; some thought these predefined
QuickSets (i.e. General Search, Health Resources, Quick Facts) provided all relevant databases
in those subject areas. Other users questioned why there were so few QuickSearch QuickSets;
many asked for more QuickSets to be developed. Some thought the option to select any of the
major subject areas (such as those listed on the Library's current "Databases by Subject"
webpage) should be available in the QuickSearch tab. Further, QuickSets for the UF Library
Catalog and the Union Catalog for the State Universities System and for specific geographic
regions/area studies (or at least the ability to limit to these regions) were recommended. One
user noted that the "Link to" message received when attempting to search a resource not meta-
searchable is useless.

Apart from using these Library-selected QuickSets, participants commented that being able to
create their own QuickSets is an attractive feature of MetaLib. However, they thought the My
Space feature to create them is too difficult to use. Many participants could add items to My
Space but did not know what to do after that. They wanted explicit instructions on how to set up
QuickSets in the form of on-screen hints/note boxes in the empty space on the right side of the
page or via an online tutorial. One student suggested that a link saying, "Now click here to
search this set" should appear after the personalized QuickSet is created. Further, most
participants did not see the secondary menu bar in My Space (e.g. the bar contains e-shelf, My
Databases, etc.); these submenus should be more noticeable in order to be useful. Users also
expressed the need to email, print, export, or save full citations.

When given the choice, participants said they would use QuickSearch before MetaSearch. Most
participants found MetaSearch confusing or inappropriate for their research needs. Most of the
experienced researchers said they would not use it because they prefer to use the native database
interface that gives them more control over the fields they can search or because some of the
database resources they want to use are not available for metasearching. Other participants
suggested they might consider using MetaSearch if the limited number of meta-searchable
databases is increased.

In discussing user access to MetaLib, some users said they would use a drop-down box feature
on the Library's homepage to access the resource. This fulfills at least four participants' desire
to have "everything on one page" and one undergraduate's concern that the tabs should not be









necessary since they are so confusing. Other users said they would prefer going into the full
MetaLib interface first to run a search. Further, some faculty members expressed their hope that
the Libraries could investigate the option of having hyperlinks to resources like MetaLib or the
Library Catalog from other library resources (e.g. databases and the library website). In addition,
a hyperlink from the Remote Logon confirmation page is also necessary.

Participants expressed concern about the discrepancy between hits returned in MetaLib and hits
returned in native interfaces. However, most admitted they would not notice the discrepancy on
their own (they would trust MetaLib to retrieve everything). They said generally this would not
be too much trouble for them, since they would just use MetaLib to get started in research and
then go to specific databases and use native interfaces to more thoroughly conduct thorough
research.

Users did not like the "jump to" box on the result pages and recommended links at the bottom of
the page which provide page numbers they can select (e.g. "page 1 2 3 4 5 6 etc"). In addition,
most participants wanted Next and Previous links to be on the top and bottom of each result
page. Most users wanted the ability to sort by clicking on the column headings in the results
view. They liked the rank default but they wanted to know what the rank sort meant.

Few users knew what SFX means but some used it in the scenario-based testing or in their own
research. These participants generally understand what it could do for them, but have used it
without understanding the terminology "SFX"; participants suggested wording including "Find it
@ UF" and "Find Full-text". In some cases, SFX screens were noted to convey little vital
information, such as available dates of coverage.












Findings & Recommendations

These findings and recommendations provide information from the Scenario-based testing as
well as the focus group meetings conducted. Overall the MetaLib interface is not intuitive, not
user friendly and requires significant modification. Following are specific issues and
recommendations for implementing MetaLib. While we can make certain display and interface
changes they do not fully address the problems users encountered. The findings and
recommendations from this usability study should be submitted to ExLibris.


Overall Navigation and
Ease of Use/Intuitiveness Issues


Finding [#1]: MetaLib is not intuitive.


* Three users indicated that they would
probably not use the resource because they
could not figure it out.
* Most users did not understand the purpose
of MetaSearch, and 6 users identified it as
one of the most difficult features to use (9
users identified My Space as most
difficult).
* Several participants recommended
"everything should be on one page."
* Users suggested that remembering another
password would be cumbersome.


* Use the Gatorlink logon information
* Provide better help and instructional
information (see Finding #2)


Comments/Supporting Evidence I Recommendations










Finding [#2]: MetaLib requires training and instruction and better help pages.


Comments/Supporting Evidence
* Users would more likely rate the
experience with MetaLib higher if they had
training and introduction to it.
* Although most users will be able to use the
Find Databases/E-Jourals features without
assistance, most participants clearly would
have benefited from instruction/training
before using My Space and MetaSearch
features.
* Many used the link to the help, but could
not find the exact information sought:
different forms of the term that was
being looked up was used
often used the term itself to define what
it was


Recommendations
* Create an instructional homepage or
tutorials and a plan for standard
instructional materials and scripts
* Modify Help pages


Finding [#3]: MetaLib interface needs consistency.

Comments/Supporting Evidence I Recommendations


* 8 participants identified "Terminology" as
the area needing most improvement and 6
participants selected "Navigation
Throughout Site." Many specifically
complained about inconsistencies that
should be easy to fix.
* Users could not locate navigational links
where they expected them to be.
* The hyperlinks on the menu bars were
difficult to see due to use of the same color
for text and links.


* Use consistent terminology
* Put sub-menus, drop-down menus, and
Next/Previous links in the same places
on every page
* Place all Next-Previous links at the top
AND bottom of pages for database/e-
journal alphabetic lists and search results
* Use a different color for links in the
menu bar than for other links throughout
MetaLib


Finding [#4]: MetaLib should be seamlessly integrated with other library resources.

Comments/Supporting Evidence Recommendations
* Many participants in the focus group stated Place MetaLib on the main library page,
that they would use a MetaLib search box implemented in a "Google-type"
on the library's home page. presentation
* Participants wanted the ability to search the Add a link to the library's homepage,
catalog and the SUS union catalog within preferably in the UF Library banner
MetaLib. Investigate providing links to the library
* Participants wanted the ability to switch homepage and catalog from all library
between multiple library resources easily resources including databases
(i.e. databases, the library catalog, and
MetaLib)










Finding [#5]: MetaLib icons need consistency and explanation.


Comments/Supporting Evidence
* Users were confused by the slight icon
change once the icon was clicked on (i.e.,
they weren't sure that the resources had been
added). Some users saw that the icon for
adding articles changed slightly, and the title
hover read that it had been added, but usually
users did not notice the transfer.
* Icons are not intuitive. Users did not know
immediately that the basket was the icon
used to select an item to save it.
* Some people used the SFX icon but
mentioned that they really didn't understand
what it stood for.
* Many users ignored icons completely at first
and many never used some of the icons.


Usw. -ulLj i .achI. 6,J ca.~r. wid I r fid
ytirkS C-r :&h, rfcdIrmkIn .Nr a tN
TO sr, selec onet s.e ot r e by
Jiiigni i rali., but-)n Thr, erk car pL or
nyve earthh leterifa d cFck W


Te.~cru trbo re~oijree oull t-
search~r. dcli c, th rname & tIh set
e~Y rhar IrJ:~=ieIh ,Fns inr~Ljd- orl,
Fe o nwny rs srei in each disrone. T,
ficrd a m ore moriPlket hst of$i e
b-- subiet v keyWa'd in FirIE F,:coxcos
Icon Kiey

C L kkto secret Qb.7 e attoseabch.


Recommendations
* Ensure that all My Space icons (e.g. add
to basket, add to e-shelf) all use the same
icon and language
* Ensure icons like the basket and the +
add icon, change more noticeable than it
currently does (e.g. change color of the
icon)
* Allow that a second click on an icon will
deselect an item that goes to My Space
* Add a key/notes box to the empty space
on the search page, similar to Harvard
Libraries' QuickSearch page (see image
below). Also consider the use of text in
lieu of some icons. Change the SFX icon
to Find it @ UF or Find Full Text icons
and text


This key/notes box from
the Harvard Library's
MetaLib project
provides instructional
information as well as a
key to the icons.











Find Databases Issues


Finding [#6]: MetaLib can be used to replace the current Database Locator, with slight
modifications.

Comments/Supporting Evidence Recommendations
* Given two tasks to locate named databases, Do not implement the Locate tab
only one participant failed on one occasion to Replace the Databases by Subject web
locate a database. page with a link to the specific Find
* The alphabetic list and search options in Find Databases->Category section of MetaLib
Databases were very easy to use, but some Change the "Category" tab to "Find by
participants had difficulty finding and using Subject"
the features within the "Category" and
"Locate" tabs.
* Many suggested that we consider changing
"Category" to "Find by Subject/Topic".


Find E-Journals Issues


Finding [#7]: MetaLib could be used to replace the current E-Journals Locator, with slight
modifications.

Comments/Supporting Evidence Recommendations


* Given two tasks to locate e-joumals, 77% of
the participants succeeded at each task. For
those who failed, the most common error
was to try searching for the journal title using
QuickSearch.
* Some participants had difficulty with e-
journals after they found them. The
participants could tell that a journal was
available electronically, for example, but
could not determine how to find the dates of
coverage.
* Participant did not understand the reason for
the list of all the databases or sources and
thought it simpler to go through the catalog
when looking for a journal
* Participant did not like looking in another
window for coverage dates


* Consider that dates of coverage should be
provided somehow in the current Find e-
Journals result list
* Remove the ISSN from the e-Joumals
results











Searching Issues

Finding [#8]: Categories in QuickSearch should be modified.


Comments /Supporting Evidence I Recommendations


* Users neither used Find
Database- Categories nor
MetaSearch-Categories successfully.
* Given two tasks to use Database Category,
over half of the participants failed (in one
case, about 85% could not find or use the
Category option). In most cases, the users
simply did not notice or think to use the
Category tab. They did, however, readily use
QuickSearch.
* Users consistently used QuickSearch for
most tasks to complete.
* Users stated their need for faster results with
least effort.


* Add the categories list from the Find
Databases-Category page to the bottom of
the QuickSearch page so people can choose
which categories to search (see image
below)
* Develop a key/notes box to explain that the
Categories provide recommended
databases for broad subject areas. Indicate
that more selected resources can be found
in the Find Databases-Category tab
* Only include cross-searchable databases in
the QuickSearch QuickSets.


Siar i Search

Search \


rur Maierch term


4 j .1 ra'l


4Jrsritr~l NOLW




JF ir-.1 Ihr lJ r ir illr



W, l- R- -~.


Ill tLIl f/'l1r 1, ir Ii- r







F I M aj, /01 i.. .
JL is


(. ilart'iv Lib ( 1>r H1'1
"CL. ..n
- flttdi~a t


JrI r


This Harvard Library
QuickSearch page enables
users to see broad subject
categories. Annotating where
possible would be helpful for
users.


Lq intmim10W,










Finding [#9]: More Advanced search options should be included in QuickSearch.

Comments /Supporting Evidence Recommendations
Users consistently used the QuickSearch to Add more options in advanced
complete tasks. Faculty and graduate students QuickSearch (e.g. title = journal title)
want to be more efficient and use the
Advanced QuickSearch which includes field
searching.

Finding [#10]: Remove MetaSearch from the initial implementation of MetaLib.


Comments /Supporting Evidence
* Most users did not understand the name or
purpose of MetaSearch, and 6 users
identified it as one of the most difficult
features to use (9 users identified My Space
as most difficult).
* Almost no participants used MetaSearch,
basically ignoring the feature and opting for
QuickSearch.
* Many found the interface confusing (i.e.,
with the drop-down menus on the left, the
resources listed below with some checked
and some not, etc.).


Recommendations
If MetaSearch is implemented:
Ensure instruction and tutorials are
available
To encourage use, make MetaSearch
the default search page or provide
more pointers to it from other
functional pages of MetaLib and within
the Library website
Ensure that MetaSearch obviously
recommends the appropriate databases
to use for the subject areas


Results Issues

Finding [#11]: Modify navigational features on the Result pages.

Comments /Supporting Evidence Recommendations
* Many users did not notice that MetaLib was Highlight searching or fetching in red or
searching or fetching. otherwise make them stand out
* Users did not immediately know how to get Make View Retrieved link larger
to the results. Show combined results total
* Users did not notice the Sort dropdown. Allow use of column headings to change
* Users did not want to use the "jump to" box. the sort of results
Provide a key/notes box to explain how
items are ranked and how to sort items
Place next and back hyperlinks on the
bottom of the page as well as the top of
the page
Use the Google-like page hyperlinks
such as "page 1 2 3 4 5 6 etc".
Make each database title link go to the
results from that resource.










Customization/My Space Issues


Finding [#12]: Remove My Space from the initial implementation of MetaLib.

Comments/Supporting Evidence I Recommendations


* All users liked the idea of My Space but
thought it was so poorly implemented that it
was useless to them.
* Users were consistently confused about how
to add resources to the saved space and could
not decipher the icons that were used to
move, save, and create the sets.
* Users could not determine that saved items
were in separate submenu, depending on the
resource type (e.g. articles, databases,
ioumals).


* Emphasize the submenu links that appear
under the main menu links in the My
Space section (such as "My Databases")
by moving them or enlarging the text
* Develop a key/notes box for QuickSet
creation and Set searching
* Allow users to search newly created
QuickSets from within the My Space
section
* Investigate ability to print, save, email
and export citations from My Space


Appendices


Assessment of Federated Searching using MetaLib: A Usability Test Protocol accepted by the
University of Florida Institutional Review Board




Full Text

PAGE 1

Usability Test Report for MetaLib Date of Report: April 7, 2006 Period of Testing: February 22-March 20, 2006 Period of Focus Groups: March 21-24, 2006 Prepared for: LeiLani Freund and Tom Minton, Co-Chairs Resource Navigation Group Martha Hruska Associate Director Prepared by: MetaLib Implementation Task Force Marilyn Ochoa, Chair Rae Jesano John Nemmers Carrie Newsom Maryellen OBrien Paul Victor

PAGE 2

2 Table of Contents Executive Summary ....................................................................................................................................4 Methodology ................................................................................................................................................5 What happened during the usab ility te st.............................................................................................................5 Who we tested.................................................................................................................. ...................................5 Where we tested................................................................................................................ ..................................7 Data we collected.............................................................................................................. ..................................7 Session I: University of Florida MetaLib Library Portal, Scenario-Based Testing .............................8 Session I, Initial Resource Impressions .....................................................................................................8 What are your initial impre ssions of this resource?............................................................................................8 What did you like a bout this re source?......................................................................................... ......................8 What did you dislike ab out this resource?...................................................................................... ....................8 What type of information would you expect to find on this site?................................................................ .......9 Session I, Scenario-Based Tasks ..............................................................................................................10 Find Data bases..................................................................................................................................................11 Find E-Journals................................................................................................................ .................................11 Searching an d Resu lts.......................................................................................................... .............................12 Customizati on/My Space......................................................................................................... .........................13 Overall Navigation and Ease of Use/In tuitiveness............................................................................................13 Session I, Post-Test Reactions .................................................................................................................15 Session II: University of Florida MetaLib Library Portal, Focus Groups ..........................................21 Findings & Recommendations .................................................................................................................24 Overall Navigation and Ease of Use/Intuitiveness Issues ......................................................................24 Finding [#1]: Me taLib is not intuitive..................................................................................... ......................24 Finding [#2]: MetaLib requires training and instruction and better help pages............................................25 Finding [#3]: MetaLib interface needs consiste ncy.......................................................................... ............25 Finding [#4]: MetaLib should be seamlessl y integrated with other library resources..................................25 Finding [#5]: MetaLib icons need consistency and explanation............................................................... ....26 Find Databases Issues .............................................................................................................................27 Finding [#6]: MetaLib can be us ed to replace the current Database Lo cator, with slight modifications......27 Find E-Journals Issues ............................................................................................................................27 Finding [#7]: MetaLib could be used to replace the current E-Journals Locator, with slight modifications. ..........................................................................................................................................................................27 Searching Issues ............................................................................................................................... .......28 Finding [#8]: Categories in QuickSearch should be modified................................................................. .....28 Finding [#9]: More Adva nced search options should be included in Qu ickSearch......................................29 Finding [#10]: Remove MetaSearch fr om the initial implemen tation of MetaLib.......................................29

PAGE 3

3 Results Issues ............................................................................................................................... ............29 Finding [#11]: Modify navigati onal features on the Result pages............................................................ ....29 Customization/My Space Issues ...............................................................................................................30 Finding [#12]: Remove My Space from the initial implementa tion of MetaLib..........................................30 Appendices ............................................................................................................................... ..................30 Assessment of Federated Searching us ing MetaLib: A Usability Test Protoc ol accepted by the University of Florida Institutional Review Board

PAGE 4

4 Executive Summary The University of Florida is in the midst of mi grating from NOTIS library management software to library management software developed by the ExLibris Group. As part of this migration, the Database Locator (which is supported by NOTIS) currently in use will be replaced with the MetaLib Library Portal (MetaLib). MetaLib is a research tool that enables simultaneous searching of multiple electronic resources from a single interface and provides links to library resources native interfaces. The MetaLib Im plementation Task Force (Task Force) was commissioned by the Resources Na vigation Group to evaluate th e effectiveness of MetaLibs overall interface design, with a special emphasis on database categorization, in lieu of MetaLibs debut in the fall of 2006. In order to accomplish its task, the Task Force developed and admi nistered usability testing to a cross-section of university users. The testing was intended to examine how well the design of the resource allows for ease of use by research ers, and how effectively MetaLib matches user expectations and needs. In developing the usability test, the Task Force outlined key areas of concern, which the testing wa s designed to answer: When researchers encounter MetaLib, do they readily know what the resource can do for them? How easy is it to understand and use the available search features to locate materials? Do users retrieve the anticipated results? Is navigation through retrieve d materials intuitive? The usability testing resulted in the following overall findings: All users had some level of difficulty utilizi ng MetaLib, resulting in very low overall task performance; Users mistook the General Search default Qu ickSet in QuickSearch for a Google-type search everything option. Ofte n users searched for e-journa ls, databases, articles and books using the QuickSearch; Users had difficulty locating subject specific databases, such as the Project Starter databases, under the Find Database tab; Users wanted to search the online catalog from within MetaLib from any screen; Users were often confused about what constituted an e-journal versus a database; Users were confounded by many of the icons utilized; Users were confused by the navigational and disp lay features in the Results pages; and Users found the MetaSearch and My Space f unctionalities particularly perplexing and difficult to use. As a result of these findings, the Task For ce makes the following general recommendations: Placement of subject-specific QuickSets on the MetaLib homepage with descriptions, i.e. use of the Description field, which displays text below the title. Seamless integration of MetaLib with other library resources. Consistent use of navigational functions/links.

PAGE 5

5 Modification of icons. Modification of the Results pages. Fall 2006 implementation of the QuickSearch, Find E-Journal and Find Database functions. Modification of the My Space and MetaSearch functions. Delayed launch of the MetaSearch and My Space tabs based on the significant difficulty had by users in using these functions Creation of instructional workshops, guides and online tutorials. The following report will provide a detailed summary of the Task Forces testing procedures and findings. Methodology What happened during the usability test The MetaLib Implementation Task Force conduc ted a usability evalua tion of the MetaLib Library Portal at the University of Florida in Gainesville, FL. The research methodology involved scenario-based testing to eval uate online user search behavior. Participants of the usability testing completed preand post-test questionnaires which the task force used to evaluate participants experience us ing online library resources and their satisfaction with using MetaLib. Participants then completed a test session consisting of structured exercises using the MetaLib search interfaces and result pages 1 During the test session we asked participants to talk out loud about their process towards fi nding an answer as well as their expectations. Test sessions we re recorded using Camtasia Studio 3.0, a software program that captures cursor movements, navigation and audio. After the test sessions participants were compensated with a $10 gift card to Borders and invited to particip ate in a focus group. Each focus group consisted of 1-3 participants, a facilitator, and at least one member of the MetaLib Task Force. The facilita tor asked a series of questions 2 while the Task Force member took notes on the session. Participants were provided a pizza lunch during the focus groups. The scenario-based testing took place from February 22-March 14, 2006 and focus groups were held during the week of March 20, 2006. Who we tested Participants of the MetaLib usability testing were recruited prior to testing by email to listservs and through personal contact. Participants were se lected from three cate gories of library users 1 Details of the tasks completed are in the Appendix, page 8. 2 Details of the focus group questions are in the Appendix, page 12.

PAGE 6

6 who provided a representation of each major subject area (natural sciences, social sciences, arts and humanities, law, and health). Fifteen participants, having the following profile characteristics, evaluated MetaLib. Resource use, computer skill level and searching skill level are all self-reported values gathered from the pre-test questionnaires. Academic Status Faculty 4 Graduate Student 6 Undergraduate Student 5 TOTAL 15 Area of Academic Interest Natural Sciences 3 Social Sciences 4 Arts and Humanities 3 Law 3 Health Professions 2 TOTAL 15

PAGE 7

7 Where we tested Following is a summary of the participants computing environment: URL of tested website: [http://metalib.fcla.edu] Computer platforms: [Dell Pentium IV with an 17 display] Browser tested: [Mozilla Firefox] Screen resolution: [1024 X 768] Operating system: [Windows XP] Connection speed: [Shared T1] In most cases, testing and observation were completed in the Marston Science Library electronic classroom. Law and Health affiliated participants were tested in on-site conference rooms using a Library-owned laptop with the following: URL of tested website: [http://metalib.fcla.edu] Computer platforms: Pentium M 713 1.1 GHz ULV Browser tested: [Mozilla Firefox] Screen resolution: [1024 X 768] Operating system: [Windows XP Professional] Connection speed: EtherNet, WiFi (wireless), and 56Kbps modem Data we collected We sought to collect data that would address the major components of the resource gateway, including the Overall Navigation and Ease of Use/Intuitiveness, Find Databases, Find E-Journals, Searching, Result pages, and Customization/My Space features.

PAGE 8

8 Session I: University of Florida MetaLib Library Portal, Scenario-Based Testing Session I, Initial Resource Impressions At the beginning of each individual scenario-based test session, we allowed the participants to preview the resource. We asked part icipants the following four questions: What are your initial impressions of this resource? What did you like about this resource? What did you dislike about this resource? What type of information would you expect to find on this site? What are your initial impressions of this resource? Thought the resource was organized and cleaner/aes thetically more pleasing/simpler to use than current system Liked the possibility of using the history information and customization features Believed that QuickSearch would make searching easier Was impressed by the possibility of some de-duplication of results Believed that MetaLib needs a homepage with a description of what the resource will provide Some points were flawless but others made me very sad: navigation through site, QuickSearch and finding full text Need to increase the wideness of the site/database areas, need consis tency in the color of the links (use of gray bars that make it difficult to see the links: seem like unavailable links) Use of Boolean operators but addition of without Need more information about what MetaSear ch is (only a few effectively used this) What did you like about this resource? Use of tabs to separate different searching options: database and journals Alphabetical list for e-Journals and databases to navigate Simplifies finding e-Journals than Serials Solutions Other resources such as Google Scholar are available to search as well One-stop shopping combining various resources to search together Potentially very useful because a larger body of resources are available to search simultaneously What did you dislike about this resource? Does not understand why there is a 15 result limit per database and how they are selected Would prefer to see the next and previous buttons on both the top and bottom of the results pages Found that in some cases the next link will take u sers to next record and not next page of the result lists Inconsistent terminology and acti on icons throughout the resource (e.g. My Space, basket and eshelf) Icons need a key or use of text other than the title hover and need to be consistent Lack of an explanation of how ranking is determined Does not understand what each component of the resource is prior to using it Seemed that the QuickSearch Advanced features had a limited number of fields

PAGE 9

9 Did not like to be linked to another resource/search engine Did not know when the search was running because the searching indicator is small What type of information would you expect to find on this site? Common answers included: Articles but not books Journals, books, and articles Want to find books in collection with call number available Quick access to the SUS catalogs

PAGE 10

10 Session I, Scenario-Based Tasks During the usability evaluation, participants were then asked to complete a number of scenarios or real-life tasks using the resource. The tasks, which were presented to participants in order on index cards, addressed one or more issues. # Scenario-Based Tasks Issue Addressed 1 How many databases are categorized as Project Starter databases? Find Databases 2 Find the database Academic Search Premier. How many articles are available for mouse physiology? Find the full-text for the first article in the result set. Find Databases/ Searching/ Results 3 What is the time-span of coverage for the database Plant Science? Find Databases 4 What is the full name of the database abbreviated EEBO? Find Databases 5 Does the Library have a print subscription to the journal Biology of the Cell? Is the journal available electronically? Find E-Journals/ Searching 6 Do a QuickSearch of the term cargo cult. How many records are retrieved? How many records are displayed on your screen? Did you retrieve any results from the Library Catalog? How many? How are the records sorted? Searching/ Results 6 Find the most recently published material from your result list of the search cargo cult. What database or resource did it come from? Add the article to your My Space. Results/ My Space 7 Perform a search on Medicare and prescriptions in Health Business Full Text through the MetaLib interface. How many results did you yield? Go to the normal/native database interface for Health Business Full Text. Repeat the search and indicate how many results you retrieved? Find Databases/ Searching Results 8 Add the databases PsycINFO and Dissertations and Theses (ProQuest) to your Sets. Create the Set called Psychology and add those two databases to it. Using the QuickSet Psychology, search for searching behavior. How many results are found and how many are retrieved. From which database does the top ranked article come? Find Databases/ My Space/ Searching/ Results 9 Log out of MetaLib and then log back in to MetaLib. Review your e-Shelf citations. Are your QuickSets still available? My Space 10 You need to write a paper on a criminal law topic and your professor suggests the 2003 article entitled Shame, Stigma and Crime: Evaluating the Efficacy of Shaming Sanctions in Criminal Law. Its citation is: 116 Harvard Law Review 2186. Locate the Harvard Law Review. Find E-Journals 10 Now locate the article cited above. Can you view the full text of the article? If you wanted to read the article later, would you be able to email it to yourself? Add the article to your my Space. Results/ My Space 11 What are the four recommended databases to search if you are looking for articles about Biochemistry? Find Databases 12 How can you simultaneously search these four databases for pine beetle? Searching

PAGE 11

11 Find Databases Most participants easily used MetaLib as a da tabase locator, specifi cally when looking for known resources. They had little/no trouble using the alphab etical list and sear ch options in Find Databases. However, many of the participan ts had difficulty locating databases by category. When asked how many databases are categorized as Project Starter databases, 54% of the participants could not accomplish this task, prim arily because they did not see or understand the "Category" tab. Even some of those participants who did correctly answer this question stated that they didn't quite understand the "Project Starter" terminology and probably would never have found it or used it on their own. When asked to locate databases under the Bioche mistry category, 85% were not able to complete the task. Most tried the alphabetic list or title search in Find Databases Locate to determine if any database titles contained the term "biochemistr y." When this didn't work, most gave up or tried to conduct a QuickSearch to see if anythi ng came up for that term. Only one participant completed this task easily by using the Categor y tab, and another pers on eventually found the Category tab solution but only after much searching. Even for these successful participants, they first tried to find the subcategory in Medicine/H ealth or Physical Sciences, and they did not consider first searching in Agricultural and Biol ogical Sciences. Some participants suggested that we consider changing Cate gory to Find by Subject/Topic. When searching for known databases such as EEBO or ASP, the participan ts were particularly successful using the alphabetic li st or the Find Database search options (i.e., "starts with" or "contains"). 100% of the particip ants were able to locate Acad emic Search Premiere, and only one participant failed to locate EEBO. Once the participants successfully located a data base, the preference was simply to click to the native interface rather than to remain in MetaLib to conduct searches or to learn more about the database. When asked to locate and query Academ ic Search Premiere, 76% of the participants used the native interface and onl y 24% used the MetaLib interface. When asked to identify the dates of coverage for the Plant Science database, 69% went to the native interface and only 31% used the MetaLib information icon to answer the question. Find E-Journals Most of the participants were able to use the E-Journals feature as easily as they used Find Databases to locate electronic resources. When asked to locate the Harvard Law Review, for example, 77% easily located the journal (or found it after a small amount of difficulty), and only 23% could not find it. When asked if we have both print an d electronic versions of the journal Biology of the Cell most of the participants (77%) were able to locat e the electronic version using the E-Journals alphabetic list or search feature. They could tell that the journal was available electronically, but could not determine how to find the dates of coverage. In addition, many of them had difficulty figuring out how to search the OPAC to locate print holdings and stated th at they would probably just leave MetaLib and open the Catalog to di scover print holdings. Please note that the UF Libraries Catalog failed in 3 of 15 sessions.

PAGE 12

12 The participants also demonstrated uncertainty about using the SFX link; most participants stated that they were not sure what it does. Even those participants w ho used SFX weren't confident in using it (e.g., some users could not find the print version of Biology of the Cell despite seeing the UF Library Catalog link in the SFX window for that resource). Some participants used QuickSearch to locate a journal title. The default General Search was usually searched, but it is not certain how many users thought about what they were using. Very few used the Advanced title in QuickSearch, but those who did were uncertain if title could be used for journal title (as opposed to ISSN). Searching and Results 100% of the participants were able to easily use QuickSearch to conduct general searches, but they had more trouble understand ing and using search results. On ly a handful of participants used MetaSearch with any success but it was obvi ous that they preferred to use the simpler QuickSearch feature. After completing a QuickSearch or MetaSearch, some users thought that the initial results screen (i.e., the Summary page that lists result totals by each resource searched) was confusing. Some questioned the discrepancy between found and retrie ved results and the lack of a total count of resources retrieved. A couple of participants want ed to have a "View Results" link beside each resource so that she could just look at those results, as opposed to one combined "View Results" link for all of the resources. One participant pointe d out that this was true for MetaSearch but not for QuickSearch. Some users thought that the "Vie w Retrieved" link was not easily noticeable at the top of the page. Most users could not immediately tell how the retrieved records were sorted. Although most were able to identify the initia l sort as by rank, many expressed conf usion as to what this meant. They did not think the green bar was useful to help determine how they were sorted, especially without a numerical percentage provided. Also, th ey were confused when duplicate results were not sorted together in the resu lt list (in one case, the duplicat e item was three items below the first mention of the item; it appeared that the ar ticle was sorted that way due to its database). 62% were unable to sort by date, primarily because they did not even see the "Sort" box on the page. Further, when sorted by year, it was uncerta in if results were alphabetically sorted by title after the initial date sort. Some participants indi cated that they were surprised that sorting could not be done using the column title. When asked to conduct a specific search in the Health Business Full Text database using the MetaLib interface rather than the native interface, 46% could not fi gure out how to search within a specific database using MetaLib. Only 31% eas ily completed this task, while the other 23% insisted on using the native interface rather th an MetaLib. Participants who successfully used MetaLib to search were asked to repeat their s earch of the Health Busi ness Full Text database using the native database interface, but not one of the participants noticed any discrepancy between the results retrieved ev en though there were significant discrepancies produced by many of the searches.

PAGE 13

13 Several participants expressed a desire to have more QuickSearch options. Many proposed that the statewide union database should be one Quic kSearch option, and the UF Catalog should be its own option as well. Customization/My Space By far, the My Space feature caused the most difficulty for participants. Even when successfully saving resources to My Space or successfully lo cating and using those saved resources, a large majority of the participants expressed confus ion and frustration. They understood how My Space could be useful (and many said that they would love to use the feature) but they thought it was very difficult to use. When asked to add an article to My Space, 54% of the participants had difficulty doing this (e.g., they couldn't find the icon immediately, they weren't sure it had worked once they clicked it, and many didn't even understand what they had just d one). 15% of the participants were unable to add an article to My Space. When asked to add two databases to My Space and save them as a QuickSet, the participants met with quite a bit of difficulty and confusion. Only one participant easily completed the task. 46% completed the task but had trouble doing so and many weren't quite sure that they had succeeded (i.e., once they had clicked on the icon they wanted some sort of feedback that the databases were successfully saved). Another 46% had considerable difficulty and were unable to complete the task. The most common problem was that pa rticipants simply could not locate the "My Databases" tab under My Space. Many participants even stated that they had completely missed the menu containing My Sets, My Databases, etc., a nd requested that it be larger or more visible in some way. The problems continued when the participants were asked to use the newly created QuickSet to conduct a search. 38% were able to search the Quic kSet, but the majority of the participants had to skip this task or was unable to complete it Because there are no instructions on the page, most users thought they could stay in th e My Space tab to search their da tabases. Frustratingly, at least 4 people indicated that they were so lost right now. The participants expressed fr ustration that the terminology was inconsistent between "My Space", adding to the "basket", e-shelf, the shoppi ng cart icon, "My Sets", etc. Also, the minor change that occurs to the icons for adding to My Space was not effective. All of the participants were able to log out and log back in to review their My Space resources. The history information seemed not to be working for some of the participants. Overall Navigation and Ease of Use/Intuitiveness The participants had numerous comments and concerns about navigation and usability of MetaLib, in addition to those i ssues described in the sections above pertaining to specific features of the application. Many participants requested relatively simple ch anges that would make the application much easier to use. For example, they requested that all Next/Previous page link s appear at the top and

PAGE 14

14 bottom of each page. Also, they requested that column headings in the results pages should be made links that re-sort the results according to the column heading clicked. Participants wanted more consistency in terminology, particularly related to My Space (e.g., e-shelf, basket, clipboard, etc.). Many also sugge sted that icons should be ex plained/identified more easily, either by using some kind of key or by using text instead. Many participants had difficulty finding menu tabs such as the Category tab under Find Database or the My Databases tab under My Space. Many of them simply didnt see these menu options, and they need to be modified to make them more recognizable. Most participants preferred to conduct searches us ing the database native interface even for those questions specifically asking them to s earch using MetaLib. Most ignored the Search/Information icons until a specific task forced them to figure out what they were (e.g., for the date span coverage question, almost all of the participants went to the native interface to answer it and simply i gnored the I icon). Many patrons encountered a problem when they opened a database in a new browser window and then returned to MetaLib without closi ng that window. When they subsequently found another database and tried to open it, it appear ed that nothing was happe ning. Actually, the new database was opening in the already-opened wi ndow but they couldnt tell because the focus didnt change to that window. So me participants asked if we could force the focus to be on the newly opened resource even if the browser window is already open behind the MetaLib window. More advanced features like My Space and MetaSearch posed the most difficulty for participants. They did not find these features to be intuitive at all, and e xpressed great frustration because they really were interested in using the features but would require instruction before they could use them with any confidence.

PAGE 15

15 Session I, Post-Test Reactions At the end of each session, we asked participants to complete a post-test questionnaire. The questions were useful in gauging how easy the resource was to learn and to use. Overall navigation, ease of use, intuitiveness, satisfaction, and results were considered. We asked the participants to rank their responses to the following questions: Overall, how easy was it to use MetaLib? Easy to Use MetaLib024681054321 V ery Easy Somewhat Easy Difficult Participants How easy was it to use the My Space feature? Easy to Use My Space012345654321Very Easy Somewhat Easy DifficultParticipants

PAGE 16

16 How would you rate the option to create your own QuickSets? Ability to Create QuickSets01234567854321Useful UnnecessaryParticipants How easy was it to create your own QuickSets? Easy to Create QuickSets01234554321 V ery Easy Somewhat Easy Difficult Participants

PAGE 17

17 How easy was it to determine how many records were retrieved by searches you conducted? Easy to Determine Records Retrieved0123456789101154321 V ery Easy Somewhat Easy Difficult Participants How easy was it to determine how results were ranked? Easy to Determine Results Ranking0123456754321 V ery Easy Somewhat Easy Difficult Participants

PAGE 18

18 How easy was it to change the sort order of the results? Easy to Change Sort Order012345654321 V ery Easy Somewhat Easy Difficult Participants How likely are you to use this resource for online research? Likely to Use MetaLib for Research012345654321 V ery Likely Somewhat Likely Unlikely Participants

PAGE 19

19 How likely are you to recommend this resource to someone for online research? Likely to Recommend MetaLib to Others0123456754321 V ery Likely Somewhat Likely Unlikely Participants How would you rate your overall experience with MetaLib? Overall Experience with MetaLib012345678954321Excellent/Satisfying Good Poor/Frustrating Participants

PAGE 20

20 In addition, we had participants answer the following, after providing a list of options from which to select. Which areas did you find most difficult to navigate? Most Difficult Areas21069510246810QuickSearchFind DatabasesFind E-JournalsMetaSearchMy SpaceIndividual Results/Finding Full TextResults Pages Please indicate which areas could be improved (provide more information if necessary) 8 participants identified "Terminology" as the area needing most improvement and 6 participants selected "Navigation Throughout Site". 4 participants picked each of the following: Site Design, Results Pages, and Navigation within Results Pages. Does it concern you that you may not be retrieving all records matching your search? 8 respondents said that it did concern them, and 4 respondents said that it did not concern them.

PAGE 21

21 Session II: University of Florida MetaLib Library Portal, Focus Groups After completing the scenario-based testing, participants were encouraged to continue to use MetaLib and then participate in a focus group. The following questions were asked to provide more information about various parts of the resource that were not covered by the post-test questionnaire that was taken. Focus Group Questions After continued use of MetaLib, what features of the resource would you definitely use? Why? After continued use of MetaLib, what features of the resource would you definitely NOT use? Why? Why would you want to use this resourceto perform a QuickSearch on a single subject or if you wanted to search several databases together? Would you use this resource? How important is it for you to find scholarly or academic articles? Does MetaLib provide you the types of resources you want to find? Have you ever used Google Scholar? If you had a choice, would you use MetaLib or Google Scholar? The MetaLib interface can be positioned on various parts of the website. Choose your preference of locationon the homepage or as a sub-page of the website. Explain why. How and from where did you connect to MetaLib after the test sessions and prior to this focus group? Were results pulled up in a reasonable timeframe (seconds versus minutes?)? Did you have any trouble retrieving results due to connection speed? Was it difficult to figure out how to search from within a database or an e-journal? What could make this easier? Would you use the QuickSets that we have in place now? How easy was it to create your own personalized QuickSets? Would you recommend other resources to add to the QuickSets, or broad areas to create a QuickSet of databases? When you did a search of multiple databases, were you concerned that you might not have retrieved all the records you normally would have if you searched those databases separately? Would this be a concern to you that you would retry the searches in each individual database? When performing searches in the library catalog or an online search engine, do you care how the results are sorted when displayed (e.g., by relevancy, by title, by author)? Explain why. When performing searches in the library catalog or an online search engine, do you care how the results are ranked according to relevancy? Explain why. Do you have experience using SFX? Does it matter to you that you cannot print or E-mail your saved citations/records within the e-shelf section? When saving search results in My Space, was it intuitive or easy to use? Did you have any strange experiences with your saved citations/records after logging off or returning to your MetaLib account later on? During the focus groups, most participants agreed MetaLib has many uses, but thought it was an extremely non-intuitive library resource. Participants suggested that while they may use MetaLib at some point in their research, they would not claim to fully understand what they were doing. More experienced researchers suggested they might use it to get started on new or quick research but then would move on to the native databases for more in-depth research.

PAGE 22

22 MetaLib requires instruction, tu torials and better help pages. The icons used throughout the resource require a key to explain their functions. Students suggested a welcome page or tutorial to explain how to find articles, databases, and other resources through MetaLib. A faculty member recommended adding instru ctional tips on the search inte rface itself, such as showing users how to type in an author search (e.g. last name first or fi rst name first). Further, some participants noted that certain help sections used the term it was de fining to define the term itself; Category is one example of this circular and non-useful help. Rewriting the help screens to avoid this was recommended. Adding help screens to explain the use of MetaLibs QuickSearc h and QuickSet features would improve usability. Users did not fully understand the purpose of the Libraryselected QuickSearch QuickSets. One user believed that using the General Search QuickSet would allow him to search all the resources to which the Libraries subscribed ; some thought these predefined QuickSets (i.e. General Search, H ealth Resources, Quick Facts) pr ovided all relevant databases in those subject areas. Other users questioned why there were so few QuickSearch QuickSets; many asked for more QuickSets to be developed. Some thought th e option to select any of the major subject areas (such as those listed on the Librarys current Databases by Subject webpage) should be available in the QuickSearch tab. Further, QuickSets for the UF Library Catalog and the Union Catalog for the State Universities System and for specific geographic regions/area studies (or at least the ability to limit to these regions) were recommended. One user noted that the Link to message received when attempting to search a resource not metasearchable is useless. Apart from using these Libraryselected QuickSets, participants commented that being able to create their own QuickSets is an attractive feature of MetaLib. However, they thought the My Space feature to create them is too difficult to us e. Many participants could add items to My Space but did not know what to do after that. They wanted explicit instructions on how to set up QuickSets in the form of on-screen hints/note boxes in the empty space on the right side of the page or via an online tutorial. One student s uggested that a link sayi ng, Now click here to search this set should appear after the pers onalized QuickSet is created. Further, most participants did not see the secondary menu bar in My Space (e.g. the bar contains e-shelf, My Databases, etc.); these submenus should be more noticeable in order to be useful. Users also expressed the need to email, print, export, or save full citations. When given the choice, participants said they would use QuickSearch before MetaSearch. Most participants found MetaSearch conf using or inappropriate for their research needs. Most of the experienced researchers said they would not use it because they prefer to use the native database interface that gives them more c ontrol over the fields they can s earch or because some of the database resources they want to use are not available for meta searching. Other participants suggested they might consider using MetaSearch if the limite d number of meta-searchable databases is increased. In discussing user access to MetaLib, some users said they would use a drop-down box feature on the Librarys homepage to access the resource. Th is fulfills at least four participants desire to have everything on one page and one undergradua tes concern that the tabs should not be

PAGE 23

23 necessary since they are so confusing. Other us ers said they would pr efer going into the full MetaLib interface first to run a search. Further, some faculty members expressed their hope that the Libraries could investigate th e option of having hyperlinks to resources like MetaLib or the Library Catalog from other librar y resources (e.g. databases and the library website). In addition, a hyperlink from the Remote Logon confirmation page is also necessary. Participants expressed concern a bout the discrepancy between hits returned in MetaLib and hits returned in native interfaces. However, most admitted they would not notice the discrepancy on their own (they would trust MetaLi b to retrieve everything). They said generally this would not be too much trouble for them, since they would ju st use MetaLib to get st arted in research and then go to specific databases and use native interfaces to more thoroughly conduct thorough research. Users did not like the jump to box on the result pages and recommended links at the bottom of the page which provide page numbers they can select (e.g. "page 1 2 3 4 5 6 etc"). In addition, most participants wanted Next and Previous links to be on the top and bottom of each result page. Most users wanted the ab ility to sort by clicking on the co lumn headings in the results view. They liked the rank default but they wanted to know what the rank sort meant. Few users knew what SFX means but some used it in the scenario-based te sting or in their own research. These participants generally understand what it could do for them, but have used it without understanding the terminology "SFX"; part icipants suggested wording including "Find it @ UF" and "Find Full-text". In some cases, SF X screens were noted to convey little vital information, such as available dates of coverage.

PAGE 24

24 Findings & Recommendations These findings and recommendations provide information from the Scenario-based testing as well as the focus group meetings conducted. Overall the MetaLib interface is not intuitive, not user friendly and requires significant modification. Following are specific issues and recommendations for implementing MetaLib. While we can make certain display and interface changes they do not fully address the problems users encountered. The findings and recommendations from this usability study should be submitted to ExLibris. Overall Navigation and Ease of Use/Intuitiveness Issues Finding [#1]: MetaLib is not intuitive. Comments/Supporting Evidence Recommendations Three users indicated that they would probably not use the resource because they could not figure it out. Most users did not understand the purpose of MetaSearch, and 6 users identified it as one of the most difficult features to use (9 users identified My Space as most difficult). Several participants recommended everything should be on one page. Users suggested that remembering another password would be cumbersome. Use the Gatorlink logon information Provide better help and instructional information (see Finding #2)

PAGE 25

25 Finding [#2]: MetaLib requires training and instruction and better help pages. Finding [#3]: MetaLib interface needs consistency. Finding [#4]: MetaLib should be seamlessly integrated with other library resources. Comments/Supporting Evidence Recommendations Users would more likely rate the experience with MetaLib higher if they had training and introduction to it. Although most users will be able to use the Find Databases/E-Journals features without assistance, most participants clearly would have benefited from instruction/training before using My Space and MetaSearch features. Many used the link to the help, but could not find the exact information sought: different forms of the term that was being looked up was used often used the term itself to define what it was Create an instructional homepage or tutorials and a plan for standard instructional materials and scripts Modify Help pages Comments/Supporting Evidence Recommendations 8 participants identified "Terminology" as the area needing most improvement and 6 participants selected "Navigation Throughout Site." Many specifically complained about inconsistencies that should be easy to fix. Users could not locate navigational links where they expected them to be. The hyperlinks on the menu bars were difficult to see due to use of the same color for text and links. Use consistent terminology Put sub-menus, drop-down menus, and Next/Previous links in the same places on every page Place all Next-Previous links at the top AND bottom of pages for database/e-journal alphabetic lists and search results Use a different color for links in the menu bar than for other links throughout MetaLib Comments/Supporting Evidence Recommendations Many participants in the focus group stated that they would use a MetaLib search box on the librarys home page. Participants wanted the ability to search the catalog and the SUS union catalog within MetaLib. Participants wanted the ability to switch between multiple library resources easily (i.e. databases, the library catalog, and MetaLib) Place MetaLib on the main library page, implemented in a Google-type presentation Add a link to the librarys homepage, preferably in the UF Library banner Investigate providing links to the library homepage and catalog from all library resources including databases

PAGE 26

26 Finding [#5]: MetaLib icons need consistency and explanation. Comments/Supporting Evidence Recommendations Users were confused by the slight icon change once the icon was clicked on (i.e., they werent sure that the resources had been added). Some users saw that the icon for adding articles changed slightly, and the title hover read that it had been added, but usually users did not notice the transfer. Icons are not intuitive. Users did not know immediately that the basket was the icon used to select an item to save it. Some people used the SFX icon but mentioned that they really didnt understand what it stood for. Many users ignored icons completely at first and many never used some of the icons. Ensure that all My Space icons (e.g. add to basket, add to e-shelf) all use the same icon and language Ensure icons like the basket and the + add icon, change more noticeable than it currently does (e.g. change color of the icon) Allow that a second click on an icon will deselect an item that goes to My Space Add a key/notes box to the empty space on the search page, similar to Harvard Libraries QuickSearch page (see image below). Also consider the use of text in lieu of some icons. Change the SFX icon to Find it @ UF or Find Full Text icons and text

PAGE 27

27 Find Databases Issues Finding [#6]: MetaLib can be used to replace the current Database Locator, with slight modifications. Find E-Journals Issues Finding [#7]: MetaLib could be used to replace the current E-Journals Locator, with slight modifications. Comments/Supporting Evidence Recommendations Given two tasks to locate named databases, only one participant failed on one occasion to locate a database. The alphabetic list and search options in Find Databases were very easy to use, but some participants had difficulty finding and using the features within the Category and Locate tabs. Many suggested that we consider changing Category to Find by Subject/Topic. Do not implement the Locate tab Replace the Databases by Subject web page with a link to the specific Find DatabasesCategory section of MetaLib Change the Category tab to Find by Subject Comments/Supporting Evidence Recommendations Given two tasks to locate e-journals, 77% of the participants succeeded at each task. For those who failed, the most common error was to try searching for the journal title using QuickSearch. Some participants had difficulty with e-journals after they found them. The participants could tell that a journal was available electronically, for example, but could not determine how to find the dates of coverage. Participant did not understand the reason for the list of all the databases or sources and thought it simpler to go through the catalog when looking for a journal Participant did not like looking in another window for coverage dates Consider that dates of coverage should be provided somehow in the current Find e-Journals result list Remove the ISSN from the e-Journals results

PAGE 28

28 Searching Issues Finding [#8]: Categories in QuickSearch should be modified. Comments /Supporting Evidence Recommendations Users neither used Find DatabaseCategories nor MetaSearchCategories successfully. Given two tasks to use DatabaseCategory, over half of the participants failed (in one case, about 85% could not find or use the Category option). In most cases, the users simply did not notice or think to use the Category tab. They did, however, readily use QuickSearch. Users consistently used QuickSearch for most tasks to complete. Users stated their need for faster results with least effort. Add the categories list from the Find DatabasesCategory page to the bottom of the QuickSearch page so people can choose which categories to search (see image below) Develop a key/notes box to explain that the Categories provide recommended databases for broad subject areas. Indicate that more selected resources can be found in the Find DatabasesCategory tab Only include cross-searchable databases in the QuickSearch QuickSets.

PAGE 29

29 Finding [#9]: More Advanced search options should be included in QuickSearch. Finding [#10]: Remove MetaSearch from the initial implementation of MetaLib. Results Issues Finding [#11]: Modify navigational features on the Result pages. Comments /Supporting Evidence Recommendations Users consistently used the QuickSearch to complete tasks. Faculty and graduate students want to be more efficient and use the Advanced QuickSearch which includes field searching. Add more options in advanced QuickSearch (e.g. title = journal title) Comments /Supporting Evidence Recommendations Most users did not understand the name or purpose of MetaSearch, and 6 users identified it as one of the most difficult features to use (9 users identified My Space as most difficult). Almost no participants used MetaSearch, basically ignoring the feature and opting for QuickSearch. Many found the interface confusing (i.e., with the drop-down menus on the left, the resources listed below with some checked and some not, etc.). If MetaSearch is implemented: Ensure instruction and tutorials are available To encourage use, make MetaSearch the default search page or provide more pointers to it from other functional pages of MetaLib and within the Library website Ensure that MetaSearch obviously recommends the appropriate databases to use for the subject areas Comments /Supporting Evidence Recommendations Many users did not notice that MetaLib was searching or fetching. Users did not immediately know how to get to the results. Users did not notice the Sort dropdown. Users did not want to use the jump to box. Highlight searching or fetching in red or otherwise make them stand out Make View Retrieved link larger Show combined results total Allow use of column headings to change the sort of results Provide a key/notes box to explain how items are ranked and how to sort items Place next and back hyperlinks on the bottom of the page as well as the top of the page Use the Google-like page hyperlinks such as "page 1 2 3 4 5 6 etc". Make each database title link go to the results from that resource.

PAGE 30

30 Customization/My Space Issues Finding [#12]: Remove My Space from the initial implementation of MetaLib. Comments/Supporting Evidence Recommendations All users liked the idea of My Space but thought it was so poorly implemented that it was useless to them. Users were consistently confused about how to add resources to the saved space and could not decipher the icons that were used to move, save, and create the sets. Users could not determine that saved items were in separate submenu, depending on the resource type (e.g. articles, databases, journals). Emphasize the submenu links that appear under the main menu links in the My Space section (such as My Databases) by moving them or enlarging the text Develop a key/notes box for QuickSet creation and Set searching Allow users to search newly created QuickSets from within the My Space section Investigate ability to print, save, email and export citations from My Space Appendices Assessment of Federated Searching using MetaLib: A Usability Test Protocol accepted by the University of Florida Institutional Review Board