Membership: Hill (leader), Carver, Dolin, Frew, Larsgaard, Nideffer, Rae, Simpson
Mission Statement of Team: The goal of the UCSB component of the Interface Evaluation Team is to evaluate the effectiveness of the ADL system from the perspective of potential users of the system. Knowledge gained from these evaluation activities is used (1) to inform the design and implementation of the ADL system on the Web, and (2) to document in detail the effectiveness of the ADL and areas calling for improvement, both in the interface design and in the underlying system functionality and content.
Research activities over the last year:
Hill, Linda L., Dolin, Ron, Frew, James, Kemp, Randall B., Larsgaard, Mary, Montello, Daniel R., Rae, Mary-Anna, and Jason Simpson. (1997). User evaluation: Summary of the methodologies and results for the Alexandria Digital Library, University of California at Santa Barbara. Proceedings of the 60th Annual Meeting of the American Society for Information Science, November 1-6, 1997, Washington, D.C. http://www.asis.org/annual-97/alexia.htm
Thirty-two representatives from the public sector, private sector, and academia came from the U.S. and one from Canada for two-and-a-half days to participate in plenary and small group discussions, software demonstrations, and informal discourse.
In preparation for the meeting, the Steering Committee agreed upon a set of objectives to guide the content of the agenda. A three-part focus was established to address issues of search, browse, and retrieval; content and processing; and interface and navigation. These three areas encompass a number of the more difficult problems to be resolved immediately. Their selection was based primarily upon feedback gained in focus groups, survey questionnaires, and videotaped or recorded interactions with actual and potential digital library patrons over the first two years of the Alexandria Project. Many specific topics were discussed including ways for users to view the contents of the library (an overview of the holdings), giving the user a ``road map'' of the services of the library, visualization (presentation) of search results, support for iterative search processes, and customization of the user interface.
From the wide range of comments and recommendations contributed by the Panelists that are detailed in the report, some strong themes can be identified. First, the Panel is supportive of the efforts of the research teams working at Santa Barbara and at Colorado. Panelists and funding agency representatives agreed that keeping a careful balance between developing an operational digital library and contributing to theoretical and applied research is necessary. The importance of the content of the library was strongly supported, as were the enabling technologies for distributed ingesting of data with attention to the representation of data provenance and verification. Collection development criteria were discussed at length, with emphasis on focusing on particular geographic areas and collecting all forms of georeferenced information about that place. Panelists recommended that ADL develop enabling technologies and specifications for connecting local collections to ADL, with distributed searching capability.
In connection with ADL user interface design, the implementation of user profiles was discussed with ideas for multiple profiles per person and for the way in which the profiles could influence the interface presentation and conduct of the search. Support for a method of providing ``something with a heartbeat'' to be available to assist in the use of ADL was expressed in several ways, with the related recommendation that ADL aim to create a professional tool for information specialists as well as for other user communities. A discussion of measures of success for ADL focused on grounding those measures in the user's work environment and information needs.
ADL Staff identified four areas in which the Design Review has had an immediate impact: (1) collection development strategy, (2) user profile strategy, (3) development of a strategy to provide human support to digital library users, and (4) the design of the new interface.
The overall thread that runs through the recommendations is that ADL needs to focus better on what can be accomplished in the short time left in the current phase of the project. Most panelists reported that the format of the Workshop worked well, but they would like for it to have been more focused and they expect to see specific responses from ADL on the recommendations made.
The Evaluation Teams continued to develop a list of the ways in which users would want to use a system such as the Alexandria Digital Library. This list of 58 scenarios, derived from ADL Target Group Members, ADL documents, Global Change Data and Information System user queries, GIPSY report (Woodruff), Petroleum Abstracts indexing staff, and analysis of the ADL Rapid Prototype questionnaire replies by the University of Colorado Team is available in the ADL User Scenarios at http://www.alexandria.ucsb.edu/ lhill/scenarios.html.
The UCSB Evaluation Team designed the components for the User Registration form and the Exit Poll form for the new interface. These designs are based on prior experience with the beta web interface and on the evaluation research questions identified by the Team. The registration form collects 16 pieces of information about each user, most of these are selections from lists of possible values. The Evaluation Team will be able to categorize user groups based on these values. The Exit Poll appears each time that a user exits the ADL interface and asks some simple questions: what was your purpose in using ADL; usability of the interface; usability of the content; and a text box for comments, suggestion and questions.
The UCSB Evaluation Team has also contributed to the design of the help components for the user interface, in particular the tutorial, the glossary, the Frequently Asked Questions, and the system to receive, categorize and respond to user comments and questions through email. Also recommended several approaches to letting users know that their comments and questions have been received, including an immediate reply giving an idea of the expected wait interval and giving a number to call for follow-up, if necessary.
The following policy was developed by the UCSB Evaluation Team and approved by the ADL Executive Committee, 11/24/97:
Requests are received to use the Alexandria Digital Library (ADL) interface/system for evaluation studies from time to time. It is in the interest of ADL to encourage and support formal evaluation studies, especially if the results are shared with ADL. However, such evaluation studies may also require support from the ADL Team that will have an impact on staff resources. The purpose of this policy is to encourage good evaluation research studies based on the use of ADL while controlling the impact on ADL staff.
Procedure:
The Evaluation Team held the first internal evaluation of the new ADL interface design on August 19-21, using volunteers from the Project and from the UCSB campus. Twenty-three testers participated by using the interface for 30-60 minutes while they were observed and videotaped. They were given the following instructions:
Redo the search for the same area, but limit the search to aerial photographs taken from 1990 to date.
A report was written of the results of the evaluation and discussed with the Interface Evaluation Team. They used the report to guide further interface development.
The second internal evaluation period was scheduled for three days, October 20-22. Thirty-one people were scheduled to test the interface. After two days of testing with 21 people participating, the evaluations scheduled for Wednesday were cancelled. We had, at that point, collected a great deal of information from the testing that had been done so far. Another factor in the decision to cut the evaluation short was that the system had been unstable during the evaluation period.
The registration process and the exit poll were tested as well as the use of the interface for searching and retrieval. The process was similar to the first round, except that each person (or group of persons) was given an hour to use the system. They were videotaped and observed. Each tester was given the following set of scenarios to use, but were also allowed to do their own explorations if they preferred.
Scenario 1:
A. Find waterfall locations in Northern California (search the gazetteer using 'falls' as the type). Set limit for the retrieval to 30 on the Submit Search page.
B. Locate Burney Falls. Draw an area around this location and find out what is available in the ADL catalog about this area. Caution: Erase the first target area.
C. Review the thumbnail images and the metadata. Select one or two to store in your workspace. Make a folder for the things you save and name it.
D. Try (1) resorting the result sets, (2) pointing to a footprint and finding the Results listing for it, viewing different parts of the metadata, and (3) displaying the thumbnail images on the map itself.
E. Resubmit the query but limit the results to tiff images only (search the catalog using 'image'- 'tiff' as the Available-as Type.
F. Search the gazetteer directly for Burney to find Burney Falls.
Scenario 2:
You'd like to learn more about possible anchoring sites for the Channel Islands where you could moor your boat.
A. Select the target area and then search the gazetteer and see what you can find.
B. Select an anchorage area and search the catalog for information about that area. Limit the date to 1980 to current. Select some items that look good and put them in a fold named ``Channel Islands'' in your workspace.
C. Store the contents of your workspace so that you can have access to it later.
Scenario 3:
For the Western United States, find what the library has that was published by the California Division of Mines and Geology.
Scenario 4:
You want to find a particular item in the catalog and you remember the record number. The number is 113 - set the target area to the world.
Testers were asked to talk aloud as much as possible while the ``observers'' took notes. Several testers also provided comments through the exit poll. Our testers ranged from those who are very familiar with ADL to those who are only vaguely aware of what its purpose and current status is. Some testers also participated in the first round of testing.
The report of the evaluation sessions contained comments on the registration process; general comments; comments on the tutorial/help sections; searching comments categorized by unpredictable results, free-text searching, setting limit on size of result set, display of search strategy, stopping query processing, search buckets; multiple windows management; using the map browser; sorting and understanding result sets; use of the Workspace Window; metadata display; and the exit poll. The Interface Design Team used the report for guidance in their continuing design process. The Evaluation Team also compared the functionality of the interface to the guiding user requirements.
Abstracts of Published Papers
Alexandria Digital Library (ADL) is one of the six digital library projects funded by NSF, DARPA, and NASA. ADL's collections and services focus on geospatial information: maps, images, georeferenced data sets and texts, and other information sources with links to geographic locations. Throughout the project, user feedback has been collected through various formal and informal methods. These include online surveys, beta tester registration, ethnographic studies of ADL users and potential users, target user group focus sessions, and user feedback comments while using the interfaces. This paper briefly describes the evaluation studies conducted and what was learned about user characteristics and about the study approaches themselves. User reactions to the ADL interface and to the functionality and content of ADL are summarized. Finally, the value of these findings to design and implementation decisions is considered.