Colorado Component of Evaluation Team

UE5

Interface Help/Smartification
PLANNED ACTIVITY
March 12, 1997: copyediting/revision of existing pages complete. Glossary and tutorial pages will begin update cycle.
July 1, 1997: work on interface smartification may need to be revisited and revised as the new interface design is completed.
February 28, 1998: iterate with UCSB testbed team as new system functions come online
ACTUAL ACTIVITY
Copies of the copyedited Web pages were delivered in March 1997, on schedule. Relevant copyedits were incorporated into the new JAVA version of the ADL interface. Task is complete.

UE6

User Scenarios/Profiles/Models
PLANNED ACTIVITY
March 12, 1997: CU will hold a third focus group session in early March, with librarians, and with software demos of scenarios on the compiled list. Additional focus groups sessions targeting earth scientists will continue through summer and fall, at various research centers (such as LTER and NCAR). Transaction logs for scenario-based structured browsing will begin.
July 1, 1997: focus groups will provide final user profiles for the three classes of users; transaction log collection will continue, and statistical models of use patterns for the three target user classes will be developed. With completion of the new interface design, new focus group sessions may be required.
February 28, 1998: Transaction logs for the new interface will summarize several months of new data; initial results of statistical modeling of user navigation will be complete. Initial reports of this work will be presented in August at the International Cartographic Conference (ICC '97) and in October at the Conference on Spatial Information Theory (COSIT '97).
ACTUAL ACTIVITY
Focus groups were held as scheduled in March and July 1997; results were reported to UCSB UIE Team. Results of transaction logging were reported in May 1997 at the NSF site visit. Findings from nearly 200,000 transactions were analyzed using spatial interaction modeling, an inferential method utilized in studying patterns of geographic transportation. Patterns of user navigating between Web pages formed the unit of observation. Findings demonstrated that users do follow structured paths through ADL, and that these patterns did not change substantially over the twelve month period for which log data were analyzed, in spite of interface design modifications. In particular, the logs demonstrated that users have trouble navigating through the testbed, and that they do not tend to access the online tutorials and help files. These findings are consistent with user-volunteered statements about the complexity of the ADL site.
Several of the findings were incorporated into the JAVA interface, for example, by merging the catalog search with the gazetteer search functions, by framing multiple concurrent windows (as opposed to a single long scrolling page), and by providing functions to save partial searches for later refinement. The Colorado Team has begun to disaggregate the Web version logs by target user group, which has proven challenging since the Web logs do not carry identifiable user-id numbers. Thus we must reconstruct target users from IP address identification alone. Disaggregation has been a major focus through the summer and fall; initial results indicate that librarians use patterns tend to differ from the full group in sometimes significant ways, ironically, in their use of the tutorials and help files.

UE6

Evaluation Criteria
PLANNED ACTIVITY
March 12, 1997: complete joint UIE report; submit preliminary presentation of results to JASIS Spring convention.
July 1, 1997: Comparison of evaluative criteria will continue according to needs developing with implementation of the new interface.
February 28, 1998: iterate with testbed team on implementation of further evaluation tools as new interface tools come online
ACTUAL ACTIVITY
This task integrated findings of evaluation by multiple methods, including focus groups, transaction logging, interviews, searching for which evaluation methods prove most reliable and which prove most informative. A report on these integrative findings was submitted to JASIS in March 1997 by the UCSB team with input from Colorado. During the past year, both teams have continued to compare results but there is at present insufficient data collected on the JAVA interface to make a formal report. A discussion in February 1998 between the UCSB and Colorado co-leaders of UIE reviewed findings and concluded that results from multiple evaluations make a clear indication that the user survey is not only too long, but placed (as it has been) at the end of the sequence of Web pages users customarily traverse, it is not being accessed. In the JAVA version, demographic information is solicited from users at the beginning of the ADL session, and tied to user registration. AT Colorado, an experimental JAVA registration applet is being designed and tested this spring to determine how much data can be surveyed from users as they enter the ADL testbed.

UE7

Spatially Dense Information Representation
PLANNED ACTIVITY
March 12, 1997: The dissertation on representing gradation will be defended in March. Results of the spatial metaphor research will be presented at AUTO-CARTO conference in April; that dissertation will be defended in fall.
July 1, 1996: Initial designs for subject testing spatial metaphors for returned query hits.
February 28, 1997: Begin initial testing on spatial metaphors.
ACTUAL ACTIVITY
Two dissertations were reported in progress last year, and the first (defining a taxonomy of gradation and uncertainty) was defended on schedule and is abstracted in the publications). The second dissertation will be defended late this spring, and sets forth the computational aspects for creating spatialized views of large numbers of query hits. Papers are abstracted above reporting on these results, and full descriptions of the computations required to generate these displays was delivered to the Implementation Team (to Robert and Nathan, January 1998). In July, Colorado began to design experiments for subject testing the spatialized interface metaphors, as scheduled in last year's workplan. A preliminary paper (Fabrikant and Buttenfield, 1997) was presented at GIS/LIS 97 and a second (Fabrikant, 1998) is being presented at the Boston AAG meetings in March 1998; both are abstracted above.

UE8

User Interface Evaluation
PLANNED ACTIVITY
March 12, 1997: preliminary screen shots of new interface with completed metaphor proposal; ready for presentation at May site visit.
July 1, 1997: Initiate user evaluation studies of the new interface and/or separate components. Presentation of user evaluation studies to DLI meeting in Pittsburgh.
February 28, 1998: Ongoing user evaluation studies of the new interface, iterating with Testbed Implementation Team and rest of UIE Team. Interface will be refined according to user response, profiled requirements, and system functionality. A third Alexandria Design Review Workshop will be scheduled for early in 1998.
ACTUAL ACTIVITY
This (as in last year's report) is an ongoing task. The Colorado Team proceeded on schedule as per last year's workplan, and will work closely with the JAVA Implementation Team and the UCSB UIE Team in the coming calendar year, as the JAVA interface design becomes fully implemented. We ran two focus groups last fall (September and October) and reported results to the UCSB UIE Team. Both focus groups were run on short notice and delivered on time.