Next: Evaluation Objectives Up: Evaluation Plan Previous: Evaluation Plan

Evaluation Methods

This project will use two methods of data collection, transaction monitoring and online questionnaires, to obtain information on how the Cheshire system is used, and on the reactions and opinions of users regarding system features and capabilities. Transaction monitoring involves recording all significant interactions between the users and the system, including the commands or menu selections used, the bibliographic records or index entries displayed, and users' judgements as to the relevance of retrieved items. The data collected by transaction monitoring can be used to completely reconstruct the users' interactions with the system for detailed analysis of the types of searches being conducted and their results. The current transaction monitoring software for the Cheshire system will be enhanced to provide additional summary information about the use of the various features of the system.

Although transaction monitoring provides reliable and objective information on how users interact with the system, it does not provide insight into why users select particular options, or how they feel about the features and capabilities used. To obtain this information an online questionnaire (that is, questions and instructions are automatically provided to the users on the same workstation that they use for searching Cheshire, or via the network for remote users) will be developed to ask the users' opinions on these issues.

The questionnaire will be based on the user questionnaire developed for the Council on Library Resources nation-wide survey of online catalog use and users[16], on a questionnaire designed to elicit user reactions to interface features developed by Prof. Ben Shneiderman of the University of Maryland's Human-Computer Interaction Laboratory[21], and on an evaluation checklist created by Susannah Ravden and Graham Johnson[20]. The advantages of using an online questionnaire are that the responses are recorded along with transaction monitor data for easy comparison, and that no staff time is required to administer the questionnaire. In addition, the online questionnaire makes it possible to collect data on network users who might be located anywhere in the world: something that would be impossible using conventional paper questionnaires.

Next: Evaluation Objectives Up: Evaluation Plan Previous: Evaluation Plan

Contact: Ray R. Larson