CHAPTER I
INTRODUCTION
No one wants to learn by mistakes, but we cannot learn enough from successes to go beyond the state of the art.
--Henry Petroski, To Engineer Is Human: The Role of Failure in Successful Design. (New York: Vintage Books, 1992), p.62.
1.0 Rationale of the Study
Online catalog users often fail in their attempts to retrieve relevant items from document collections using existing online library catalogs. Most users experience problems especially when they perform subject searching in online catalogs. Confronted with an online catalog that lacks guidance or adequate help features, users tend to abandon their searches without questioning the causes of search failures and the effectiveness of the online catalog.
Although it is users who usually endure online catalogs with ineffective user interfaces and struggle with inflexible indexing and query languages, their involvement in the analysis of search failures is seldom sought. Studies with no user involvement tend to focus on what might have happened during a search, rather than what actually happened. Causes of search failures in online catalogs can be studied best when the users provide invaluable feedback regarding their search queries and retrieval results.
This study is an attempt to investigate the causes of search failures in a third generation experimental online library catalog. It is particularly concerned with the evaluation of retrieval performance in online library catalogs from the users' perspective. The analysis of retrieval effectiveness and search failures was based on transaction log records, questionnaires and critical incident technique. User-designated ineffective searches in an experimental online catalog have been compared with transaction log records in order to identify the possible causes of search failures. The mismatch between the users' vocabulary and the vocabulary used in online library catalogs has been studied so as to find out its role in search failures and retrieval effectiveness. An attempt to develop a conceptual model to categorize search failures in online library catalogs was made.
This study evaluates the retrieval performance of an experimental online catalog by: (1) using precision/recall measures; (2) identifying user-designated ineffective searches; and (3) comparing user-designated ineffective searches with the precision/recall ratios for corresponding searches.
Findings obtained from this study can be used to design better online library catalogs. Designers equipped with information about search failures should be able to develop more robust online catalogs which guide users in their search endeavors. Search failures due to vocabulary problems can be minimized by strengthening existing indexing languages and/or by developing "entry vocabulary systems" to relate users' terms to systems' terms. The results may help improve our understanding of the role of natural query languages and indexing in online catalogs. Furthermore, the findings may provide invaluable insight that can be incorporated in future retrieval effectiveness and relevance feedback studies. The conceptual model developed can be used in other studies of search failures in online catalogs. From the methodological point of view, using critical incident technique may prove to be invaluable in studying search failures and evaluating retrieval performance in online library catalogs.
1.1 Objectives of the Study
The purpose of the present study is to:
1. analyze the search failures in online catalogs so as to identify their probable causes and to improve the retrieval effectiveness;
2. measure the retrieval effectiveness in an experimental online catalog in terms of precision and recall;
3. compare user-designated ineffective searches with the effectiveness results obtained through precision and recall measures;
4. ascertain the relationship between performance of the system as measured by precision and recall and variables that defined user characteristics and users' assessment of retrieval effectiveness;
5. ascertain the extent to which users' natural language-based queries match the titles of the documents and the Library of Congress Subject Headings (LCSH) attached to them;
6. identify the role of relevance feedback in improving the retrieval effectiveness in online catalogs;
7. identify the role of natural query languages in improving the match between users' vocabulary and the system's vocabulary along with their retrieval effectiveness scores in online catalogs;
8. develop a conceptual model to categorize search failures that occur in online library catalogs.
1.2 Hypotheses
Main hypotheses of this study are as follows:
1. Users' assessments of retrieval effectiveness may differ from retrieval performance as measured by precision and recall;
2. Increasing the match between users' vocabulary and system's vocabulary (e.g., titles and subject headings assigned to documents) will help reduce the search failures and improve the retrieval effectiveness in online catalogs;
3. The relevance feedback process will reduce the search failures and enhance the retrieval effectiveness in online catalogs.
1.3 Method
Transaction monitoring and critical incident techniques were used for data gathering in this study. The former method allows one to study the users' search behaviors unobtrusively while the latter helps gather information about user intentions and needs for each query submitted to the system. The critical incident technique, which will be described in Chapter III, is used for the first time, to our knowledge, in this study to examine search failures in online library catalogs.
Users participating in the study were allowed access to an experimental online catalog with more than 30,000 records for a period of one semester (14 weeks). Search queries that the users submitted to the system, the items they retrieved and displayed were recorded in transaction logs along with some other relevant data. These transaction logs were later reviewed to find out the retrieval effectiveness of the online catalog under investigation.
As the logs also included data about the users (e.g., their login id) it was possible to identify the person who submitted each query to the system. Users were later invited to share their experience with regard to the searches they performed on the system. Their comments were audiotaped. A critical incident report was completed for each query based on the user's experience. They also were asked to fill out a questionnaire for each search.
The information furnished by the user for each query regarding its effectiveness was compared with the transaction log records. The searches that the users designated as being `failures' were identified from the critical incident forms and corroborated with the transaction log records. Users' audiotaped comments were also used to analyze the probable causes of the search failures. Thus, it was possible to determine the performance of the online catalog for each search query using both retrieval effectiveness measures such as precision and recall and the user designated search effectiveness.
The critical incident technique proved useful in the analysis of search failures in online catalogs. Incident reports provided invaluable information about each search query regarding its effectiveness. Furthermore, comparison of critical incident reports with the transaction log records was very helpful in identifying and, consequently, analyzing search failures.
1.4 Organization of the Study
This report consists of eight chapters, a select bibliography, and accompanying appendices. The rationale, objectives, hypotheses, and method of the study are introduced in Chapter I, while Chapters II and III form the theoretical foundations of the present study.
Chapter II examines document retrieval systems in general terms. Retrieval effectiveness measures are defined in Chapter II. Relevance feedback and clustering techniques are also discussed in this chapter.
Chapter III opens with a critical review of methods used in the analysis of search failures in document retrieval systems. A comprehensive review of failure analysis studies is given here.
Chapter IV develops a conceptual model to categorize search failures that occur in online catalogs. Types of search failures are examined by means of a four-step ladder model.
A detailed account of the experiment conducted for this study is presented in Chapter V. It explains the environment in which the experiment has been carried out, provides information about the subjects who participated in the study, and illustrates the tools and methods that were used to gather, analyze and evaluate data.
Findings obtained in this study are presented in Chapter VI and VII. Chapter VI summarizes the descriptive data obtained from the transaction logs, questionnaire forms, and critical incident reports. The results of multiple linear regression analysis are also presented in Chapter VI. The detailed analysis of search queries and search failures is given in Chapter VII.
Chapter VIII gives a brief summary of the findings obtained in this study along with conclusions and recommendations for further research.