APPENDICES
Appendix A: Background Information About CHESHIRE and Guidelines for CHESHIRE Searches 237
Appendix B: Access to CHESHIRE: An Experimental Online Catalog (Instructions) 241
Appendix C: Transaction Log Record Format 266
Appendix D: Questionnaire 269
Appendix E: Critical Incident Report Form for Effective Searches 272
Appendix F: Critical Incident Report Form for Ineffective Searches 274
Appendix G: Invitation Letter Sent to MLIS Students 276
Appendix H: Invitation Letter Sent to Ph.D. Students 279
Appendix I: Queries Submitted to CHESHIRE 282
Appendix J: Retrieval Performance in CHESHIRE 292
APPENDIX A
BACKGROUND INFORMATION ABOUT CHESHIRE AND
GUIDELINES FOR CHESHIRE SEARCHES
This appendix reproduces a handout distributed to potential participants before the experiment began. It introduces the experiment to participating users and explains what they are asked to do for the experiment.
BACKGROUND INFORMATION ABOUT CHESHIRE AND
GUIDELINES FOR CHESHIRE SEARCHES
Before I explain what I would like you to do for this research, let me briefly summarize what CHESHIRE is all about and why your participation is important.
CHESHIRE is one of the next generation online catalogs that is designed to accommodate sophisticated information retrieval (IR) techniques based on sound theoretical backing. The database for the CHESHIRE system consists of some 30,000 records representing the holdings of the Library School Library here at UC Berkeley. The size of the database makes CHESHIRE one of the largest systems that has ever been used for IR research and experimentation.
As it is well known, existing online catalogs in use are based on Boolean logic and simple keyword matching techniques, which are hard to use, brittle, and unforgiving: more than one third of the searches retrieve nothing! CHESHIRE, on the other hand, offers further improvements: it accommodates search queries in natural language form. The user describes his/her information need using words that are taken from the natural language and submits this statement to CHESHIRE. CHESHIRE "evaluates" the query, identifies the records that are most similar to the user's query and comes up with a ranked list of "would-be" relevant records. Furthermore, CHESHIRE is able to incorporate users' relevance judgments through what is called "relevance feedback process," which increases the chance of retrieving more relevant documents. That is to say, no matter how poorly the information need is explained, CHESHIRE always retrieves some relevant documents and it helps users to clarify their intentions by way of relevance feedback mechanism. Such features are lacking in traditional online catalog systems.
As for my research, I am trying to find out the causes of search failures in online catalogs. Catalog searches may fail due to a variety of reasons such as a clunky user interface, indexing and vocabulary problems, rigid command languages and retrieval rules. Findings to be obtained from this research can be used in designing better online library catalogs. Designers equipped with information about search failures should be able to develop more robust and "fail-proof" online catalogs. The size of the CHESHIRE database offers a remarkable opportunity to obtain more reliable research results since most IR experiments in the past have been conducted on small test collections, findings of which do not necessarily "scale-up" to large bibliographic databases.
You are to play a very important role in this research. I am sure you are familiar with some other online catalogs. Yet most, if not all, of you presumably never used CHESHIRE before. You are kindly requested to try CHESHIRE and do some searches on it. We will record your entire search (i.e., query entered, records displayed, relevance judgments) in a transaction file so that we can later analyze these records and determine the retrieval effectiveness of CHESHIRE.
Here is what I would like you to do:
1. Go to the Computer Laboratory in the second floor of South Hall and log on to the SLIS Local Area Network using your regular login and password.
2. Once you are in the Main menu (of the SLIS network), follow the instructions in the document entitled "Access to CHESHIRE: An Experimental Online Catalog" in order to connect to CHESHIRE. (This document will be handed out in one of your classes.)
3. When you get to the disappearing cat screen (smiling CHESHIRE cat), type a natural language query of your choice (an example of the search process is provided in the above document). Please note that since the CHESHIRE database is restricted to the holdings of the Library School Library in South Hall, questions that could be answered from this collection will get the best results.
Examples of subfields which are supported by the Library School Library are as follows: librarianship and information science in general, publishing and the book arts, management of libraries and information services, bibliographic organization, censorship and copyright, children's literature, printing and publishing, information policy, information retrieval, systems analysis and automation of libraries, archives and records management, office information systems, use of computers in libraries and information services.
4. Mark relevant clusters and records by pressing "s" and skim through records. (You might want to write down or download relevant ones so that you can obtain them from the stacks of the Library School Library.)
5. Perform a "relevance feedback search" and see if it improves the search results.
6. Repeat the above process whenever you have a query that can be answered from the collection of the Library School Library.
7. You might wish to try the same searches on GLADIS or MELVYL and compare the results with that which you obtained through CHESHIRE.
8. After a couple of searches on CHESHIRE, you will be able to compare and contrast the following features of online catalogs:
a) natural language-based queries vs. command languages (e.g., "subject access in online library catalogs" (in CHESHIRE) vs. "FIND SU SUBJECT ACCESS AND ONLINE CATALOGS" (in MELVYL)).
b) relevance feedback mechanism and its use in CHESHIRE.
c) use of LC classification system for subject access in CHESHIRE.
d) "information overload" and "zero retrieval" in traditional online catalogs. (CHESHIRE solves "overload" problem by presenting records in ranked order so that the most promising records will be displayed first. CHESHIRE almost always retrieves something from the database, unless there is no record in the database for a given query.)
yt 9/91
APPENDIX B
ACCESS TO CHESHIRE: AN EXPERIMENTAL ONLINE CATALOG
(Instructions)
This appendix reproduces a handout distributed to MLIS and Ph.D. students who agreed to participate in the study. It includes step-by-step instructions as to how to get access to CHESHIRE through the local area network of the School of Library and Information Studies at the University of California at Berkeley. It also explains how to perform an online search on CHESHIRE.
Access to CHESHIRE:
An Experimental Online Catalog
(Instructions)
by
Yasar Tonta
Berkeley
September 1991