Table of Contents Appendices Bibliography

APPENDIX J

RETRIEVAL PERFORMANCE IN CHESHIRE

Appendix J contains precision and recall ratios for all search queries submitted to CHESHIRE throughout the experiment. The causes of search failures, if applicable, are also given. The figures in Column 1 refer to search query number (Q. no.); the full text of the query can be found in Appendix I.

Precision and recall ratios obtained before the relevance feedback searches are given in Columns 2 and 3, respectively. No precision and recall ratios are available for discontinued searches. Columns 4 through 9 give precision and recall ratios obtained after relevance feedback searches. No figures are available if a relevance feedback search was not performed for a given search query. Average precision and recall ratios are given in Columns 10 and 11, respectively.

Search effectiveness for each query is given in Column 12. No data is provided in this column for out-of-domain search queries or queries that was discontinued for some reason.

The cause of search failure, if applicable, is briefly explained in Column 13.

  Precision (P) & recall (R)

ratios before relevance

 Precision & recall ratios after relevance feedback searches Average

Search

perfor-

mance:

effec-

  
Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

tive (E)/

ineffec-

  
no. P R P R P R P R P R

tive (I)

 Causes of search failure
001 .000 .850 .166 .812         .083 .831

E

  
002 .150 .750 .055 1.00         .105 .875

I

 Collection failure
003                       No clusters selected; user did not like the clusters
004                       No clusters selected; user wanted to revise the query
005 .210 .850 .105 .818         .158 .834

E

  
006 .5 .231 .00 .00         .167 .077

E

  
007 .00 .231             .000 .231

I

 Specific query
008 .263 .312 .200 .273 .00 .250     .154 .278

E

  
009                       No clusters selected; user wanted to revise the query
010 .214 .750 .667 1.00         .440 .875

E

  
011                       Faulty cluster selection
012 .00 .833 .00 .00         .00 .417

E

  
013                       Collection failure; zero retrieval
014 .00 .400 .461 .500         .230 .450

E

  
015 .167 .00             .167 .00

I

 Collection failure
016 .105 .00 .133 .00 .00 .00     .079 .00

I

 Collection failure
017                       No clusters selected; telecommunication problems
018                       Cluster failure; no clusters selected as being relevant
019 .053 .033             .052 .033

I

 Collection failure
020                       Collection failure; zero retrieval
021                       Collection failure; zero retrieval
022 .222 .333             .222 .333

E

  
023 .316 .333 .200 .333         .258 .333

E
  • (Continued)
    •
    •   Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Causes of search failure
    024 .316 .727 .312 .555         .314 .641

    E

    		  
    025 .00 .100 .00 .00         .00 .500

    E

    		  
    026 .368 .467 .00 .375         .184 .421

    E

    		  
    027                       No clusters selected; user interface problems
    028 .067 .850 .00 1.00         .033 .925

    I

    		 Search statement
    029 .400 .133 .00 .385         .200 .259

    I

    		 User interface problems
    030 .00 .00             .00 .00

    I

    		 Out of domain search query
    031 .316 .350 .150 .461 .30 1.00     .255 .604

    E

    		  
    032 .667 .417 .100 .286 .00 .00     .255 .234

    I

    		 Vocabulary problem
    033                       No clusters selected; out of domain search query
    034 .00 .00             .00 .00

    I

    		 Collection failure
    035                       No clusters selected; out of domain search query
    036 .667 .600             .666 .600

    E

    		  
    037                       Zero retrieval; call number search
    038 .00 .00             .00 .00

    I

    		 Broad search query
    039 .00 .400 .00 .417         .00 .408

    E

    		  
    040                       No clusters selected as being relevant
    041                       No clusters selected; user interface problems
    042 .053 .250 .055 .333 .50 1.00     .202 .528

    E

    		  
    043                       Zero retrieval; help request
    044                    

    I

    		 Help request
    045                       Zero retrieval; "the" is a stop word
    046 .00 .00 .00 .500         .00 .250

    E
  • (Continued)
    •
    •   Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Causes of search failure
    047 .053 .555             .053 .555

    E

    		  
    048 .273 .643 .00 1.00         .136 .821

    E

    		  
    049 .75 .9 .00 1.00         .375 .95

    E

    		  
    050                       No clusters selected; user wanted to revise the query
    051 .053 .25 .091 .667         .072 .458

    I

    		 Collection failure
    052 .267 .25 .00 .00         .133 .125

    E

    		  
    053                       No clusters selected; user interface problems
    054 .5 .85 .00 .778         .25 .814

    E

    		  
    055                       No clusters selected; out of domain search query
    056 .5 .5 .00 .00 .00 .00     .167 .167

    I

    		 Collection failure
    057 .067 1.00             .067 1.00

    E

    		  
    058                       No clusters selected; user wanted to revise the query
    059                       No clusters selected; user wanted to revise the query
    060 .263 1.00 .053 .00         .158 .5

    E

    		  
    061 .474 .45 .00 .15 .44 .7     .306 .433

    E

    		  
    062 .00 .00 .00 .00 .00 .00     .00 .00

    I

    		 Collection failure; search statement
    063 .1 1.00             .1 1.00

    E

    		  
    064                       Zero retrieval; user entered gibberish characters
    065                       Collection failure; no clusters selected
    066 .00 .25             .00 .25

    I

    		 Collection failure; most recent items needed
    067                       Scope failure; periodical literature search
    068                       Cluster failure; user did not like the clusters
    069                    

    I

    		 Scope failure; periodical literature search (Continued)
      Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Causes of search failure
    070                    

    I

    		 User did not like clusters; no clusters selected as relevant
    071                       Out of domain search query; zero retrieval
    072                    

    I

    		 User interface problems; no clusters selected as being relevant
    073 .00 .00             .00 .00

    I

    		 User interface problems; collection failure
    074                    

    I

    		 Stemming algorithm; did not recognize "C"; zero retrieval
    075                    

    I

    		 Stemming algorithm failure; no clusters selected as relevant
    076                       No clusters selected; user wanted to revise her query
    077 .00 .850 .00 .650 .40 1.00     .133 .833

    E

    		  
    078 .091 1.00 .00 .00         .045 .500

    I

    		 Known-item search
    079                       User typed "quit" in the query description screen
    080 .667 1.00             .667 1.00

    I

    		 Collection failure
    081 .500 .200             .500 .200

    E

    		  
    082 .500 .650             .500 .650

    E

    		  
    083 .579 .850 .250 .450 .11 .100 .00 .00 .235 .350

    E

    		  
    084                       Out of domain search query
    085                       No clusters selected; out of domain search query
    086 .555 .800             .555 .800

    E

    		  
    087                       Out of domain search query
    088 .00 .555             .00 .555

    E

    		  
    089 .800 .650 .833 .454         .817 .552

    E

    		  
    090                       Author search; not supported in CHESHIRE; false drops
    091                       Author search; not supported in CHESHIRE; zero retrieval
    092                       Author search; not supported in CHESHIRE; zero retrieval
      Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Causes of search failure
    093                       Author/title search; not supported in CHESHIRE
    094                       Quit entered in the query description screen
    095 .00 .00             .00 .00

    I

    		 Collection failure
    096 .368 1.00 .100 .00         .234 .500

    E

    		  
    097                       Author search; not supported in CHESHIRE
    098                       No clusters selected as being relevant
    099 .500 .650 .500 .800         .500 .725

    E

    		  
    100                       Out of domain search query
    101 .667 .300 .00 .350         .333 .325

    E

    		  
    102 .500 .00             .500 .00

    I

    		 Collection failure
    103 .210 1.00             .210 1.00

    E

    		  
    104 .105 1.00 .00 1.00         .053 1.00

    E

    		  
    105 .263 .500 .00 .00         .131 .250

    E

    		  
    106                       Out of domain search query
    107                    

    I

    		 Collection failure; no clusters selected as being relevant
    108                    

    I

    		 Collection failure; zero retrieval
    109 .222 .300 .300 .300 .00 .150     .173 .250

    E

    		  
    110                    

    E

    		  
    111                    

    I

    		 Telecommunication problem; no cluster selected
    112 .053 .00 .050 .00 .00 .00     .051 .00

    I

    		 Collection failure
    113                    

    I

    		 False drops; no clusters selected as being relevant
    114* .158 .210 .450 .600 .00 .00 .00 .00 .122 .162

    E

    		  
    115 .500 .00 .00 .00 .05 .00     .183 .00

    E
  • (Continued)
    •
    •   Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Cause of search failure
    116 .158 .200 .150 .875 .05 1.00     .119 .692

    E

    		  
    117 .263 .263 .105 .143         .184 .203

    E

    		  
    118                    

    I

    		 Stemming algorithm; "r&d" not recognized; zero retrieval
    119 .684 .850 .250 .450         .467 .650

    E

    		  
    120                       Out of domain search query
    121                       Out of domain search query; no clusters selected as relevant
    122 .333 .333 .500 1.00         .117 .667

    I

    		 Out of domain search query
    123 .263 .750 .100 .500 .10 .50     .154 .583

    E

    		  
    124 .00 .800 .00 1.00         .00 .900

    E

    		  
    125                    

    I

    		 Collection failure; no clusters selected as being relevant
    126 .00 1.00             .00 1.00

    E

    		  
    127 .00 .400             .00 .400

    I

    		 Library of Congress Subject Headings
    128 .272 .600 .00 .00         .136 .300

    I

    		 Library of Congress Subject Headings
    129 .00 1.00             .00 1.00

    E

    		  
    130                       Out of domain search query; user wanted to revise the query
    131 .00 .500             .00 .500

    I

    		 Collection failure
    132 .00 1.00             .00 1.00

    E

    		  
    133                       Out of domain search query
    134                       Out of domain search query
    135                       Out of domain search query; no clusters selected as relevant
    136                       Out of domain search query; no clusters selected as relevant
    137                       Out of domain search query; no clusters selected as relevant
    138                      
  • Out of domain search query; zero retrieval (Continued)
    •
    •   Precision (P) & recall (R) ratios before relevance Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Cause of search failure
    139                       Out of domain search query; zero retrieval
    140 .105 1.00 .050 .00 .00 .00 .00 .00 .039 .250

    I

    		 Collection failure
    141                    

    I

    		 Collection failure
    142 .368 .615 .077 .200         .223 .408

    E

    		  
    143                       No clusters selected; user wanted to revise her query
    144                       No clusters selected; user wanted to revise her query
    145 .00 .615 .00 .00         .00 .308

    E

    		  
    146 .500 .00             .500 .00

    I

    		 Collection failure
    147 .316 .545 .250 1.00 .00 .00     .205 .515

    E

    		  
    148 .500 .769 .00 .00         .250 .385

    E

    		  
    149 .00 .500             .00 .500

    E

    		  
    150 .158 1.00             .158 1.00

    I

    		 Collection failure
    151                    

    I

    		 Cluster failure; no clusters selected as being relevant
    152 .053 .769 .125 .333 .00 .00     .059 .367

    E

    		  
    153                    

    I

    		 Collection failure
    154                    

    I

    		 Collection failure; no clusters selected as being relevant
    155                       Out of domain search query; zero retrieval
    156 .00 .900 .00 .400         .00 .650

    E

    		  
    157                    

    I

    		 No apparent reason; relevant clusters, but not selected
    158 .00 1.00             .00 1.00   Out of domain search query
    159                       Out of domain search query
    160 .00 1.00 .00 .00         .00 .500   Out of domain search query
    161                      
  • Out of domain search query (Continued)
    •
    •   Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Cause of search failure
    162 .00 .333 .00 .00         .00 .167

    I

    		 Specific query
    163 .053 .100             .053 .100

    I

    		 Collection failure
    164 .00 .667             .00 .666

    E

    		  
    165 .00 .00                

    I

    		 Collection failure
    166                    

    I

    		 Cluster failure
    167                       No clusters selected; user wanted to revise his query
    168                       No clusters selected; user wanted to revise his query
    169 .526 .950 .00 .437         .263 .694

    E

    		  
    170 .526 .850 .300 .500         .413 .675

    E

    		  
    171 .143 .500 .050 1.00         .096 .750

    E

    		  
    172                       No clusters selected as being relevant
    173                       No clusters selected as being relevant
    174* .167 .500 .100 .500 .17 .500 .00 1.00 .097 .500

    I

    		 Library of Congress Subject Headings
    175 .263 .00             .263 .00

    I

    		 Collection failure
    176 .538 1.00 .267 1.00 .10 1.00 .00 .00 .226 .750

    I

    		 No apparent reason given
    177                       Out of domain search query; no clusters selected as relevant
    178 .500 .429             .500 .429

    I

    		 Search statement; abbreviation used
    179 .053 1.00             .053 1.00

    E

    		  
    180 .421 .00 .00 .00         .210 .00

    I

    		 Collection failure
    181 .368 .214 .250 .364 .25 .714     .289 .431

    I

    		 Library of Congress Subject Headings
    182                       Out of domain search query
    183                       Out of domain search query
    184 .158 .250 .250 .210         .204 .230

    I

    		 Search statement; truncated word not recognized (Cont'd)
      Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Cause of search failure
    185 .200 1.00 .00 .00         .100 .500

    E

    		  
    186 .00 .00 .00 .00         .00 .00

    I

    		 Collection failure
    187                       No clusters selected as being relevant
    188 .00 1.00 .00 .00         .00 .500

    E

    		  
    189 .00 .00             .00 .00

    I

    		 User interface problems
    190 .00 .00 .500 .00         .250 .00

    I

    		 User interface problems
    191                       Collection failure; no clusters selected as being relevant
    192 .214 .571 .062 .333 .00 1.00     .092 .635

    I

    		 Library of Congress Subject Headings
    193                       Collection failure
    194 .053 .200 .00 .00 .00 .00     .017 .067

    I

    		 Search statement
    195 .250 .300 .100 .300         .170 .300

    I

    		 Out of domain search query
    196                       Out of domain query; no clusters selected as being relevant
    197                       Out of domain search query
    198 .00 .00             .00 .00

    I

    		 Collection failure
    199 .263 .818 .050 1.00         .156 .910

    E

    		  
    200                       Title search; not supported in CHESHIRE
    201 .100 1.00             .100 1.00   Title search; not supported in CHESHIRE
    202                       Title search; not supported in CHESHIRE
    203                       Title search; not supported in CHESHIRE
    204                       Title search; not supported in CHESHIRE
    205                       Title search; not supported in CHESHIRE
    206                       Title search; not supported in CHESHIRE
    207                      
  • Title search; not supported in CHESHIRE Continued
    •
    •   Precision (P) & recall (R)

    ratios before relevance

    		 Precision & recall ratios after relevance feedback searches Average

    Search

    perfor-

    mance:

    effec-

    		  
    Q. feedback searches First iteration Second iteration Third iteration precision & recall ratios

    tive (E)/

    ineffec-

    		  
    no. P R P R P R P R P R

    tive (I)

    		 Causes of search failure
    208 .00 .00             .00 .00

    I

    		 Collection failure
    209                       Stemming algorithm failure; abbreviation ("e") not recognized
    210 .789 .750 .700 .867         .744 .808

    E

    		  
    211 .200 .650 .00 .100         .100 .370

    E

    		  
    212                       Out of domain search query
    213                       No clusters selected as being relevant
    214                       No clusters selected as being relevant
    215 .00 .00             .00 .00   Out of domain search query
    216                       Out of domain search query; no clusters selected as relevant
    217 .947 .900 .800 .800         .874 .850

    E

    		  
    218                       Out of domain search query; no clusters selected as relevant
    219                    

    I

    		 No reason given; relevant clusters retrieved but not selected
    220                       Out of domain search query; no clusters selected as relevant
    221 .316 .350 .222 .210         .269 .280

    E

    		  
    222                    

    I

    		 Collection failure; no clusters selected as being relevant
    223 .053 .00             .053 .00

    I

    		 Search statement
    224 .053 .400 .100 .400 .06 .200 .00 .250 .054 .312

    I

    		 Faulty cluster selection
    225 .526 .950 .454 .700 .00 .850     .327 .833

    I

    		 Search statement
    226                       Misspelling; zero retrieval
    227 .105 .625 .00 .333         .053 .479

    E

    		  
    228 .846 .55             .846 .550

    E

    		  
     
    Table of Contents Appendices Bibliography
    © Yaşar Tonta, 1992
    tonta@hun.edu.tr