Selecting Success Criteria: Experiences with an Academic Library Catalogue

• Published in: Information Access Evaluation. Multilinguality, Multimodality, and Visualization pp. 59 - 70
• Main Authors: Clough, Paul, Goodale, Paula
• Format: Book Chapter
• Language: English
• Published: Berlin, Heidelberg Springer Berlin Heidelberg 2013
• Series: Lecture Notes in Computer Science
• Subjects: digital libraries; Evaluation; success criteria
• ISBN: 9783642408014; 364240801X
• ISSN: 0302-9743; 1611-3349
• DOI: 10.1007/978-3-642-40802-1_7

Multiple methods exist for evaluating search systems, ranging from more user-oriented approaches to those more focused on evaluating system performance. When preparing an evaluation, key questions include: (i) why conduct the evaluation, (ii) what should be evaluated, and (iii) how the evaluation should be conducted. Over recent years there has been more focus on the end users of search systems and understanding what they view as ‘success’. In this paper we consider what to evaluate; in particular what criteria users of search systems consider most important and whether this varies by user characteristic. Using our experience with evaluating an academic library catalogue, input was gathered from end users relating to the perceived importance of different evaluation criteria prior to conducting an evaluation. We analyse results to show which criteria users most value, together with the inter-relationships between them. Our results highlight the necessity of conducting multiple forms of evaluation to ensure that search systems are deemed successful by their users.