|Title:||Search result list evaluation versus document evaluation: similarities and differences|
|Author(s):||Iris Xie, (School of Information Studies, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin, USA), Edward Benoit III, (School of Information Studies, University of Wisconsin-Milwaukee, Milwaukee, Wisconsin, USA)|
|Citation:||Iris Xie, Edward Benoit III, (2013) “Search result list evaluation versus document evaluation: similarities and differences”, Journal of Documentation, Vol. 69 Iss: 1, pp.49 – 80|
|Keywords:||Comparison, Document evaluation, Evaluation activities, Evaluation criteria, Evaluation elements, Evaluation time, Information retrieval, Relevance criteria, Search result list evaluation, Searching|
|Article type:||Research paper|
|DOI:||10.1108/00220411311295324 (Permanent URL)|
|Publisher:||Emerald Group Publishing Limited|
|Acknowledgements:||The authors thank the University of Wisconsin-Milwaukee for its Research Growth Initiative program for generously funding the project, and Tim Blomquist and Marilyn Antkowiak for their assistance on data collection and Huan Zhang for her assistance on data analysis. The authors would also like to thank the anonymous reviewers for their constructive comments.|
|Abstract:||Purpose – The purpose of this study is to compare the evaluation of search result lists and documents, in particular evaluation criteria, elements, association between criteria and elements, pre/post and evaluation activities, and the time spent on evaluation.
Design/methodology/approach – The study analyzed the data collected from 31 general users through prequestionnaires, think aloud protocols and logs, and post questionnaires. Types of evaluation criteria, elements, associations between criteria and elements, evaluation activities and their associated pre/post activities, and time were analyzed based on open coding.
Findings – The study identifies the similarities and differences of list and document evaluation by analyzing 21 evaluation criteria applied, 13 evaluation elements examined, pre/post and evaluation activities performed and time spent. In addition, the authors also explored the time spent in evaluating lists and documents for different types of tasks.
Research limitations/implications – This study helps researchers understand the nature of list and document evaluation. Additionally, this study connects elements that participants examined to criteria they applied, and further reveals problems associated with the lack of integration between list and document evaluation. The findings of this study suggest more elements, especially at list level, be available to support users applying their evaluation criteria. Integration of list and document evaluation and integration of pre, evaluation and post evaluation activities for the interface design is the absolute solution for effective evaluation.
Originality/value – This study fills a gap in current research in relation to the comparison of list and document evaluation.
SocialStudiesOf.Info was set up in 2012 as a complement to the newly established Social Studies of Information Research Group in SOIS, the iSchool at the University of Wisconsin—Milwaukee. When setting up our research group we were looking around for a broad but coherent identity that could bring together faculty with a variety of disciplinary backgrounds and interests committed to an interdisciplinary dialogue and whose research was grounded in the methods of the humanities and social sciences.
While we individually participate in a number of related communities none of the existing labels seemed to quite fit for the group as a whole. Social Informatics is perhaps the closest, but was intended to legitimate social issues as an area of study of equal importance to other varieties of “informatics” and computer science. It therefore puts human relationships with technology, rather than with information itself, at the center of analysis. This did not encompass many topics within areas such as information history or archival studies. Science and Technology Studies inspired us with its vibrancy and inclusion of diverse disciplinary approaches, but it seemed to us more appropriate to identify information rather than science and technology as the shared object that held us together. Internet Studies and the Association of Internet Researchers are of great interest to several of our members, but clearly limit the scope of research to a specific technology.
Our informal sense of Social Studies of Information is essentially as Science and Technology Studies for researchers in the iSchool and communication school worlds. A growing number of faculty with doctoral training in science and technology studies or the history of science and technology have been hired in these schools, and many more have been heavily influenced by these approaches. A growing number of scholars with a background in communications, legal studies, or media studies are also finding common interests with Science and Technology Studies.
More formally, we realized that our interests could be described as involving the historical, social, and cultural dimensions of information, including its institutions, practices, industries, technologies, disciplines, users, policies and ethics. We believe that by framing studies broadly, for example in terms of practices and institutions, and by being tolerant of different disciplinary approaches we can engage scholars in more productive discussion. In contrast the traditional functionalist separation of scholars within the library and information science field into areas such as information seeking, information behavior, information retrieval, archives, public libraries, and so on tends to isolate scholars within very narrow fields of interest.
As our new research group began to publicize its postdoctoral position and set up our speaker series we realized that there was no efficient way to reach this community because its members were spread over so many different associations and email lists. SocialStudiesOf.Info and its associated email list is intended to act as a hub through which these scholars can interact and share resources. Our initial goals are modest but attainable. The email list will allow for discussion, announcements and questions to be shared within the community. Our syllabus repository allows participants to learn from each other and is increasingly important now that most syllabi are hidden inside course management systems. The member directory holds a short profile created by participants when they register for the site, and will help scholars find each other. We hope that the site will be particularly useful for graduate students.
Participation in the SocialStudiesOf.Info community is free and open to an international audience. Making our plans work and expanding our activities to new areas will only be possible with the active involvement of many people. The content management system we are using makes it easy to add additional features such as blogs, discussion areas, or groups focused on particular topics. So if you would like to get involved with the project then please contact us.
DLTRe serves as a supplemental resource for students of the Certificate of Advanced Studies in Digital Libraries at UW-Milwaukee, students in the L&I SCI 682: Digital Libraries course, and anyone interested in digital libraries. The website is presented in five major modules for digital library development and execution including: Collection Development, Digitization, Metadata, Interface Design, and Evaluation. Each of the modules contain introductory information on the topic, software & tutorials, sample files, practice exercises, and suggested readings.