[The DjVu original here http://www.larryblakeley.com/Articles/scopus_wp1_usability_testing20040616.djvu] Abstract This paper describes how the User-Centered Design (UCD) method has been applied to the development of a large bibliographic search and navigation system called Scopus, which is due to be launched in the final quarter of 2004. In cooperation with 21 research institutions, more than 300 researchers and librarians have participated in numerous rounds of onsite and remote testing of various concepts, using mock-ups and functional prototypes of the product. The verbal and behavioral feedback from these user test sessions was employed to acquire a detailed understanding of the scientists' information needs, tasks and workflow. This understanding is fundamental to content, functionality and design decisions. The University Library System of the University of Pittsburgh was one of these partners, and in this paper the UCD process is discussed in general and with specifics from both the developer standpoint and the librarian experience. Background Researchers require access to a huge amount and variety of information sources in order to accomplish their work. They use scientific information to design their research, to stay current with developments in their own and related fields, to work collaboratively with others, to critically evaluate and examine the work of others, and, perhaps most importantly, to contribute to their teaching efforts. However, searching for and evaluating this information is a time-consuming and daunting task. Researchers can feel overwhelmed by the ever-increasing rate of information becoming available, and may not feel adequately equipped to keep abreast of, or to evaluate this information. A challenge faced by librarians, who attempt to provide access to this universe of information, is that many current information systems created to help researchers find the resources they need are driven primarily by technological feasibility and not enough by end-user needs. It is often technologists who design these systems, with little input from users. These systems must be complex enough to offer adequate access to limitless information sources, often at the expense of the user experience. They tend to be counter-intuitive and require specialized training to take the best advantage of them. Therefore, the challenge is to develop an information system that is sophisticated enough to provide researchers with the resources they need, while also being relatively intuitive. It must be quick and easy for the user to find relevant information. Designing such a system requires a reliable and detailed grasp of how researchers approach an information retrieval task. It requires an understanding of a user's workflow, information seeking behavior and evaluation. Understanding research behavior is not an easy task and cannot be done in isolation. It requires partnering with researchers to build the information systems of their dreams. User-Centered Design The approach adopted to build Scopus is known as User-Centered Design. It forms a comprehensive part of product development. It is task-oriented and focuses on user needs, limitations and preferences rather than system capabilities. Generally, the User-Centered Design approach consists of three phases: The first phase is focused on understanding the users, their tasks, and their work environment. In early phases of development, central questions are "Who are the people that are going to use the system?" "What do they want to accomplish?" and "What are their specific types of information needs?". In later stages of development questions are focused on very specific types of user tasks. In the second phase, designs are produced based on feedback from the previous phase, as well as on general knowledge of human information processing capabilities and limitations. In early stages of development this usually means designing static 'wireframes' of screens conveying the proposed information architecture. In later stages, a detailed and functional prototype including a graphical style is developed. During the third phase, evaluation of the designs takes place. Any hypotheses and assumptions about user behavior in specific task contexts are tested during this phase. Depending on the maturity of the ideas, the evaluations may be in the form of conceptual tests or usability tests of specific functionalities with a sophisticated prototype. Iterations of these phases take place throughout the development cycle. The iterations evolve from using high-level concepts early in development to detailed functional prototypes in later stages of development. User-Centered Design applied to Scopus In total 302 researchers and librarians from 21 institutions on 4 continents have participated in 14 rounds of user test sessions. About two years ago, the first contextual interviews were held. These are structured interviews onsite during which users explain and show how they work with existing systems. Observing what users do and hearing what they say about how they work are complementary pieces of information. Some parts of the work may be so obvious to users that they may forget to mention it. That is one reason why the observational component proves so effective. On the other hand, understanding the motivations behind why researchers choose to work in a particular way is crucial for interpreting observed behavior. During these first interviews, the groundwork was done for setting up a taxonomy for researchers' tasks and goals. During the early rounds of testing, mock-ups of screens were presented and in later rounds a functional prototype was used so that both concepts and specific functionalities could be tested and discussed in detail. Examples of concepts tested during the design of Scopus are: the value and use of citations during research, and using and searching across different types of information. Examples of specific functionalities tested are: the functionality of the search form, refine functionality on the results page, and functionality for searching for authors. The sessions took approximately one hour each. Initial questions focused on the researchers' professional backgrounds and use of information sources. This was done to assist in the interpretation of observed behavior during analyses. This was followed by a period during which the researchers explored Scopus and were asked to perform a few searches, in the way that they would normally prefer to do. Specific concepts and assumptions were then tested. The order in which different functionality designs were tested was varied with different researchers to avoid bias in the results. After having used and commented on the functionalities, participants were asked to rate them on a scale from 1 to 4. (Disregard, like it, use it, value it). The session ended with a general discussion. Results The results are best understood when presented in a number of general categories: Goals and behavior The feedback from the sessions pointed to five major tasks researchers have when using online resources: - Find new articles in a familiar subject field. Researchers look for published articles on which to base their own research, or to ensure that they do not start work on an area that has already seen some published results. - Find author-related information. Three kinds of information proved of interest in this case: articles by a specific author or related to the author, contact information, like the email address, and information that would help in evaluating a specific author. - Staying up-to-date: repeating the same search at regular intervals just to see what is going on, what's new since last they searched, and what might be relevant for them. - Getting an overview or understanding of a new subject field. Researchers are interested in who the important authors are, what journals and institutions play a big role and what the jargon is. - Looking for a known item. Researchers might know of the existence of a specific article, they might even have it somewhere, but find it faster by looking it up online. The observed search behavior differs per task. The preferred behavior of most researchers seems to be to start with a broad search and then to refine it. A user most commonly starts a search by typing in one to three terms. The next most common way to begin a search is to search on author names. Searching through other fields was rarely observed. To narrow down results, researchers seemed to prefer to refine their search rather than to follow links to citing or cited articles. Search form design Participants were presented with three different search forms including: a very basic one, containing only one search box; a complex one that had many features; and one in between, with a few screen elements. Although the search behavior indicated that a more simple design would be desired, there was an overwhelming preference for the most complex form. One possible explanation of the difference between behavioral and verbal feedback may be that the more complex design provided more clues about the content of the database, even though most of the offered functionality was rarely used. Refine options on the results page One of the features that was most positively received in testing is a box on the results page which gives an overview of the search results, based on various dimensions including, source title, author name, year of publication, and document type. The way researchers made use of this functionality is exactly in line with the earlier observed behavior of starting with a broad search and refining later. Use of citations Special attention was given to understanding the value and use of citations. One of the early designs that was presented to users was an interactive author citation graph, showing the references and citations to a particular author. The x-axis represented the publication date; the y-axis displayed the number of times cited. The researchers who were presented with this graph generally liked it, but were not convinced of its use. When asked how and in what context they would use this functionality, representative answers were: "I would print it out and paste it above my bed", or "Give it to my children". Observing researcher behavior, it also become clear that the majority of researchers do not require sophisticated ways to view and analyse citation data. Using and searching across different types of data Researchers were asked about the way, and how often, they searched simultaneously across various databases. Researchers feel that web searching and searching across peer-reviewed literature are "separate worlds." Searching these types of resources have different purposes. Scientists will go to the web for non-peer-reviewed information, such as author homepages, department pages, conference information, or finding material for use in teaching. They view information on the web as being of varying quality and consider it the second place to look. However, researchers specifically asked for a single entry point to navigate all the scientific information they need, including the web, and they do not want to learn different interfaces. General preferences and comments Always interesting, and in some ways most insightful, are user comments. These included: "You are going to get me addicted" "When is it going to be available? I'm definitely going to be a user!" "The combination of all these features is very interesting" "This is ... searching for dummies" "Neat! Really nice! Like this!" What is missing? "Nothing, I'm really impressed!" Conclusion Generally, a great deal of information about how researchers approach an information task was gathered. Some assumptions proved incorrect (search form preference, use of citations, use of graphs), while others were confirmed. The basic needs for a system like Scopus were confirmed to be full-text linking, coverage, and ease of use. The study also raised new questions and emphasized the need for continued focus on how users are searching and how they are approaching information tasks. We know, for instance, that users seem to appreciate the basics in a simple, easy to use interface, but also expect a range of linking and search and browse options which, even if not used, prompt them with ideas of how to refine an existing search. The librarian's experience Amy E. Knapp PhD, Assistant University Librarian, University of Pittsburgh Library feels very positive about the impact of such a partnership when developing an information system. She comments that the model developed for the creation of this information system seems to have worked very well, and has been a positive experience for the libraries involved, the users (who really appreciated being involved in the development of a tool that may one day be heavily used by them) and the developers. Hopefully, the knowledge of users' motivations and work situations gathered for this product will contribute to an increased awareness of researchers' online information retrieval behavior. This model may also have future implications for others who are working to come up with new product ideas. - "Scopus White Paper Series, Number One: Applying the User-Centered Design (UCD) process to the development of a large bibliographic navigation tool: a partnership between librarian, researcher and developer," S. P de Groot MSc., User Interface Architect, Scopus. Amsterdam, The Netherlands and A. E Knapp PhD., Assistant University Librarian, University of Pittsburgh. PA, USA, Elsevier, June 16, 2004 http://www.info.scopus.com/news/whitepapers/wp1_usability_testing.pdf