Reference Evaluation of the Public Library
Abstract
In summer 1997, a project concerning reference service evaluation of the Karlsruhe public library was led through by students of the Hochschule für Bibliotheks- und Informationswesen Stuttgart. The aim was to assess the quality of the reference service not so much in direction to the formal correctness of answers but in direction to the communication and interviewing skills of the staff, from which the overall satisfaction of the user and their willingness to return depend. The results show that only the half of the test users were satisfied with the given answer - a result that correspond with those of other projects of this kind - and that only three quarters would come back if they had other information needs. These outcomes should make think about the need for special trainings for the staff, which strengthen their skills in communication and interviewing.
Reference Evaluation of the Public Library
1 Methodical Annotations
2 Overall orientation
3 Overall Communication
3.1 Statistical evaluation
3.2 Evaluation of the written
reports
4 Interviewing competence
4.1 Statistical evaluation
4.2 The written reports
5 Quality of the reference service
5.1 Statistical evaluation
5.2 The written reports
6 Willingness to return
7 Willingness to return in re to other criteria's
8 Conclusion
Report on a project at the Hochschule für Bibliotheks- und Informationswesen Stuttgart in summer 1997
For evaluating the reference service of a library, various approaches are suggested in specialized literature. They can be classified as methods pursuing either quantitative or qualitative aims. Quantitative methods usually count how many questions of what kind were asked and how many of these were answered correctly. From these results follows a conclusion about any possible problems concerning the reference work. Even though this kind of methods brings forth objectively verifiable results, there is a weak point: the results do not cover if a particular answer was useful for a particular user. An answer can be formally correct but still useless, maybe because the recommended literature is not of the level the user needs (example a scientific monograph for a pupil preparing a paper or an introducing manual for a specialist) or because it is delivered too late. Moreover, if the user gets a useless answer but does not come back a second time to ask again and goes away without an answer after all, an actually wrong information seems to be correct in the eyes of the librarian and gives himself a wrong impression of his own work.
These problems became obvious when the Karlsruhe public library realised such a quantitative counting of reference questions: the result was an amount of certain figures covering the categories of questions but still no approach to a direction to act in the future.
Considering these methodical problems, a qualitative method was chosen for the evaluation project that was realised in summer 1997 on the library's request. It is a method described in American specialized literature which proved its worth in practical projects. 1
The main aim was to evaluate different aspects of the reference service. The point was not the formal correctness of the answer but the usefulness for the asking person and the satisfaction with the answer. In this method the patron's „willingness to return" is regarded as the main indicator for reference service quality, because this element ultimately is the decisive factor by which user satisfaction can be measured. Generally speaking: the greater the willingness to return, the more the whole answering process was satisfying for the user, whether the answer was formally correct or not. From this follows a consideration rather of the communication elements of the reference process than of the formal correctness.
The evaluation contains three parts: (1.) an assessment of the criteria friendliness, communication skills, satisfaction with the answer, usefulness of the answer and willingness to return on a scale ranging from -3 to +3; (2.) a detailed written report concerning the reference situation including the way to the reference desk; (3.) a summary of the results listing helpful and cumbersome points.
The evaluation was carried out undercover, that is by persons unknown to the library's staff, though the staff was generally informed in the run-up. The evaluation was carried out by ten students of the Hochschule für Bibliotheks- und Informationswesen Stuttgart because of their special qualifications: an exceptionally critical attitude and an idea of a qualitatively good reference work.
The questions used in the interviews were chosen from an amount of some hundred questions that had been asked in daily reference work in different libraries. In the run-up to the project they were assessed by experienced reference librarians in accordance with their degrees of difficulty so as to make the questions comparable.
First, a successful reference work requires that the user gets a possibility to orient himself in the library, to find the person who is supposed to answer the user's question - and to find the library at all. Besides a suitable orientation system the overall atmosphere of the library is important.
Considering these aspects, it is interesting to see that most of the participants in the project had difficulties to find the libraries: suitable signs could not be found immediately, for there were no signs at all or they were located in places hard to see. That implies many additional difficulties for users who come to the library for the first time and who do not know the location exactly. The same insecurity appears sometimes inside the buildings, if the libraries exceed a certain size: orientation maps or signs were placed in corners where they could not be found directly.
The last „hurdle" to be token is to find a member who can be asked. In this respect the greatest obstacle consisted in the fact that sometimes it was not clear wether a certain person was a librarian or not because the persons did not wear a plate with their name or with a logo of the library. Furthermore, in most cases nameplates were missed on the reference desks: though such plates had been there, they were not placed on the desk.
An important element of the reference situation is the communicative „framework" in which the interview takes place: For the user, it is not only important whether he gets a correct or useful answer _ to put it briefly: whether he gets an answer at all - but also how he gets this answer.
The elements an ideal communicative behaviour consists of are first a certain attention already in the run-up: the user should be noticed before he stands in front of the librarian. Secondly, the librarian should not wait for the user to ask, so actively offered help is required. However, this active help should not turn into obtrusiveness. Thirdly, the opening of the conversation should be made by the librarian. Verbal and non-verbal behaviour should make clear that the user can get a fast and competent answer at this place and that he is welcome.
One should not only consider these elements atmospheric viewpoint: as explained below, there is a significant relation between „friendliness" - or communication skills - and the willingness to return. That makes sense immediately: if a librarian spreads an unpleasant atmosphere, he or she hardly would be asked for an information a second or a third time.
The diagram „friendliness" shows how the testing persons assessed the competence in communication of the librarians. Here and in all following statistical evaluations the scale-points from -3 to 0 were summarised as negative, the scale-points from 1 to 3 as positive. 55 forms were evaluated. (See diagram 1)
In the entier system, three quarters of the interviewed librarians were seen as friendly (77%), 23% of the interviews were judged as negative. The results of the main library show a somewhat worse judgement on an average: 26% of the interviews were judged as negative and 74% as positive. In the branches and in the youth library, on the other hand a much better judgement than the average could be obtained: 19% of the interviews were assessed as negative and 81% as positive.
3.2 Evaluation of the written reports
In the main library, there is a focal point on a certain welcoming behaviour was paid attention to. Though some users were welcomed actively, much more often negative experiences were reported. Often the interviewing person was not recognised until he or she was standing in front of the reference desk and was speaking to the librarian. Sometimes the user was not welcomed at all:
„When I come up to the desk, the librarian looks up at me as if I was getting on her nerves, as if she wanted to say `Not again!'. No welcome, no expecting look at me, but a look that says `What's the matter now?!'."
The openings of the conversation show that only in half of all cases the librarian opens the interview. That seems to be far to little.
As to the end of the interviews, it is obvious that mostly there is no clear conclusion nor an explicit good bye. Either the librarian goes away without any good bye at all or it is obvious that he or she considered the interview as ended by giving the reguested information. From the lack of an explicit good bye follows a feeling of insecurity on the side of the user, as the comments in the reports show. This unsecureness influences the total assessment throughout the interviewing process in a negative manner and could be regarded as impoliteness, in the worst case.
The communicative behaviour in the main library during the interview was mostly judged as positive, as „friendly", „open", „ concentrated" or „relaxed". Unfortunately, here also some negative manners were mentioned. Particularly the spontaneous and unreflected denial of any reference interview was considered as very negative:
„An initial uncommitted smile on her face tuned into an expression of indignation signalling that I may ask anything but this. / `No, we haven't got anything on this subject. Nothing." The librarian looks at me as if she expected me to vanish in air immediately."
This manner was considered negative, particularly in cases as the one reported here. A later search in the online catalogue definitely showed media fitting the item asked for. Furthermore, unfriendly looks, insecurity of the librarian and the tendency to get rid of the user as fast as possible were negative elements.
As follows from the reports, the communicative behaviour in the branches and in the youth library was much more positive. The staff members showed „by looking up shortly that they wee ready to help any time". Interest in the asking person and in the question was also mentioned.
4.0.1 Methodical annotations
Interviewing competence includes various points. Particularly the following are important: Summarising the question or saying it in other words allows the librarian to check if he or she understood the question correctly. The same aim is reached by checkbacks or specifying the desired information. During the reference interview, inclusion of the asking person in the searching process is necessary in order to allow the user to take influence on the results. Finally, with follow up questions, the librarian should check at the end of the searching process if the user has found what he or she has looked for. The user should be asked to come back if further questions arise.
Diagram 2 shows how the testing persons assess the interviewing competence of the librarians. In the entire library, 23% of all reference processes were judged negative and 77% as positive, i.e. only three quarters of all testing persons had the impression that their questions were understood correctly. In the main library, a clearly worse assessment than the average was
reached: only 65% were judged negative and 35% as positive. Like here, in the branches and the youth library, a very good judgement of the interviewing competence compares to the average: 5% of the interviews were assessed as negative and 95% as positive.
Though there were beginnings of reference interviews and check backs, the testing persons often reported from the main library that their questions were not heard to the end or it seemed that the librarians did not listen at all. Sometimes statements like the following occur: „without a further comment on my question, she began searching in the online catalogue". It is clear that the lack of deeper interviews sometimes prolonged the search, because it was not clear enough what kind of information was desired.
„I asked her if there was anything about nature reserves in Australia. The librarian answered: `No, there's only something about nature reserves in Africa.' She said it in such a convinced way that I do not get the idea to follow up again and to ask her to search again in the catalogue."
Some participants in the project report that they were included in the searching process, for example when the librarian turned the screen of the terminal in their direction so that they could take part in the search. But mostly the asking persons could neither see what happened nor the librarian explained what he or she did sometimes, even though they expressed their desire to participate in the search.
„I cannot see what she is searching for, then she says: `All ended already.'" - „Wordless she began to search in the catalogue." - „I sit and look at her, for I cannot see the screen, and nod to the pieces of information she throws at me time after time."
„I began: `How do I have to search, if I want to find something', I point at the user's online catalogue, `about dentists and hypnosis - well, If I want to get both together, dentists working with hypnosis?' She smiled, nodded and said: `I will look for you.' Then she began to search in the catalogue. I actually hoped that she would show me how to search. But it seemed that she would not move away behind her desk, so I sat on the chair in front of it."
Furthermore, mostly the librarians did not comment on their activities that follow the search in the catalogue; the users missed being asked explicitly by the librarians to come with them, though in most cases this became clear through non-verbal behaviour or through the context.
Although follow up questions and requests to come back are considered as very important elements in the interview, they were used very seldom and obviously more at random than systematically. Follow up questions were mentioned only three times in fifty-five reports, a request to come back even only one time.
For all mentioned elements, the reports on the branches and the youth library show much more positive results. Here, check backs were made, or the librarians usually asked the user if something was not totally clear. In most cases the users could participate in the searching process. But here also explicit conclusions of the interviews or follow up questions were not made in most cases.
5 Quality of the reference service
In the evaluation of quality, first of all professional competence plays an important role. This includes the knowledge of reference media available in the library in all existing publication forms, deeper knowledge of the library's stock and knowledge and use of various retrieval strategies that are suitable for getting the desired information. Secondly, it must be asked, how useful or satisfying the answer and the media are that could be lent. Furthermore it is important to know if the desired media are available in the library or if visits to other libraries are necessary. If references to other places are made, at which the information could be obtained, it is necessary here that the person who referred to another place surely knows that the desired information can be located there, otherwise the users have to go unnecessary ways without success.
In diagram 3, the participants in the project assessed the quality of the reference service by answering the questions „how useful was the answer?" and „how satisfying was the information?".
For the whole system, 51% of the reference results were assessed as negative, 49% were assessed as positive. That means, only half of all participants had the impression that the answer they got was correct, suitable for their desires and available. This figure is equivalent to the results that can be found in relevant literature. This result should set librarians thinking and motivate them to become aware of necessary training in this area.
Again, the main library shows a somewhat worse result than the branches and the youth library. In the former, 56% of the reference processes were judged as negative and 44% as positive. In the latter, 43% of the reference results were assessed as negative and 57% as positive.
The diagram „satisfaction with the answer" shows the following results (see diagram 4):
On an average, 45% of all reference processes were assessed as negative, 55% as positive. This is equivalent to the foregoing result: only half of all persons were satisfied with the answer they got.
In the main library, 53% were judged as negative and 47% as positive, in the branches and the youth library only 33% of the participants were of the opinion that the delivered answer was unsatisfying, 67% had a positive opinion.
In the main library, searches were carried out, with the help of the online catalogue, in most cases followed by further searches in the stocks together with the user. Some negative experiences were made here too: In some cases, the librarian began to search in the stocks immediately and took the - sometimes unsuccessful - result for final, though a search in the OPAC would
have brought some more results. Vice versa, in other cases the results of the OPAC research - if they had a result or not - ended the reference process; the testing persons missed an extentions of the search to other reference media. Sometimes inexact or unsuitable retrieval strategies were used for the OPAC.
References to other libraries were made too often in the opinion of the participants. On the other hand, often some further help as offered, e.g. the possibility to reserve lent media or accelerated processing of media that were not yet to be lent. However, the willingness to offer these special services differed extremely from person to person.
The impressions concerning professional competence in the branches were much more positive. Often the good knowledge of the own stocks was mentioned in the reports. If an answer was not possible with the help of the stocks of the branches, telephone calls to the main library were made to get the desired information on different ways. In all branches there was the willingness to get the information from beyond the limits of the own stocks.
As mentioned above, the willingness to come back and ask the same librarian a second time for an information is one of the central points in the evaluation of the whole reference service, not least because it is the main concern of the library that the patrons come back again.
The statistical evaluation (Diagram 5) shows that on an average 75% of the testing persons would come back to the same librarian a second time. 25% would not come back. Considering the fact that the reference service in that special library is not frequented very strong, this result seems to be too worse.
Comparing the main library to the branches shows a further time that the branches reach a better assessment than the main library: In the branches and the youth library, 86% of the participating persons had a positive opinion on the willingness to return, 14% of the answers were negative. In the main library, 68% of all would come back, 32% would not.
7 Willingness to return in re to other criteria's
The decisive point about all the foregoing analyses for drawing conclusions for daily work is, why users are willing to come back to the library's reference service staff. Indications on this point can be obtained by relating the willingness to return to all other evaluated criterias.
Diagram 6 shows the correlation between the criteria. The strongest correlation can be found for the criteria „friendliness" (that is communication competence) and „understanding of the question" (that is interviewing competence). Most persons who were willing to return had a positive opinion of the named criteria: among a total of 41 persons, „friendliness" was assessed as positive by 39, „understanding of the question" by 37, but „usefulness" only by 24
and „satisfaction with the answer" only by 28.
Differentiated by main library and branches, similar results can be recognised:
In the main library, among a total of 23 persons, „friendliness" was assessed as positive by 23, „understanding of the question" by 20, „usefulness" by 13 and „satisfaction with the answer" by 15 persons.
In the branches, among a total of 18 persons, „friendliness" was assessed as positive by 16, „understanding of the question" by 17, „usefulness" by 11 and „satisfaction with the answer" by 13 persons.
These results have their equivalent in those of the international specialised literature: A positive assessment of the librarian's friendliness and of the impression as to whether the questions have been understood or not have a stronger influence on the willingness to return than the usefulness of the delivered answer or the satisfaction with the information.
The project brought forth two interesting outcomes. First, the results of satisfaction with the received answer are similar to those of other investigations concerning the quality of reference service: only half of all users had the impression that the information they got was useful for them. It seems to be worth asking if these results can be the final status quo. Despite the positive results for friendliness and interviewing competence, the effectiveness of work cannot be assessed as very high, if the aim of reference service is to give as many users as possible information and references as useful as possible. The project confirmed the necessity of suitable staff training concerning retrieval strategies, behaviour in communication and interview skills. Libraries that implemented those training could raise positive assessments to 60-90% of all asked users.
Secondly, a look at the library branches showed that the users had a much more positive overall opinion of the performance of the work, though - maybe because - these libraries were much smaller, had less opportunities concerning their stocks and not as many staff as the main library. So it seems that these disadvantages can be - and have been -compensated by increased efforts for the user such as showing interest in their needs and using all direct and indirect sources of information to meet these needs. How much more intensive effects this attitude would have if it found its way into the reference work of larger libraries. This fact also shows the necessity of specific training.
1See Dewdney, Patricia ; Ross, Catherine S.: Flying A Light Aircraft. Reference Service Evaluation from a User's Viewpoint, in: RQ 34/2 (1994), S. 217-230.
Druckerversion
