Graeme Oke, Science Subject Liaison Librarian, Victoria University of Technology, Werribee Campus

Brian Tyrrell, Campus Librarian, Victoria University of Technology, Werribee Campus

The Collection Development Policy Committee of the library considered a number of collectionevaluation methods, including conspectus, to be undertaken by subject liaison librarians.

The Cumulative Approach to collection evaluation was chosen as it uses aspects of several methods toprovide checks and balances to overcome what were seen as other method’s drawbacks. The coreof this approach is to base the evaluation on the course programs offered by Departments or Faculties at a campus.

This paper provides the method, procedure, and basis of a final report for Cumulative Approach undertaken by a Subject Liaison Librarian who trialed the method on behalf of the Collection Development Policy Committee. The input during the process and final report have contributed to the methodology having a sound foundation from which the rest of the library’s subject liaison librarians can move forward in evaluation of their collections.

To present an approach to collection evaluation that is an alternative to the more widely accepted approach of Conspectus.

To hear and learn about the pros and cons of this method.

Consider the potential use of this method to evaluate their library collections.

Being aware of the relevance of the library collection to your users.

Working with academics to make the library collection more relevant to their teaching and research requirements.

In September of 1996 the Collection Development Policy Committee ran a forum where four collection evaluation methods were overviewed. These four methods were Conspectus, Checklist,Use or User, and Cumulative Approach. The intended outcome of the forum was to choose a method for evaluating the Victoria University library collection.

Conspectus was first used by the Research Libraries Group (RLG) in the USA. It utilises library classification schemes, gives a collection level indicator, and as a result of a library undertaking conspectus gives an acquisitions commitment, a collecting goal and a preservation commitment.

The Checklist method is a collection oriented measurement rather than a user-oriented measurement. It is one of the oldest and most widely used methods of evaluation. Allows one to check the holdingslist numerically and then do a verification study on what is held compared to the checklist. Types of checklists include: - standard lists for core collections

- specialised subject bibliographies for subject collections from national bodies, eg Australian Society of Certified Practising Accountants’ library.

- lists from other Universities, and

- citation analysis eg a study of footnotes, bibliographies

- reference lists in published scholarly works.

Use studies are a well know form of collection evaluation. They can be conducted in various waysusing various tools but have a common basis in that they rely on the library’s clients, either directly or indirectly, to demonstrate the strengths and flaws within the collection. This appears to be both the greatest advantage and disadvantage of this type of study as there is an element of flexibility and clientfocus in measuring the collection’s strengths but use studies also operate on the assumption that the value of a collection directly equates with its level of use. There are several major issues associated with use studies and these include the determination of an appropriate methodology by which to measure the collection’s use and the definition of what constitutes use. Use of a collection can be a very subjective concept and one that varies across the wide range of formats that make up a library’s total collection. For instance, one might determine that use of a book equates with it being borrowed using the automated circulation system. This seems a relatively straight forward approach until serials become part of the picture. If the serials are not for loan as is the case at Victoria University, the same information gathering tools cannot be used and the questions regarding what the word "use" means in relation to serials arises. In this example "use" has different meanings for parts of the same total collection and a combination of measurement tools may be required.

Another important factor appears to be who is using the collection. Initially it would seem obvious that areas of the collection that relate to undergraduate course work would register highly or show clear flaws. The fact that there are fewer research students than undergraduates might mean that the quality of a research collection is undervalued. This largely depends on what method or combination of  methods is used. The major methods of gathering data for evaluation do not appear, in general to require introducing significant levels of infrastructure. An example of this is circulation studies whichrely on the sophistication of the library’s existing automated circulation system and its ability to put together lists of information.

Automated circulation studies are not technically difficult at the information gathering stage due to the ease of collating and producing statistics. Although planning is required to determine what theobject of this kind of study is and what type of lists and statistics will give the most valuable pictureof the collection. The major problem associated with this kind of study is the fact that they show, often where the collection is strong and not where the user failed to find what they were looking for.Also not represented are users who use the material in-house and this is a big disadvantage.

In house use studies are often used in conjunction with circulation studies to provide a better idea of the use of material generally and particularly that which does not go through the automated circulation system. In house use studies measure the use of items as they are used within the library. Often this can be applied to the serials/reference collections and there are a number of ways it can be done. Some studies have relied heavily on the good will of the users asking them to do things such as signing a sheet when they use a particular volume of a journal. Another method used has been to have staff members with portable barcode readers scan items as they are reshelved.

There are three types of document delivery measures which can facilitate some evaluation of the collection. The first one includes an analysis of types of interlibrary loan requests in order to determine gaps in the collection. This type of test could also be looked at at a campus level with intercampus loans. The second type of study is perhaps more appropriate for looking at flaws in procedures rather than the collection, as it measures the time that it takes to fulfil users requests. This again could be applied in the area of interlibrary loans or intercampus loans. The third type of test related to this area is a shelf availability test. This type of measure looks at whether items are available on the shelf when a user requires them. CAUL has developed a survey along this line which requires both the cooperation of clients and staff resources.

And finally the Cumulative Approach to Collection Evaluation of which this paper is about and the method will be expanded on further.

The Cumulative Approach to collection evaluation was first mentioned in an article entitled, Meeting Modern Demands of Collection Evaluation: A new approach, by Sushella N. Rao (1994), References Services Librarian, at the University of Wisconsin. When looking for a collection evaluation method in the library literature the author decided that all single methods had drawbacks. Her approach was to devise a "Cumulative Method" which combined several methods to provide checks and balances to overcome each method’s drawbacks. The core of this new approach was to base the evaluation on the course programs offered by Departments or Faculties at your campus.

The first step in the six step procedure is to find out what major course programs are offered on your campus by using a combination of your own experience, the Faculty handbook and academic advice.

Step two is to find for each course program a similar course at another institution, again using a combination of your own experience, handbooks and academic advice. Step three is to assign Library of Congress Subject Headings (LCSH) to the topics covered in the course. Using the library catalogue and the similar institution’s library catalogue, numbers of items in each subject area are compiled.

The results show how your library’s collection rates with the other institution on the quantity of items. Step four is to compare your library’s collection holdings to standard core lists of titles in course program subject areas. This gives some idea of the quality, accuracy and completeness of your collection. Step five is to check collection growth using the library catalogue to compile the number of titles in the chosen subject area added in the last ten years and the number of titles added in the last five years. This indicates whether in the last five years growth has increased.

This procedure is then repeated using the catalogue of the similar institution. A comparison of growth rates should allow you to gauge your success. The final step is to use Books In Print and Periodicals in Print to find the total number of items available in each of the chosen subject areas; and then to find the number of these items in the library collection. The percentage of matches is called the "sample rate" and is an indication of the currency of the collection.

I was asked on behalf of the Collection Development Committee to use a modified version of this six step task as a trial on our "fire" collection, and two further steps were added. Step seven to use statistics on circulation, and step eight to assign an RLG Conspectus Collection Depth Indicator to describe an overall assessment of the library’s collections in the course program subject area. Thisfinal step is done in line with the Australian Council of Libraries and Information Services (ACLIS) collection development policy guidelines which recommended that whether or not the library adopts conspectus, a conspectus like approach be adopted for the description of evaluated collections.

While undertaking this trial I was to report and keep close liaison with Brian Tyrrell the head of the Collection Development Policy Committee to discuss any problems or queries encountered. Another Subject Liaison Librarian undertook the trial on the subject area of acupuncture.

Initial consultation with the Centre for Environmental Safety and Risk Engineering (CESARE) who primarily do research into fire, provided ten international research institutions and Universities whichwere identified as offering similar research interests to the Centre.

Appropriate subject headings were identified using LCSH and the library catalogue. A keyword search was done using the term fire* (~650 titles), limited to Werribee (~230 titles). The full entries of this search result were viewed. Recurring subject headings were identified, and subsequent library catalogue Subject searches were carried out on both catalogues.

An initial problem was identified as to how far extensions to subject headings should be included, eg was a generic Fire prevention sufficient as a subject heading, or should there be separate headings such as Fire prevention, Fire prevention - Australia, Fire prevention - Equipment & supplies, etc. It was decided that where an appropriately identified subject heading, identified a specific research area of the centre, the extensions should remain. On this basis twenty Library of Congress Subject Headings were identified.

The next step was to identify one out of the ten international research institutions and/or Universities offering similar research interests to the Centre as a measure of how the collection rates compared tothe quantity of items. With internet connections it was reasonably easy to log onto appropriate library catalogues, and start an evaluation. Progress was being made with one University library when another problem was identified.

It was found that some library catalogues, although searching by "subject", were in fact searching by keyword. Thus specific subject headings were not being identified as a direct comparison. Tofind out if specific subject headings had been used one had to go into each individual bibliographic record which was considered a major and laborious undertaking. Thus further explorationand investigation of library catalogues were undertaken. The majority of international library catalogues were found to not allow Library of Congress Subject Heading specific searching. After a fair amount of investigation one library’s catalogue was found to be comparable, the University of Edinburgh. The statistics of the student and staff numbers from The World of Learning 1997, were also shown to give a relatively close comparison. Victoria University of Technology 16,000 students, 1,200 teachers, The University of Edinburgh 17,197 students, 2,398 teaching staff.

A factor observed with the direct comparison process was that we were comparing a campus of Victoria University to, in some cases, a whole University. It was discussed whether this should be taken into consideration when evaluating results. In the direct comparison it appeared in the library catalogue that at the University of Edinburgh there was an Engineering library (Robertson) and a main library, as well as branch libraries. The library catalogue did not allow limitation to a specific branch library. To make a comparison of branch libraries would have involved a laborious individual bibliographic look at each record. It was decided to not worry about it, as we did not know the full history or information about the University library set up etc., so the amount from the University of Edinburgh’s library catalogue for an individual subject heading, was considered a direct comparison to our campus’s result.

Where an item had been assigned two or more of the identified subject headings should the itembe considered one, or two in the comparison analysis of subject areas? It was decided to consider it two. Thus the 113 total for Victoria University of Technology Werribee campus equated to about 81 items. It was concluded that we were compiling a comparison of subject areas, not items. If we were looking at items rather than subject areas another problem may be that there were twenty of one title, one of another, etc. In our comparison analysis we were just looking at subject headings, not actual numbers of items.

Standard core lists now at the site http://www.nla.gov.au/niac/conspectus/bibliog.html were inspected.

There were no standard core list for this subject area, and so this step was left out, as there were no other known core lists available.

Collection growth using the Victoria University library catalogue to identify titles in the LCSH areasadded in the last 10 years could not currently be undertaken. Victoria University of Technology was established six years ago with the amalgamation of the Footscray Institute of Technology and the Western Institute. Thus retrospectivity was only available for a maximum of six years. Anotherproblem identified was that the library catalogue holdings of the two institutions were uploaded onto the then new Innopac (Innovative III) library catalogue system around the 12-15/2/1992. A collection growth analysis of subject areas over six years was used, from 1992/1993 to 1996/97, excluding the loading dates to the library catalogue. Given the amount of individual items (113) it was considered easier and quicker to go into the individual bibliographic records to get this information rather than to create lists using the management system available within Innopac. There was a total number of holdings of 113, the total number of items added in the last year was 22, and the total number of items added in the last six years (after 16-4-1992, and incorporating holdings added in last year after the 16-4-1996) was 91.

An appropriate bibliographic database of materials in print to find the total number of materials in each of the LCSH areas was thought to be Global Books in Print (GBIP), available on our CD-Rom Network. However it was soon apparent that GBIP did not appear to use Library of Congress Subject H eadings. For some items GBIP used much more general subject headings. GBIP did not seem to have a through coverage of conference proceedings, local Australian in print items, videos, or periodicals. It became apparent that there were variances in the GBIP assigned subject headings versus the Victoria University library catalogue subject headings assignment for certain items. This was resolved by using the general GBIP subject headings.

The method of random sampling used, was that if the sample list was less than one hundred entries the whole list was checked against the Victoria University library catalogue. Where there were more than one hundred titles every tenth title was selected until a random sample of 100 was achieved, and these titles were checked against the Victoria University library catalogue. Where the list was less than one thousand, the list was worked through again as required to make up a sample of 100. The results of the sample rate percentages were not high.

Innopac statistics on circulation were found whilst observing the full records in carrying out the task of collection growth. This undertaking was feasible for the small sample being tested (113). Circulation activity by title was the prime measure of use. The Innopac computer catalogue system allows the creation of lists of these and other circulation statistics by relatively simple methods.

The CESARE library collection at Werribee campus would be defined by the RLG Conspectus Collection Depth Indicator Definitions (1997) as a Level 2 - Basic Information Level. There is only a limited collection of monographs, reference works, and limited collection of representative periodicals.

Ideally the collection should aim towards a Level 4- Research Level. Given the subject specificity of the research centre and the fact that there is doctoral study and independent research being carried out by the Centre there must be components of the library collection which could be considered at Level  4 - Research Level. However due to limited funding a Level 4 will not be achieved for a number of years, but is something to strive towards.

Since undertaking the trial the Collection Development Policy Committee have refined the process to an eight step activity to be undertaken by all Subject Liaison Librarians. Currently there are thirteensubject areas being evaluated by different subject liaison librarians over six campuses of Victoria University. The completed evaluations can be seen at the Collection Evaluation Project Victoria University Library Home Page (http://www.vut.edu.au/library/collections/evaluation/cep.html). As subject areas are competed their reports will be added on this home page. The challenge for the convenor of the Collection Development Policy Committee is to ensure the project continues in times of decreasing staff levels and increasing work loads. An annual objective for the Library Client Services Section will be for every Subject Liaison Librarian to submit an evaluation report on the resources available in the library for a course program. This is being encouraged by regular reminder emails and requests for progress reports from each subject liaison librarian. A further challenge facing the University is the recent amalgamation between the University and the former Western Melbourne Institute of TAFE, doubling our campus libraries. As the library works towards integrating the two collections, and   a common library catalogue and automated circulation system, the Cumulative Approach to collection evaluation will hopefully continue to fulfil the collection assessment and evaluation needs of the library and have a positive impact on collection development.

Finally I’d like to acknowledge the Collection Development Policy Committee members as it was their work that explored the various collection evaluation methods which were expanded upon at thestart of this paper. They were also the ones who presented their findings at the forum at our library in 1996. Gehan Aboud, Mark Armstong-Roper, Jenny Comley, and Jane Miller.

Clayton, Peter. Sept. 1990 "Conspectus reconsidered" Australian Academic and Research Libraries,

vol. 21, no. 3 p 179-186.

Collection Evaluation Project Victoria University Library Home Page Available: http://www.vut.edu.au/~library/collections/evaluation/cep.html Accessed 8th Sept., 1998.

Ferguson, Anthony W. 1997, Revised RLG Conspectus collection depth indicator definitions, Email,

Received 12 March 1997.

Gorman, G.E. and Kennedy, J. 1992, Collection development in Australian libraries, Centre for

Information studies, Charles Sturt University, Wagga Wagga, NSW.

National Library of Australia, Australian Conspectus: Australian conspectus information sheet

online guide to collection assessment and evaluation tools. Available: http://www.nla.gov.au/niac/conspectus/bibliog.html Accessed 18 Aug, 1998.

O’Neil, Frances, 1992 Conspectus project report Victoria University of Technology Library.

Roa, Sushella N. 1994 "Meeting modern demands of collection evaluation: a new approach"

Collection Building Vol. 13, no. 1, p 33-36

The world of learning 1997, Allen and Unwin, London.