Library Vendor Assessment Literature Review

by Susan C. Vargas

Over the past decade, libraries have formed relationships with vendors of new kinds of services and of markedly changed products. One source of new relationships is the outsourcing of services formerly provided within a library’s staff. Some libraries now contract with vendors to provide professional librarian functions such as cataloging, selection, and reference services. In addition, these new relationships, along with libraries’ relationships with traditional vendors, are increasing in complexity. Thus, contracting with vendors of new services and with vendors of information in newer formats requires new management skills on the part of librarians. Part of the broad issue of managing these vendor relations is the narrower issue of vendor assessment. This review will focus first on the topic of vendor assessment in the general scholarly management literature and then on the coverage of this topic in the scholarly managerial literature within library science.

General Management Literature

The strategic importance of vendor selection and subsequent assessment is well established in the literature of business purchasing, beginning with the seminal work on vendor selection criteria published in 1966 (Weber, 1996). Significant research continues to be done on this topic in part because of its particular relevance to manufacturers in a just-in-time environment where vendor price, quality, and delivery are key performance criteria. There are even numerous trade journals and scholarly journals devoted to vendor relations and performance evaluation. Among them are Supply Management, Materials Management and Distribution, Journal of Supply Chain Management, and Summit (a Canadian magazine about public sector purchasing).

Weber describes the three common and flawed current approaches to vendor evaluation (1996). First is the “categorical or key-factor rating method” in which potential vendors are assigned subjective, largely intuitive, ratings based on the evaluator’s judgment and experience. A second current method is the complex “cost-ratio method” which requires a comprehensive, precise cost-accounting system to determine the buyer’s internal operating costs associated with the vendor’s quality, delivery, and service. The third common approach, the “linear average or weighted-point method,” modifies the first method by providing numerical weights to the subjective evaluation criteria.

Many other approaches have been described and proposed in the literature. Researchers have suggested using more objective quantitative approaches, including mathematical programming models, statistical approaches, and analytical hierarchical processing. Weber, for instance, demonstrated the use of a mathematical programming model employing data envelopment analysis to measure vendor performance on multiple criteria and to identify comparative benchmark values (1996, p.28; see also Talluri, Narasimhan, & Nair, 2006, p.212). Dogan and Sahin employed mathematical models to select vendors using activity-based costing and fuzzy present-worth techniques (2003, p.420). Babu and Sharma gave an example of analytical hierarchy processing (2005, p.101). Still other researchers proposed methods which combine both objective and subjective data. An example is Li, Fun, & Hung, who used two-dimensional analysis to propose a performance measure based on both quantitative and qualitative criteria (1997, p.753). Going the opposite direction are other research groups which proposed vendor performance measurements based on an evolutionary fuzzy system for evaluating attributes described linguistically (Ohdar & Ray, 2004, p.723; Jain, Tiwari, & Chan, 2004, p.735).

Given the vast array of approaches to vendor evaluation and the fact that no single evaluation method is completely satisfactory, there is now research relating to strategies for choosing an evaluation technique. Purdy and Safayeni discuss the advantages and limitations of a variety of methods for evaluating potential and current suppliers, and they classify the methods by whether the focus is on information from product- or process-based domains and whether the information acquisition mode is direct or indirect (2000, p.435). They posit that buyers normally must use indirect methods for evaluation of potential vendors and that buyers should use methods in all four categories once a vendor relationship has been established (p.441).

Vendor evaluation strategies exist within the context of an organization and its processes. Choosing an assessment approach is just one of the seven steps of a process of developing and deploying vendor assessment outlined by Gordon (2005, p.20). Although this article is found in a trade press, it includes a useful bibliography of books about supplier evaluation and managing relationships with suppliers. Another aspect to consider is that many of the assessment techniques require technological tools such as software packages. Examples of descriptions of such software implementations were provided by Choy, Lee, and Lo (2004, p.191), by Humphreys, Huang, and Cadden (2005, p.147), and by Lau et al. (2005, p.61).

Library Management Literature

Libraries, like other organizations described in the general managerial literature, must deal with vendors and hence must assess both potential and ongoing vendor relations. While the scholarly library literature contains few descriptions of general vendor assessment approaches, it does contain many descriptions of specific approaches to evaluating particular types of library vendors. First the general and then the specific vendor type approaches will be described below.

General Strategies for Assessing Library Vendors

There are two often-cited classic articles in the library literature of vendor assessment. Alessi listed relevant criteria for selecting new library vendors and evaluating the performance of on-going library vendors (1992, p.117). Barker’s account of a comprehensive library vendor evaluation project at the University of California, Berkeley provided methodological insights, both in establishing the goals of the evaluation and in carrying it out (1986, p.265). Barker’s article also provided helpful examples of vendor selections, coding, and adaptation of automated systems.

Lam described the practical details of optimizing spending and maximizing efficiency by librarians in their day-to-day dealings with vendors (2004, p.146). Lam explained how to set up a vendor assessment system using common spreadsheet software such as Excel and how to decide what items to monitor – commonly turnaround time, missing titles, discount rates, invoice accuracy, and quality. Lam’s advice is most suitable for assessing monograph vendors but could easily be applied to assessment of other vendors.

Walther gives some very general guidelines for evaluating all types of library vendors as well as offering insights into the decision of how many vendors a library will choose to contract with (1998, p.149). The thrust of this article, however, is in explaining library-vendor communications, a factor that cannot be discounted in an environment of complexity in products and services.

The journal Library Acquisitions: Practice & Theory offered an issue on supplier performance and evaluation in 1994. Among the articles in that issue are some relating to general vendor evaluation. Black identifies the reasons why acquisitions librarians undertake vendor evaluation, and describes the organizational benefits of these assessments (1994, p.57). Crotty describes how the forces of change in library acquisitions impact vendor evaluations and offers practical suggestions for incorporating evaluation into a library’s workflow (1994, p.51). Dolby relates the assessment of vendor quality to the basic tasks of acquisition (1994a, p.45). Practical tips are provided for evaluating suppliers in a manual system (Roberts, 1994, p.71) and with automated systems (Kent, 1994, p.79). Dolby authored a second article which discusses interpretation of raw results and the advantages of preparing a formal evaluation report (1994b, p.89)

Approval Plan Vendor Assessment

Multiple strategies for evaluation of approval plan vendors can be found in the library literature. Brown has published articles about how academic librarians evaluate approval plan vendors in an environment where technological development is pervasive and approval plans offer a proliferation of new products and features. Brown, along with Forsyth, offer a very detailed article which reports the results of a survey which requested ratings of 79 criteria in approval plan vendor selection (1999, p.231). The criteria divide into 6 categories, ranked here from most important to least: corporate reputation and business practices; approval plan management expertise; acquisitions services; outsourcing of physical processing; outsourcing of cataloging; and electronic financial transactions. The article describes which criteria are most important to which kinds of academic institutions and provides data-driven hints for selecting an approval plan vendor (p.259-262).

In another article, Brown offers additional insights from the same survey of academic librarians who evaluate approval plan vendors (1998, p.341). Brown found differences in practices based on library budget size and based on the extent to which libraries use new vendor-offered services such as electronic financial transactions and outsourcing of physical processing and cataloging. Brown predicts that evaluation will change as these newer services become more sought by libraries.

More has been written on approval plan vendor selection and assessment. Carpenter described evaluation of approval plan vendors in an approach much simpler than that of Brown and Forsyth (1998, p.329). Carpenter simply relates the most important issues that Oberlin College Library found in evaluating its approval plan with a particular vendor, Blackwell. Carpenter also identified a number of other resources for identifying evaluation criteria, including a bibliography and an ALA guide. Approval plan vendor selection was also discussed as part of a presentation by Schatz and Baldwin in 1998.

Monograph Publisher Assessment

In 2000, Lewis assessed the quality of publishers of political science monographs by surveying librarian members of the Association of College and Research Libraries Law & Political Science section (p.313). This article could aid collection development librarians in the field of political science but is not otherwise useful for vendor assessment as it does not describe the criteria used by librarians in their evaluations.

Monograph Vendor Assessment

A standard guide cited in the library literature is the American Library Association’s Guide to Performance Evaluation of Library Materials Vendors (1988). The title is misleading because the guide’s scope is evaluation of in-print monograph vendors. The guide provides evaluation criteria and various approaches to statistical analysis, and is a good place to start when planning any evaluation process.

One of the few assessments of individual vendors within the scholarly library literature is Orkiszewski’s study of Amazon.com as a library book and media vendor (2005, p.204). Orkiszewski compared Amazon.com to the standard library vendors used by Appalachian State University on the issues of availability, pricing, predictability of discounting, fulfillment, and speed. The study shows that Amazon.com’s database is quite comprehensive, its system for ordering out-of-print copies is convenient, and it fulfills orders faster than typical library vendors. Amazon.com’s prices would have been 7.7% higher than those of typical library vendors had books and media of all types been ordered. However, Amazon.com offered certain types of materials at greater discounts than typical vendors, including books about art, computer science, home economics, and theater; trade books, especially recent best sellers; and DVD’s more than other formats.

Serials Vendor Assessment

A standard guide to assessment of serials vendors is the American Library Association’s Guide to Performance Evaluation of Serials Vendors (1997). This guide “describes the factors to be considered before beginning a review and explains criteria that may be utilized” (p.1). The guide, as well as its annotated bibliography, provides an excellent starting point for any library vendor assessment project. The guide notes that serials vendor evaluations have typically been conducted in crisis situations and recommends that serials vendor evaluation instead “be incorporated into the regular work flow and that clear objectives for vendor performance be established and communicated to vendors” (p.1).

Additional strategic choices in serials vendor selection and evaluation are discussed by Bonk (1985, p.51), Derthick (1986, p.1), McDonough (1991, p.221), and Kent (1994b, p.83). Bonk’s article is much cited and discusses both the analysis necessary to devise a serials vendor evaluation and the criteria to consider. Derthick describes the criteria upon which members of the Association of Research Libraries base their vendor selection. McDonough offers a weighted criteria plan for serials vendor performance studies and reviews objective and subjective criteria to consider in the process. Kent focuses the assessment process on just two factors: price and service.

Electronic Journal Vendor Assessment

There is considerable variety among electronic journal providers. The smallest vendors offer individual electronic journals, next are commercial publishers which offer their own journals, and biggest of all are distributors of electronic journals from multiple publishers. Pavelsek provides a checklist for evaluating providers of multiple electronic journals which will allow a clear picture of the products, features, and services offered (1998, p.39). Pavelsek applies this evaluation technique to two vendors: JSTOR, an archiving service that provides only back files, and Project Muse, a provider of current electronic journals.

Miscellaneous Vendor Assessments

The library trade journals offer additional advice about contracting with vendors of other kinds of services. Libraries which are considering contracting with a vendor to provide digital reference services should look at Hirko’s article to learn about criteria for assessing these services (2002, p.16). Evaluation criteria for document delivery suppliers are discussed by Jackson (2004, p.242) and Walters (1994, p.14). Bridge briefly describes the use of site visits and demonstrations as techniques for selecting a library automation vendor (1993, p.56). A current and much more thorough, book-length treatment of evaluation of library networked services is provided by Bertot (2004).

Summary

Many of the articles in the library literature of vendor assessment provide general ideas and areas to consider in the evaluation process. Pavelsek (1998, p.40), however, argues that vendors should be evaluated using prescribed guidelines so that decisions can be better informed. Such precise guidelines are not often found, with even fewer guidelines able to accommodate new criteria necessary to evaluate rapidly changing vendor programs and services. Librarians should begin a vendor assessment project by reviewing the ALA’s Guide to Performance Evaluation of Library Materials Vendors, which mainly addresses monograph vendor assessment, and the ALA’s Guide to Performance Evaluation of Serials Vendors. Practitioners seeking guidance will find little direct assistance from general managerial literature but will find helpful articles in the library literature for both practical and intellectual strategies for evaluation of most types of vendors now working with libraries.

References

• Alessi, D. (1992). Vendor Selection, Vendor Collection, or Vendor Defection. Journal of Library Administration, 16, (3) 117-30.
• American Library Association. (1988). Guide to Performance Evaluation of Library Materials Vendors (Acquisitions Guidelines, No. 5). Chicago, IL: American Library Association. (20 p.)
• American Library Association. (1997). Guide to Performance Evaluation of Serials Vendors (Acquisitions Guidelines No.10). Chicago, IL: American Library Association. (38 p.)
• Babu, T.K.S. & Sharma, K. (2005). Analytical Hierarchy Process for Vendor Evaluation – A Case with a Research Institute. South Asian Journal of Management, 12, (1) 101-116.
• Barker, J.W. (1986). Random Vendor Assignment in Vendor Performance Evaluation. Library Acquisitions: Practice & Theory, 10, (4) 265-280.
• Bertot, J.C. (2004). Planning and Evaluating Library Networked Services and Resources. Westport, CN: Libraries Unlimited.
• Black, G. (1994). Why Do Evaluation? Library Acquisitions: Practice & Theory, 18, (1) 57-60.
• Bonk, S.C. (1985). Toward a Methodology of Evaluating Serials Vendors. Library Acquisitions: Practice & Theory, 9, (1) 51-60.
• Brown, L.A. (1998). Approval Vendor Selection—What’s Best Practice? Library Acquisitions: Practice & Theory, 22, (3) 341-351.
• Brown, L.A. & Forsyth, J.H. (1999). The Evolving Approval Plan: How Academic Librarians Evaluate Services for Vendor Selection and Performance. Library Collections, Acquisitions, & Technical Services, 23, (3) 231-277.
• Carpenter, E.J. (1998). Is the Partnership Working?: Evaluating the Approval Plan Vendor. Library Acquisitions: Practice & Theory, 22, (3) 329-333.
• Choy, K.L., Lee, W.B., & Lo, V. (2004). An Enterprise Collaborative Management System—A Case Study of Supplier Relationship Management [Non-paginated electronic version]. Journal of Enterprise Information Management, 17, (3) 191-?.
• Crotty, A. (1994). Why Bother With Evaluation? Library Acquisitions: Practice & Theory, 18, (1) 51-56.
• Derthick, J., & Moran, B.B. (1986). Serial Agent Selection in ARL Libraries. Advances in Serials Management, 1, p.1-42.
• Dogan, I. & Sahin, U. (2003). Supplier Selection Using Activity-Based Costing and Fuzzy Present-Worth Techniques. Logistics Information Management, 16, (6) 420-426.
• Dolby, E.G. (1994a). Quest for Quality: Quality Aspects of Supplier Evaluation. Library Acquisitions: Practice & Theory, 18, (1) 45-49.
• Dolby, E.G. (1994b). Closing the Circle: The Final Stages of the Evaluation of Library Suppliers. Library Acquisitions: Practice & Theory, 18, (1) 89-92.
• Gordon, S. (2005). Seven Steps to Measure Supplier Performance [Non-paginated electronic version]. Quality Progress, 38, (8) 20-? (6 pages).
• Hirko, B. (2002). Live, Digital Reference Marketplace. School Library Journal Net Connect (Fall) 16-19.
• Humphreys, P., Huang, G., & Cadden, T. (2005). A Web-Based Supplier Evaluation Tool for the Product Development Process. Industrial Management +Data Systems, 105, (1/2) 147-163.
• Jackson, M. (2004). Selecting the “Best” Document Delivery Supplier. Interlending & Document Supply, 32, (4) 242-243.
• Jain, V., Tiwari, M.K., & Chan, F.T.S. (2004). Evaluation of the Supplier Performance Using an Evolutionary Fuzzy-Based Approach [Non-paginated electronic version]. Journal of Manufacturing Technology, 15, (8) 735-?.
• Kent, P.G. (1994a). How to Evaluate Suppliers with an Automated System. Library Acquisitions: Practice & Theory, 18, (1) 79-82.
• Kent, P.G. (1994b). How to Evaluate Serials Suppliers. Library Acquisitions: Practice & Theory, 18, (1) 83-87.
• Lam, H. (2004). Library Acquisitions Management: Methods to Enhance Vendor Assessment and Library Performance. Library Administration & Management, 18, (3) 146-154.
• Lau, H.C.W., Lau, P.K.H., Fung, R.Y.K., and Chan, F.T.S. (2005). A Virtual Case Benchmarking Scheme for Vendors’ Performance Assessment. Benchmarking, 12, (1) 61-80.
• Lewis, J.S. An Assessment of Publisher Quality by Political Science Librarians. College & Research Libraries, 61, (4) 313-323.
• Li, C.C., Fun, Y.P., & Hung, J.S. (1997). A New Measure for Supplier Performance Evaluation. IIE Transactions, 29, (9) 753-758.
• McDonough, J. (1991). Planning, Conducting, and Analyzing Serials Vendor Performance Studies. Serials Librarian, 19, (3/4) 221-223.
• Ohdar, R. & Ray, P.K. (2004). Performance Measurement and Evaluation of Suppliers in Supply Chain: An Evolutionary Fuzzy-Based Approach [Non-paginated electronic version]. Journal of Manufacturing Technology, 15, (8) 723-?.
• Orkiszewski, P. (2005). A Comparative Study of Amazon.com As a Library Book and Media Vendor. Library Resources & Technical Services, 49, (3) 204-209.
• Pavelsek, M.J. (1998). Guidelines for Evaluating E-Journal Providers with Applications to JSTOR and Project Muse. Advances in Librarianship, 22, p.39-58.
• Purdy, L. & Safayeni, F. (2000). Strategies for Supplier Evaluation: A Framework for Potential Advantages and Limitations. IEEE Transactions on Engineering Management, 47, (4) 435-443.
• Roberts, P. (1994). How to Evaluate Suppliers in a Manual System. Library Acquisitions: Practice & Theory, 18, (1) 71-77.
• Schatz, B., & Baldwin, J.A. (1998). Approval Plans and Approval Vendor Selection in an Outsourcing Environment. Library Acquisitions: Practice & Theory, 22, (4) 423-429.
• Talluri, S., Narasimhan, R., & Nair, A. (2006). Vendor Performance with Supply Risk: A Chance-Constrained DEA approach. International Journal of Production Economics, 100, (2) 212-222.
• Walters, S. (1994). Commercial Document Delivery: Vendor Selection Criteria. Computers in Libraries, 14, (9) 14-16.
• Walther, J.H. Assessing Library Vendor Relations: A Focus on Evaluation and Communication. The Bottom Line: Managing Library Finances, 11, (4) 149-157.
• Weber, C.A. (1996). A Data Envelopment Analysis Approach to Measuring Vendor Performance [Non-paginated electronic version]. Supply Chain Management, 1, (1) 28-?.