Approval Plan Comparison: A Methodology

By Mark Funk
Head, Collection Development
Weill Cornell Medical Library
New York, NY

March 2005 | Printable version (pdf)

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this li-cense, visit
http://creativecommons.org/licenses/by-nc-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

Introduction
Most medical school libraries use an approval plan to receive a majority of their book purchases. Once set up, a plan is convenient for selecting books, generally saves time for the selector, and usually gets books to the users more quickly than other methods. However, there may come a time when the selector responsible for an approval plan becomes vaguely dissatisfied with it—either with the quality of the books coming in, their quantity, or the speed with which they arrive after publication. The selector may feel the need to find a new approval plan vendor. But how does one make a decision to switch vendors? One certainly doesn’t want to make a decision on a sales pitch or a single visit to an exhibitor’s booth. One can approach other libraries and ask them about their experiences. However, other libraries’ situations may be very different from yours, so their experiences may not apply. In 1989, two years after I arrived at Cornell, I became unhappy with the quality of our approval plan, and decided to do a head-to-head comparison with another vendor. As a result of that test, we switched vendors. In 1994, another vendor approached me, asking for a similar comparison. In that case, we did not switch vendors. Since the methodology I used was very useful for me and others, I am sharing it via the internet. This is a practical, “how I did it” presentation, not a scholarly treatise. Adjust as necessary for your needs.

General Approach
Switching an approval plan to a different vendor is a significant undertaking. However unhappy you are with your current vendor, you should have objective data showing the new vendor is better than the old before switching. I felt that a head-to-head comparison would give me the data I needed. The basic design of the test would be as follows:
• I would take the current approval plan profile and translate it, as closely as possible, to the profile offered by the competing vendor.
• The competing vendor would be doing a “virtual” approval plan during the test. No actual books would be sent, only forms indicating a selection they made for my library, based on the approval plan profile I completed.
• The test would last three months, consisting of 12 weekly shipments.
• Both vendors would be notified of the test plans and methodology.

Factors to Measure
In deciding how to evaluate an approval plan, I decided there were three factors that were important to me:
• Accuracy of Selections: Was one vendor more accurate in their selection decisions? Inaccuracy means more returns and more library staff processing.
• Number of Selections: Did one vendor supply more items than the other? Being assured of receiving items you expect means peace of mind, and less checking of catalogs and blurbs.
• Speed of Processing: Does one vendor get books from publishers and ship them to customers more quickly than the other? I hate it when books already in the medical bookstore don’t arrive on approval until months later.

Before Starting the Test
Translating your existing approval plan profile into the format of the competing vendor is probably the most important factor in getting meaningful results. Approval plan vendors have their own unique approval plan formats. They may be subject oriented, publisher oriented, call number oriented, or any combination of these. It is essential to take enough time to make both profiles as closely alike as possible. Be sure you understand how each vendor defines and handles formats you want to include or exclude from your profile, such as proceedings, pocketbooks, workbooks, numbered series, etc. Even when you are sure they are as similar as possible, it is a good idea to have a colleague look at them for discrepancies before starting the test.

Some libraries receive only books on approval. Other libraries receive both books and forms (slips). If your current plan includes both books and forms, it is better (although more work) to compare both. For the “virtual” plan, be sure this vendor labels their weekly selections as either book selections or form selections. If your current plan has only book selections, stick to comparing those.

Be sure you understand how each vendor dates their shipments and forms. Books arrive every day at a vendor, but an approval shipment is generally sent once a week. It is the week of “shipment” (either real or virtual) that will be tracked, not the date the books were received by the vendor, or entered into their computer system. Both vendors may not use the same weekday to ship. I normalized both vendors’ ship dates to the Friday of each week. For example, if one vendor ships on Wednesday, and the other on Friday, I used the Friday date for both vendors. I only measured differences in speed by weekly increments (7 days, 14 days, 21 days, etc.

Let the Test Begin: Evaluating the Selections
Twelve weeks of a duplicated approval plan means keeping track of a lot of forms. A spreadsheet or database is required. If you are evaluating both book and form selections, there will be four piles of forms to maintain and record:
• Book selections from Vendor A
• Form selections from Vendor A
• Book selections from Vendor B
• Form selections from Vendor B
Each selection must be evaluated for accuracy—how you felt it did or did not fit into the profile you created. For ease of tracking and sorting in the spreadsheet, I created a short code for each level of accuracy:

• Categories for book selections:
o Correctly identified as a book selection (CIB)
o Book selection that should have been a form selection (XF)
o Should not have been sent at all (XX)

• Categories for form selections:
o Correctly identified as a form selection (CIF)
o Form selection that should have been a book selection (XB)
o Should not have been sent at all (XX)

Tracking Items
Every selection sent by each vendor is entered into a spreadsheet. It is best to do this at least weekly, so as not to get behind. By evaluating and entering data often, you can also catch misunderstandings from either vendor, and have an opportunity to correct them. Here’s a sample of my spreadsheet for the book selections:

	Vendor A	Vendor A		Vendor B	Vendor B	Vendor A 1st (+days)
TITLE	WEEK	PERF.	COMMENT	WEEK	PERF.	Vendor B 1st (-days)
Abdominal pain: guide to rapid diagnosis	12/9/94	CIB		12/23/94	CIB	14
Acute coronary care	1/13/95	CIB
Adult intergenerational relations	11/25/94	XF	Proceedings
Analysis of addictive and misused drugs	12/9/94	CIB		11/18/94	CIB	-21
Buprenorphine			Publisher not in profile	1/27/95	XX

Each item received gets its own row. The date represents the Friday of the week each item was shipped by the vendor. A performance code is entered for each item received from each publisher, indicating how accurate the selection was. If it would be useful, I also entered a short comment. The last column automatically calculates the number of days a vendor shipped an item before or after the other vendor. I set up a similar spreadsheet to track the form selections sent by each vendor.

Summary Statistics
In order to make sense of the huge number of items received over the test (we had over 500 books and over 600 forms in our three month test), a summary is necessary. I measured three aspects: quantity, quality, and speed. The relative importance of each factor will vary by librarian. I wanted to look at a combination of all factors. Since a vendor could theoretically “win” in quantity by sending everything they received, I deducted for selection errors to account for this. My summary included: total books sent, unique books sent, the number of selection errors (XF and XX), adjusted unique books (the number of unique books from each vendor, minus the number of selection errors each made), and shipment speed. Here is how the initial summary for books looked:

Total Books sent by Vendor A: 440
Total Books sent by Vendor B: 241

Unique Books Sent by Vendor A: 287
Unique Books Sent by Vendor B: 88

Selection Errors by Vendor A: 124
Selection Errors by Vendor B: 40

Adjusted Unique Books (Unique — Errors) by Vendor A:163
Adjusted Unique Books (Unique — Errors) by Vendor B:48

Books Sent By Both Vendor A and B:153
Vendor A First: 126 | Avg. # Days First 18.7
Vendor B First: 13 | Avg. # Days First 15.6
Ties 14

A similar summary was done for the form selections.

Preliminary Results
After the 12 weeks of approval shipments, I sent both vendors the spreadsheets and summary statistics, as shown above. In a cover letter, I explained the performance abbreviations, how I calculated adjusted unique books and forms, and asked both vendors to check on titles they did not send, but which their competitor did. (These titles may have been processed by a vendor outside the test dates.) I also asked for comments and corrections.

Final Spreadsheet
After I received replies from both vendors, I adjusted and corrected the spreadsheet as necessary. I added dates for titles shipped before or after the test period. (Because I could not evaluate the performance of these titles not sent during the test, I gave the same performance for both vendors. In these cases the date became the only differentiating factor.) I also corrected any mistakes I made that were pointed out by the vendors (e.g., why they did or didn’t sent a selection; why I mistakenly marked a selection as incorrect, etc.) After all corrections and additions were completed, I recalculated the summary statistics.

Final Decision
With the final spreadsheet completed, I made my evaluation and sent a summary letter to both vendors. In the letter, I included the pertinent data that effected my decision. The appendix is an edited version of one of the summary letters.

In the comparison test we did, the final decision was relatively easy to make. The results, that Vendor A performed much better than Vendor B, were obvious. Close results will make the decision difficult. If the results are close enough to be insignificant, it may be easier to stay with the current vendor and work on making them better, rather than making a switch that comes with many new adjustment problems.

After the Decision
In our case, we stayed with our current approval plan vendor. However, during the test, I identified some problems with their selection accuracy. I worked with our selector on these problems and we were able to hone our approval plan profile, so that inaccurate selections were decreased. Whether you decide to switch approval plan vendors or stay with your current vendor, a head-to-head comparison will give you an opportunity to improve your approval plan process.

Appendix: Summary Letter to Vendor

Approval Plan Coordinator
Vendor A

Dear APC:

After receiving comments and corrections from both Vendor A and Vendor B, I have been able to complete my performance analysis of the three-month approval plan test. I have enclosed the updated spreadsheets, as well as my previous letter that explained my abbreviations, dating, and counting techniques.

Some of the numbers have changed from the original spreadsheet I sent you, because some title errors and just plain mistakes were pointed out to me. Also, I have included titles from each vendor that were reported as shipped before or after the three-month period. These cases received the same performance rating, with the appropriate dates added.

Because of the large size of the test, I received detailed reports/corrections for the Book Selection data, but not for the Forms Selection data. To me, book selection is by far the more important issue, although I will report on what I have for the Forms Selection.

I was interested in three factors: the number of selections made, the accuracy of these selections, and the shipped date of the selections. Since a vendor could potentially send every single book received to create a large number of selections, I adjust the number of unique selections from each vendor by subtracting the number of selection errors.

Book Selection Results

Total Books sent by Vendor A	447
Total Books sent by Vendor B	273

Unique Books Sent by Vendor A	250
Unique Books Sent by Vendor B	76

Selection Errors by Vendor A	126
Selection Errors by Vendor B	49

Adjusted Unique Books (Unique — Errors) by Vendor A	124
Adjusted Unique Books (Unique — Errors) by Vendor B	27

Books Sent By Both Vendor A and B	197
Vendor A First	160
Vendor B First	21
Ties	16

Average # Days Vendor A Was First	25.5
Average # Days Vendor B Was First	17.3

Form Selection Results

Total Forms sent by Vendor A	446
Total Forms sent by Vendor B	347

Unique Forms Sent by Vendor A	326
Unique Forms Sent by Vendor B	227

Selection Errors by Vendor A	78
Selection Errors by Vendor B	89

Adjusted Unique Forms (Unique — Errors) by Vendor A	248
Adjusted Unique Forms (Unique — Errors) by Vendor B	138

Forms Sent By Both Vendor A and B	120
Vendor A First	102
Vendor B First	5
Ties	13

Average # Days Vendor A Was First	21.7
Average # Days Vendor B Was First	18.2

Looking at the adjusted unique books, it appears that Vendor A is able to supply quite a bit more books than Vendor B. Even when ignoring the errors (since Vendor B was working from little experience with Cornell, and would be expected to improve over time with feedback), the number of unique books supplied by Vendor A is triple the number of unique books supplied by Vendor B. In addition, of the 197 books selected by both vendors, Vendor A processed 160 first, and Vendor B only 21. Results for form selection mirror those for books.

I have had a long association with both vendors, and have friends in both companies. It is difficult to choose between friends. However, the results of this test indicate to me that Cornell Medical Library would receive better service by keeping our approval plan with Vendor A.

Sincerely,

Mark Funk
Head, Collection Development

encl.

Medical Library Association

Approval Plan Comparison: A Methodology