v30#6 From Affordable to Open: Evaluating Open Educational Resources

by | Feb 18, 2019 | 0 comments


by Mark Cummings  (Editor and Publisher, Choice)  

[The following is the text of a presentation given by Mr. Cummings at the Charleston Library Conference on 8 November 2018.  It was delivered as part of a three-person panel exploring “The Library’s Opportunity in Affordable Textbooks.”  Also presenting were Mark McBride, Library Senior Strategist at SUNY, and Gwen Evans, Executive Director of OhioLINK.]

  1.  My name is Mark Cummings, and I’m the editor and publisher at Choice, a publishing unit at the Association of College and Research Libraries.

Those of you familiar with Choice know that, unlike the organizations represented by my two colleagues, we are not a provider of instructional materials, affordable or otherwise, so at first blush our role on this panel may not seem obvious.  That said, we are close observers of the selection process for scholarly materials, and in that vein, in the spring of this past year we conducted a survey of undergraduate instructors that has direct bearing on the topic.  The survey was designed to discover two things relating to the issue of textbook affordability: First, how instructors discover, evaluate, and select materials for classroom instruction, and second, what, if anything, is different about the criteria or methods employed when the instructor sets out to use open educational materials.1

Our survey was deployed to about 88,000 instructors in the United States, and although the number of responses was low, just under 1,400, we were able to derive some interesting information from them nonetheless.  A few words about the distribution of responses are probably in order here, so let me briefly note that our respondents were split almost equally between two- and four-year schools (52%/48%) but overwhelmingly (83%) employed at public institutions.  Enrollment at the institutions represented by our respondents was fairly evenly distributed, with no one of the eight FTE ranges provided garnering even 20% of the total responses. Not surprisingly, STEM instructors accounted for almost half (47%) of all respondents, followed by the humanities (30%), social sciences (22%), and “other” (5%).  Introductory-level courses comprised almost 60% of the courses taught by these instructors. Our by-no-means “typical” instructor, therefore, teaches introductory algebra at a mid-sized community college that is part of the state university system in, say, California.

Before going any further, I need to point out the most obvious statistical anomaly in our results:  the overlap between instructors who claim to use all (7%) or some (60%) OER in classroom instruction and those who use at least some commercial materials (93%).  From this high degree of overlap in the choice of instructional materials we can infer that our respondents represent a self-selected — and perhaps minority — instructor population already aware of and favorably disposed to explore open educational materials.  Significantly, then, many of their responses betray a notable lack of precision as to what constitutes an open educational resource. Among the “OER” cited by some of the respondents were TED talks, Khan Academy, eBooks, “websites,” YouTube videos, and (interestingly) library holdings.  That these materials are free (to the student) but not licensed for modification and redistribution was a distinction not widely observed.

With those facts in mind, let’s take a look at the results of the survey as they relate to affordability and the issue of open education.

  1.  The survey itself comprised thirty questions, but here I’m going to limit my discussion to three elements of particular relevance to our topic: the discovery, selection, and typology of classroom materials.  With respect to discovery, Slide 1 below shows that among those using commercial materials, peer recommendation is the most important factor, followed by web searches and reviews. Significantly, only 4% of the respondents listed librarian recommendation as an element in the discovery process, a fact that bears exploration beyond the scope of this paper.

Among OER respondents, if one aggregates bibliographic research, OER repositories, and “other” (many of which listed “Google search”) generically as “search,” the same ranking of the three principal discovery elements occurs: peer recommendation, search, and reviews. And as before, librarian recommendation appears near the bottom of the ranked elements.

Next we queried instructors about factors influencing the selection of the materials discovered, asking them to tell us the relative importance of quality, cost, scope and sequence, accessibility, peer recommendations, and so forth.  Not surprisingly, among instructors using OER, affordability is the key factor in motivating the selection of course materials, second only to the quality of the materials themselves. But as Slide 2 shows, contrary to the image of instructors as oblivious (or worse) to the cost of textbooks, the same result obtained among instructors using commercial materials.  Even given the overlap in the two populations responding to this survey, it appears likely that the issue of cost has now been elevated to a position of prominence generally, especially among those teaching the large introductory STEM courses in which expensive commercial textbooks predominate.

Finally, we asked, What types of instructional materials are used in your courses?  (Slide 3) As expected, textbooks constitute the core of all course-material adoptions, used almost universally (93%) by instructors preferring commercial materials and with lower but still substantial frequency by instructors using OER (60%).  That textbook use is lower among the OER group is probably a consequence of the fact that there are fewer OER textbooks available for selection. Note also that in OER classrooms, the use of workbooks, test banks, study guides, and digital courseware falls dramatically, likely owing to the fact that these elements are frequent ancillaries to commercial textbooks.

III.  From these responses I believe that we can infer certain conclusions about the relationship between affordability and the adoption of open educational resources.  First of all, OER selection appears to be based more on affordability than on an ideological commitment to open education. The responses to the selection questions, plus the imprecision around the term OER noted above, seems to indicate that it is “free” and “low cost” that are the drivers of OER selection, at least thus far.  Second, OER appear to be selected more often as one-to-one replacements for commercial textbooks than as part of a systematic implementation of open pedagogy.  Although this is more surmise than demonstrable fact, the popularity of the excellent OpenStax textbooks (and their not-so-adaptable print formats) among our respondents suggests as much.  

Given this, I think it is reasonable to conclude that for those advocating for OER as a part of a larger program of educational transformation, affordability can no longer be the strategy of choice.  Not that affordability is not a good way to introduce OER to skeptical instructors. After all, the notion of free or low-cost course materials is so appealing on the face of it, and so morally attractive from the standpoint of social justice, that it is tempting to believe that affordability in-and-of-itself is sufficient reason for OER adoption.  But the formidable incentives now posed by inclusive access programs have seriously undermined the advantages of OER from the standpoint of affordability. It is time to move on.

The facts on the ground as elicited in our survey suggest that instructors occupy various positions on a spectrum of “Open,” from affordable to Open Education, and that as a transitional strategy for OER advocacy, as a way to move instructors and others along this spectrum, it is the quality and availability (that is, the ease of discovery) of OER material that should be now be emphasized.  (Slide 4)

  1.  Which brings us to the matter of reviews.  We have seen that peer recommendation is the single most important factor in the discovery process, while content quality dictates selection.  Functioning both as discovery tools and as sources of peer evaluation of content quality, reviews provide much of the information required by instructors prior to their own examination of the material.  But existing reviews of OER tend to be brief and lacking in formal structure, so as part of our strategic initiative around OER, and informed by our long experience in the creation and publication of reviews, Choice has created a detailed review template2 for the evaluation of open educational resources.  The template elicits evaluation in twelve areas: format and source, provenance, subject, target audience, licensing, accessibility, adaptability, content quality, pedagogy, interface design, ancillary materials, and competing works.  A set of standardized rubrics accompany these elements, along with text boxes for more detailed analysis.

It is important to recognize that course materials are evaluated and adopted by the instructors themselves, who care first and foremost about the quality of the instruction they offer.  If they are advocates of open education, they have become so only after a thoroughgoing assessment of its value for their students. For OER to become accepted as alternatives to commercial works, it is essential that instructors have confidence in them, meaning, specifically, that their quality be judged equal to or better than that of their commercial counterparts. Rigorous, objective reviews, written not as advocacy but as analysis, can play an important role in this process, creating quality benchmarks supporting the enormous creative energies liberated by the open education movement.  


  1.  The results of our study are summarized in a Choice white paper written by Steven Bell, Associate University Librarian at Temple University, available on our website at http://www.choice360.org/librarianship/whitepaper.
  2.  The Choice review template is available at https://choice360.org/content/1-openchoice/choice-oer-review-template.pdf and is published under a CC-BY license.  Please use freely and share your suggestions for improvement with us!



Submit a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.


ATG News & Announcements – 1/22/21

ATG News & Announcements - 1/22/21 New choice report studies academic collection development, reveals shift to ebooks Library Technology Guides notes that "To better understand ebook collection development in academic libraries, Choice, a publishing unit of the...

ATG News & Announcements – 1/22/21

ATG News & Announcements - 1/22/21 New choice report studies academic collection development, reveals shift to ebooks Library Technology Guides notes that "To better understand ebook collection development in academic libraries, Choice, a publishing unit of the...

ATG Quirkies: 35 Home Library Ideas with Beautiful Bookshelf Designs

Admittedly, this post in Architectural Digest isn't that quirky. However, it does offer a number of uniquely personal ideas that showcase home book collections in ways that reflect their collector's personality. Photo: Robert C. Lautman Authors Kate Jerde and Lindsey...

ATG News & Announcements 1/20/21

ATG News & Announcements 1/20/21 Now Available: The Internet Archive Makes More Than 100 Years of Editor & Publisher Fully Accessible Online infoDOCKET reports that when Mike Blinder owner of Editor & Publisher magazine learned that "the Internet Archive...


Share This