Regan A. R. Gurung
University of Wisconsin-Green Bay, gurungr@uwgb.edu
Editors’ Commentary
Open Education is a relatively new phenomenon. As such, it will have skeptics and detractors. Among the common doubts concerning OERs are questions related to their quality. Indeed, in many markets price is a partial indicator of quality. The free-of-charge nature of open education means that prospective consumers have one less indicator of potential quality with which to judge these resources. It is in this context that author Regan Gurung tackles the foundational question of whether OERs are any good. He points to preliminary research on the topic. While early research shows mixed results Gurung also offers insights as to how we ought to be evaluating quality more broadly. Among his recommendations are outcomes research, materials that have been peer reviewed, and expert authorship.
How do you know when you have a quality product? If it is a meal at a fine restaurant or a beverage concocted by a skilled mixologist there are some dead giveaways. It is expensive. It tastes good. It looks good. It makes you feel good. Can we use the same rubrics for Open Educational Resources (OERs)? How do you know if an OER is high quality? There are some direct similarities to a good meal but of course, many more differences. As someone relatively new to the whole concept and use of OERs, this seemingly simple question provoked some deep digging. I have some answers and I’d like to share some factors for your consideration.
There are many ways to learn. I like to think that armed with a curious mind and the right resources and motivation, anyone can learn by themselves. Of course, when we think of learning we do not think of the solo pursuits of motivated individuals. We tend to think of schools and colleges. We all want a quality education for our children and lists extolling the virtues of select institutions to deliver such educations abound (think US News and World reports). When we dive deeper into what makes college a quality experience we easily settle on faculty and the classes they teach. While master teachers can inspire with their passion and masterfully deliver content, most students rely heavily on course materials the faculty assign (though the students may not always read all of it) to solidify content acquisition. Sure classroom discussion and deep processing may inspire the master student, but it is the textbook that is the crutch of the average student. Consequently, the quality of course material is of tantamount importance. Yes, of course you know the quality of the material is important, but I state this at the outset because I fear that unconsciously at least, some instructors may believe their brilliance transcends the need for quality course materials. Passionate, organized, motivating, and knowledgeable instructors who build student-rapport are important to learning, but quality material is important as well. I am getting off the soapbox now.
Is it Expensive?
Once upon a time, you could rely on the simple heuristic that pricey equals quality. As the social psychologist Robert Cialdini (1993) showed some time ago, we humans are easily influenced by price. He tells of a sales trick where cheap dollar jewelry actually sold more when its price was hiked up many times. People actually paid more for a pair of earrings when the price tag was changed to be higher. The expensive meal must be better than the cheap meal. The gourmet taco for ten dollars should be better than the two-dollar food truck taco. This is sometimes the case and reasons are clear. Gourmet taco making chefs may be trained to best coax flavors from hitherto untried pairings. They may use more expensive ingredients. A similar intuition accompanies the common belief that expensive textbooks are better. After all, expensive textbooks have the backing of major publishing companies who have invested large sums of money to ensure quality products right? The development editors, slew of peer reviewers examining every draft of every chapter, and focus groups should ensure a quality product right? There is some evidence that this is the case.
There is no doubt that most expensive books come with a lot of bells and whistles. Big Publisher Books (BPBs) as I like to call them, are multicolored affairs. They are packed with pictures, can often afford the rights to comics and cartoons, and come with a wide array of textbook technology supplements (online quizzes, etc.). BPBs also tend to have well known authors, recognizable for their research chops, and undergo a long arduous process of review. We assume that for all of these reasons, the expensive BPB must be high quality. We can, and I have, tested many of these assumptions.
Do students like all BPBs the same?
Do students like some books more than others? This question is difficult to test without comparisons across class sections or universities, as each class uses only a single book. The good news is a number of such tests exist.
One of the best way to compare books is to have the same student read a number of books. I did a study some years ago where I had students rate a number of most adopted textbooks in the introductory psychology market.1 I had students first read a chapter from each of seven textbooks (yes, it was a long study and I paid each student US$10.00), and then rate the book using a 28 item survey. I randomized the order of the books so students saw different books at different points in the line up. The bulk of the questions came from the validated Textbook Assessment and Usage Scale2 and measured opinions about the figures, tables, photographs, research examples, application examples, pedagogical aids, visual appeal, and writing quality. Students did differentiate between texts rating some books better than others. From a student opinion perspective, not all BPBs are rated the same.
In a number of national studies, colleagues and I had students rate the quality and helpfulness of their introductory psychology textbooks.3 The students also rated how much they thought they learned from the books and then took a quiz on material taken from the learning and biological psychology chapters. All students took our quiz so we had a common measure of learning. All students in our studies used BPBs. We found some significant differences in how students rated how much they learned and their textbooks (some books were rated higher in quality and helpfulness than others), but quiz scores were similar. Regardless of which textbooks the student was using, their quiz scores on our test did not vary. In this case, students did like some books better but did not learn better from some books. Given learning should be an instructor’s key focus, this outcome bears a great focus.
How does Learning Vary Between Books?
Fine you may say, so students like some books better than other. Regardless of preference do students learn better from some books over others? I took the lab study mentioned above a step further. In a second study I had students come in to my laboratory, read chapters from two different books, and take a quiz on what they had read. Students read a biology chapter from the first book and the learning chapter from the other book. They also rated two additional books. When I compared the quiz scores I did not find any significant differences between quiz scores regardless of the chapter tested or textbook used. Students seem to learn the same in a lab test of textbooks but of course the lab is not the classroom. Wouldn’t it be great to be able to control for instructor and lecture content but still test at least two books?
One semester some years ago I did exactly this. I selected two well adopted BPBs that varied in ‘look.’ One was in what is called a magazine format: lots of pictures, a layout resembling Cosmopolitan or some other glossy. The other was a standard brief edition of an introductory psychology book. Both books had a similar reading and difficulty level. I worked with my campus bookstore and gave students in my Intro psychology class a choice. They could buy whichever book they wanted. The bookstore even set up a little booth where they could page through each book and decide which one they wanted. About 60% of the class picked the magazine format book. When it came to exam time, I wrote two forms of the exam tailoring exams to the book bought by the student − the bulk of the questions were the same but a small number explicitly mentioned material from the book the student picked. At the end of the semester, the students rated the magazine format book higher in visual quality (no surprise) but there were no differences in exam scores. Again, students do not seem to learn differently from different BPBs. But now for the big question: How do OERs compare?
OERs and BPBs: Head to Head
Research testing OERs and BPBs is still in its infancy but a number of studies have attempted to assess if OER use influences student learning.4 The story is mixed. The best studies using standardized or similar exams5 show no differences in exam scores between OER users and BPB users. A number of studies demonstrated questionable statistical validity. For example, although one suggests OER users show higher exam scores in psychology classes, the study did not include statistical tests of the difference and the exams used (across two different semesters) may not have been equivalent in difficulty.6 In another study of OER use in English classes, authors state students who used OERs scored higher on reading tests than peers using traditional means, but again no statistical tests were reported and it was unclear if the assessments were comparable.7 In short, the current research comparing OERs and BPBs is fraught with limitations and validity issues.
In an attempt to transcend the limitations of extant studies, I recently compared a group of OER users to BPB users. In collaboration with the NOBA project, instructors at seven different schools invited their students to take part in a study of learning. Over a thousand students took part in the study. A little over half of the students used an OER (a NOBA intro psych textbook) and the rest used a BPB. I compared student perceptions of the material and similar to the other studies discussed above, also had all students take the same test (mine). In one of the first studies of its kind pitting OER against BPBs, students using an OER rated the material as more applicable to their lives. Score one for OER. Students using the OER also rated the quality of the photographs, figures, tables, and boxed information as lower than the BPB. Score one for BPB. When it came to study aids, writing quality, the examples, research studies or the extent to which components of the textbook helped students understand the material, there were no differences in quality between OER and BPBs. Given that the OER was free and the BPBs on average cost over US$100, multiple major scores for OER here. This finding was tempered by test score data. Students using a BPB did significantly better on the quiz. Unfortunately, this finding is potentially contaminated by the reality of the study test items coming from the psychology Advanced Placement exam, something that most BPB testbanks also draw from. A better test of OER users to BPB users would come from using a more neutral set of test questions (a study currently in progress).
It is important to note that the playing field is not equal. There are some significant differences in the current state of many OERs and BPBs. Whereas these differences may not be as important for upper level classes, the differences may be particularly important for lower level classes. In general, BPBs do tend to have higher production value. In addition to more color and comics mentioned before, they also have numerous pedagogical aids built into them. Not only do the books themselves have many features such as multicolored fonts, boxes, and running glossaries, but the books also have numerous textbook technology supplements. These study aids on the publisher websites allow students to test themselves on the material, often in a variety of engaging formats such as crosswords, matching games, and application questions. These textbook supplements do help students learn.8 Many publishers also provide ready-made flashcards to accompany the book, something few if any OERs provide (OERs are quickly catching up though). Whereas these pedagogical aids may not necessarily always aid learning,9 the perception that they do is often enough to get instructors to adopt BPBs and rationalize the price tag of the same. Finally, whereas many BPBs are also available online students tend to still prefer hard copy books. Most OERs are online by default and whereas some OERs afford a hardcopy version (for a small charge), few of the hard copy paper versions are as colorful as even the cheapest BPBs. While one should not judge a book by its cover, it is possible that students associate flashy production values with quality and are more likely to read a book awash in multiple cartoons, graphics, photographs, and figures.
So the story so far is this. There are some differences between the major BPBs in terms of how students’ perceive them though no significant testing differences. There is a dearth of research comparing OER learning/test scores with learning/test scores on BPBs (most studies assess cost savings and perceptions),10 though one study suggests BPB users may have an advantage on standardized tests. In short, there is currently no definitive answer to whether the quality of OERs especially as compared to BPBs and as measured in learning is high. What then can we use to measure quality? It is time to return to the usual suspects.
One Open Educational Resources site nicely summarizes the main places to look.11 When assessing the quality of an OER, you can look at the reputation of the author of the piece, the institution or body curating the piece, the accuracy, and also factors such as accessibility and fit for purposes. The last two factors may not directly influence learning, one of my main criteria for ‘quality’ but are important nonetheless. When we talk about quality in higher education we tend to rely on the credibility of authors and the peer review process and this is where I put my money. If you want a quality OER, the fact is that it is going to be difficult to get learning outcome data. In fact, there is little learning outcome data for the use of BPBs but faculty still adopt these books. One of the major reasons why many faculty do adopt BPB books is because other faculty do (crazy circular logic on one hand). If many people adopt BPBs they must be good is the thinking. If it is put out by a publisher whose name is recognizable it must be good. If it written by an author who is familiar, it must be good. In fact, these are all empirical questions that are never really tested. The market research that big publishers cite and the student and faculty endorsements peppering the back covers and promotional materials of BPBs rarely (if ever) represent true comparisons of learning. To be fair, true comparisons of learning are difficult. A variety of factors − the student, the teacher, the textbook- all influence learning, which makes such research difficult.
So where does that leave me? If I know the author of an OER has a strong reputation and I know the piece has been reviewed by peers that will make me more likely to entertain the use of the OER. Fortunately openly published reviews by faculty colleagues is becoming more common (e.g., Open Textbook Network). Beyond that, we faculty have the responsibility to monitor the accuracy of material, something difficult to do when you are using resources for a class whose breadth expands beyond your own personal expertise (e.g., Introductory classes). The more faculty who use OER, the more these materials will be vetted. One of the most appealing features of OERs is that users are invited to modify the resources and with more faculty using OERs the better these resources will be. In many ways this is a form of academic freedom that will preclude having instructors bending the course to conform to a textbook. The sifting and winnowing of material will help OERs evolve. BPBs have inherited a veneer of quality not based in empirical tests of their links to learning. Well curated OERs, those where the writing and content is monitored and reviewed by peers and contributed by credible sources, deserve to likewise bask in the reflected glory of BPBs. My in-depth perusal of many OERs in psychology and research on faculty perceptions of OERs12 show that the OERs are ready for their time in the spotlight while scholars of teaching and learning work to assess true quality of all educational resources. OERs are tasty meals at the right price, free.
Notes
1 Gurung & Landrum, 2012.
2 Gurung & Martin, 2011.
3 Gurung, Daniel & Landrum, 2012; Gurung, Landrum & Daniel, 2012.
4 Allen, Guzman-Alvarez, Molinaro & Larsen, 2015; Bowen, Chingos, Lack & Nygren, 2014; Fischer, Hilton, Jared-Robinson & Wiley, 2015; Hilton, Gaudet, Clark, Robinson & Wiley, 2013; Hilton & Laman, 2012.
5 Allen et al., 2015; Bowen et al., 2014; Hilton et al., 2013.
6 Hilton & Laman, 2012.
7 Pawlyshyn, Braddlee, Casper & Miller, 2013.
8 Gurung, 2015.
9 Gurung, 2004.
10 Open Education Group Publications, n.d.
11 Open Educational Resources, n.d.
12 Allen & Seaman, 2014.
References
Allen, G., Guzman-Alvarez, A., Molinaro, M., & Larsen, D. (2015). Assessing the impact and efficacy of the Open-Access ChemWiki Textbook Project. Educause Learning Initiative Brief. Retrieved from https://net.educause.edu/ir/library/pdf/elib1501.pdf
Allen, I. E., & Seaman, J. (2014). Opening the curriculum: Open educational resources in U.S. higher education, 2014. Retrieved from http://www.onlinelearningsurvey.com/oer.html.
Bowen, W. G., Chingos, M. M., Lack, K. A., & Nygren, T. I. (2014). Interactive Learning Online at Public Universities: Evidence from a Six-Campus Randomized Trial. Journal Of Policy Analysis And Management, 33(1), 94−111.
Cialdini, R. B. (1993). Influence: The psychology of persuasion. New York, NY: Morrow.
Fischer, L., Hilton, J. I., Robinson, T. J., & Wiley, D. A. (2015). A multi-institutional study of the impact of Open Textbook adoption on the learning outcomes of post-secondary students. Journal Of Computing In Higher Education, 27(3), 159−172.
Gurung, R. A. R. (2004). Pedagogical aids: Learning enhancers or dangerous detours? Teaching of Psychology, 31, 164−166.
Gurung, R. A. R. (2015). Three investigations of the utility of textbook teaching supplements. Psychology of Learning and Teaching, 1, 48−59.
Gurung, R. A. R., Daniel, D.B., & Landrum, R. E. (2012). A multi-site study of learning: A focus on metacognition and study behaviors. Teaching of Psychology, 39, 170−175. DOI: https://doi.org/10.1177/0098628312450428
Gurung, R. A. R., & Landrum, R. E. (2012). Comparing student perceptions of textbooks: Does liking influence learning? International Journal of Teaching and Learning in Higher Education, 24, 144−150.
Gurung, R. A. R., Landrum, R. E., & Daniel, D. B. (2012). Textbook use and learning: A North American perspective. Psychology of Learning and Teaching, 11, 87−98.
Gurung, R. A. R., & Martin, R. (2011). Predicting textbook reading: The textbook assessment and usage scale. Teaching of Psychology, 38, 22−28. DOI: https://doi.org/10.1177/0098628310390913
Hilton, J. I., Gaudet, D., Clark, P., Robinson, J., & Wiley., D. (2013). The adoption of Open Educational Resources by one community college math department. International Review Of Research In Open And Distance Learning, 14(4), 37−50.
Hilton, J., & Laman, C. (2012). One college’s use of an open psychology textbook. Open Learning, 27(3), 265−272.
Open Education Group Publications. (n.d.). Available at http://openedgroup.org/publications
Open Educational Resources. (n.d.). Available at https://openeducationalresources.pbworks.com/w/page/24838164/Quality%20considerations
Pawlyshyn, N., Braddlee, D., Casper, L., & Miller, H. (2013). Adopting OER: A case study of cross-institutional collaboration and innovation. Retrieved from http://er.educause.edu/articles/2013/11/adopting-oer-a-case-study-of-crossinstitutional-collaboration-and-innovation
How to cite this book chapter:
Gurung, R A R. 2017. Are OE Resources High Quality?. In: Jhangiani, R S and Biswas-Diener, R. (eds.) Open: The Philosophy and Practices that are Revolutionizing Education and Science. Pp. 79–86. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbc.f. License: CC-BY 4.0