Brian A. Nosek
University of Virginia & Center for Open Science, nosek@virginia.edu
Editors’ Commentary
When many people think about ‘open’ their minds jump immediately to notions of free resources, or perhaps to the ability to collectively edit resources, as in the case of Wikipedia entries. Often overlooked in discussions about open are applications to science. In an era where science is plagued by problems such as non-replication, p-hacking, and data fabrication open offers a solution for scientific self-correction. In this chapter, author Brian Nosek showcases a number of ways in which scholarly collaborations and professional societies are using open to improve the quality of research. He focuses especially on evaluating empirical evidence, offering better training, and providing simple incentives for increasing the openness, transparency, and rigor of science.
When my 9 year-old daughter Haven learned about the scientific method in school, she learned that a scientist starts by observing what happens in the world. After collecting enough observations, the scientist generates a question and perhaps a prediction. Then, the scientist designs a study to investigate the question and test the prediction. After collecting data, the scientist learns whether the results are consistent with the prediction or not. Either way, the scientist learns something. The scientist shares the study and data with others so that they can learn too, or try the study themselves to see if they get similar results. Finally, the data are observations for making new questions and predictions.
At dinner, Haven described how her class tried out the scientific method themselves by guessing how many times a coin will show heads when they each flipped it ten times. They observed, predicted, measured, evaluated, shared what they did with the class, collated the observations, and began again. It wasn’t hard, it made sense, and it was completely exciting.
When I said that a lot of science doesn’t actually work that way, Haven was puzzled. What is different? Well, sometimes I don’t share the results of studies we do with others. Why not? Well, they don’t come out the way we expected. But, you still learned something didn’t you? Well, yes, but other scientists are not as interested in those studies. But, isn’t it important for other people to know about when you were wrong so they can learn or try it out themselves? Sure, but it takes a lot of time to share what we did. But, if you aren’t going to share it, then why did you do it? I was running out of answers, and Haven was losing interest.
The ideals of science versus the reality of science
How science is supposed to work and how it actually works are not the same thing. In a survey of more than 3,000 practicing scientists, more than 90% endorsed the norms of science such as transparency, skepticism, and disinterestedness over the counternorms such as secrecy, dogmatism, and self-interestedness.1 When asked how they behaved on a daily basis, fewer but still most scientists said that they behaved more according to the norms than the counternorms. But, when asked how others in their discipline behaved, most respondents perceived their peers to behave according to the counternorms over the norms.
This cynicism about science among scientists is not unfounded. A substantial body of evidence shows that transparency and sharing of data and methodology is the exception rather than the rule,2 and that a variety of suboptimal research practices undermine the integrity and reproducibility of the published literature. Sub-optimal practices include underpowered research designs, selecting reporting of results, and failing to distinguish between exploratory and confirmatory approaches.3
At the same time, the survey revealed that scientists want to behave according to the norms of science. The problem is that the culture of science has skewed the incentives such that researchers perceive that they are not rewarded for transparent, reproducible research, but rather for shaping data − even at the cost of accuracy − to make the most exciting, bold claims possible in order to achieve the reward of publication.4
Origins of the Center for Open Science
It is uncomfortable to be in a culture that is perceived to be misaligned with one’s values. How can we change that? One step is to show that the norms and values are actually shared, even if they are not rewarded in practice. A second step is to make it easy for people to behave according to their values, and particularly to not be punished for doing so. A third step is to surface when people are practicing the valued behaviors to signal to others that it is possible, practical, even prevalent. A fourth step is to show that the counternorms are having negative consequences on the quality of research, providing a means of reinforcing the normative behaviors. And, a final step is to shift the cultural incentives so that they actually support and reinforce the normative behaviors.
Since I joined the faculty at the University of Virginia in 2002, the members of my laboratory have contributed to addressing the first four steps, if only for ourselves. We talk about our scientific values and the practical and cultural barriers to practicing them. We look for ways to address the barriers with existing tools. And, when nothing exists to support the ideals we aim to practice, we create new tools. And, while, the members of my laboratory have no control over the cultural incentives, we also do not accept the status quo. To maintain integrity with our values, we adopted an ethic of pragmatic idealism. What does that mean? Each day, we ask ourselves ‘what can we do today to behave more closely to our ideals while still working and succeeding in the present culture?’
For example, low-powered research is a pervasive problem and increases the likelihood of both false negatives and false positives.5 That is a lose-lose situation. No good that comes from collecting smaller samples than needed to properly evaluate our research questions. To ensure that we conduct properly powered research, we created and maintain a website called Project Implicit to collect data via the internet.6 With an engaging blend of education and research, the site attracts about 1 million participants per year. That was a couple orders of magnitude more successful than we could have anticipated, but it solved our power problem for the kinds of research that could be administered via the web.
In 2006 and 2007, we wrote grants to tackle another gap between our values and practices − transparency. We wanted to improve sharing our research workflow, materials, and data, but the infrastructure didn’t exist to support this practically. Every option we found added a substantial amount of work, and we were already busy enough. We wanted to develop tools so that sharing was a benefit, not a burden. Unfortunately, our grant applications were not sufficiently compelling and they failed. Partly, we may not have had the right pitch, but another challenge might have been that the pitch was ill timed. One reviewer criticized the proposal with a simple point, ‘Researchers don’t like to share their data.’ In 2007, that was a compelling argument. Ultimately, we shelved the idea, and did the best approximation for our sharing goals by using existing tools.7
The idea was re-energized in 2011 when Jeff Spies, a senior member of the laboratory, was choosing among a variety of possible dissertation topics. As a software developer prior to coming to graduate school in quantitative psychology, Jeff had a strong sensibility for building tools that could improve his and others’ research workflow. Jeff recognized that building infrastructure was a very unusual choice for a dissertation topic in psychology, but simultaneously saw huge potential in building tools to make the research lifecycle more transparent. Jeff jumped in with both feet to build and evaluate the Open Science Framework8 as his dissertation project.
At the same time, the lab was starting a study called the Reproducibility Project: Psychology to evaluate the growing concerns about reproducibility of scientific research.9 For this project, we planned to conduct replications of a sample of published studies and compare our findings with the results of the original studies. However, we could not conduct enough replications on our own to get a meaningful sample of studies. So, we opened the project to the research community for anyone to join.
In our good fortune, there were many others with similar interests. Moreover, many were willing to donate some of their time to conducting replications. The project became a collaboration of 50, then 100, then 150 researchers. And, with the alpha version of the Open Science Framework released, we had infrastructure to coordinate the large collaboration and conduct the project transparently.
We didn’t have grant funding, but I had resources to support these projects from giving presentations to organizations about implicit bias, my substantive area of research. Unexpectedly, in the Fall of 2012, the Reproducibility Project: Psychology and Open Science Framework kindled interest from funders that was not present just a few years earlier. Following some press coverage, we were contacted by multiple foundations. After a series of emails, virtual demos, and a visit to Houston, the Laura and John Arnold Foundation gave us a grant for US$5.25 million to launch a non-profit called the Center for Open Science (COS). There had been two self-funded lab projects, and suddenly there was a well-funded non-profit. This might seem a rather abrupt acceleration. Yes, yes it was.
We launched COS with a mission to increase openness, integrity, and reproducibility of scientific research. In its first three years of operation, COS received US$18 million from a combination of private and federal funders to support its mission. The Open Science Framework and Reproducibility Project: Psychology provided a foundation to which we added a full suite of investigations, products, and services to evaluate and improve research practices. COS provides free and open tools and services to the stakeholders in science − funders, journals and publishers, universities, societies, research producers, and research consumers.
COS’s Approach
COS has three teams − metascience, community, and infrastructure − that form the basis of its strategy to increase openness, integrity, and reproducibility of scientific research.
The metascience team conducts and supports scientific research about scientific practices. The goal is to accumulate evidence about the problems and opportunities to improve research practices, and to evaluate interventions aiming to address those problems. For example, the Reproducibility Project: Psychology estimated the reproducibility rate in a sample of psychology articles and explored predictors of reproducibility. The completed project involved 270 co-authors and included 100 replications.10 That project produced a spin-off Reproducibility Project for cancer biology,11 and related efforts conducting replications of the same research protocols in many labs to examine variability in replicability12 and independent analysis of the same data by many teams to examine variability in analysis decisions and their impact on observed effects.13 These investigations provided insights about the current state of research practices.
COS also empirically evaluates whether its initiatives have an impact on research practices, positive or negative. For example, a recent grant from the National Science Foundation is supporting a randomized trial to evaluate the impact of receiving training to use the Open Science Framework. Research laboratories at University of California-Riverside will be randomly assigned to receive the training or not (and an orthogonal factor will evaluate training on the responsible conduct of research). Likewise, we evaluated the impact of one of our first and simplest initiatives to incentivize openness – offering badges on journal articles to signal open practices.14 The journal Psychological Science adopted badges to signal open data, open materials, and preregistration on January 1, 2014. From 2012 to 2013, before badges, approximately 3% of the journal’s published articles had open data. After introduction of badges, open data practices increased each half year starting in 2014 reaching 38% of published articles in the first half of 2015. Comparison journals maintained very low data sharing rates across the entire time period. Direct evidence for the effectiveness of initiatives to improve openness will facilitate their adoption and impact.
Training
Researchers possess the values of transparency and reproducibility but if they do not have appropriate training, they may not be able to translate them into practice. The community team produces articles or chapters providing guidance on reproducible practices.15 The team also creates and maintains free text and video content on the Open Science Framework for improving reproducibility and transparency, and conducts webinars and on-site trainings for laboratories, departments, and other research groups. Finally, the team offers free one-on-one virtual consulting to address statistical or methodological challenges related to reproducible research.
Even the combination of values and training may not be sufficient for increasing openness and reproducibility in daily practice. Academic researchers are busy and have significant pressures to be productive and publish in order to be competitive for jobs, earn tenure, and advance in their careers. Without realigning the incentives shaping researchers’ behavior, even the best interventions may not stick. The community team works with stakeholders in the scientific community – particularly funders and journals/publishers – to strengthen incentives for open, reproducible practices.
Badges to acknowledge open practices are a simple, low-risk, low-cost nudge toward openness.16 Badges offer no onerous requirements of journal or authors, they just offer an opportunity for authors to signal that they met specifications for open data, open materials, or preregistration.17 Badges also offer an opportunity for journals to signal that such practices are valued, even if they do not directly impact publication decisions.
Registered Reports shift publishing incentives more fundamentally to address publication and reporting biases.18 For journals that adopt Registered Reports, authors can submit the introduction and methodology for peer review before the research is conducted. Peer reviewers evaluate the importance of the research question and the quality of the methodology that will investigate it. The journal provides in-principle acceptance to submissions that survive review.19 After that, the researchers collect the data, analyze it, and report what they found. As long as they conduct the methodology effectively, the results are published whatever the outcome. In standard peer review, the incentives focus on having beautiful results, even at the cost of accuracy. In Registered Reports peer review, the incentives drive researchers to ask the most important questions and have the most beautiful methodology to evaluate those questions. Already more than 20 journals have adopted Registered Reports, and COS facilitated publication of a special issue of Social Psychology in 2014 demonstrating the viability of the approach.20
Another approach to shifting the incentives is to directly incorporate incentives for transparency and reproducibility into publication and funding. The Teaching of Psychology (TOP) Guidelines achieve this by defining eight modular transparency standards for journals and funders to adopt as policies for their authors and grantees.21 The guidelines also ease the barrier to adoption by having multiple levels of stringency so that journals and funders choose to nudge toward openness, or make a bolder requirement of their authors and grantees, depending on the circumstances for their journal or discipline. As of early 2016, more than 50 organizations and 500 journals were signatories to the TOP Guidelines.22
One of the TOP Guidelines is preregistration of analysis plans to promote a clear distinction between confirmatory and exploratory research.23 Preregistration is the law in clinical trials, but is relatively unknown in basic or preclinical sciences. Of the variety of new practices to promote transparency and reproducibility, preregistration may have the highest barrier to entry because it requires actions that are different from most researchers’ current practices. It is common practice to ‘plan’ one’s analysis strategy while conducting the analysis itself. This is fraught with risks of reasoning and reporting biases because analysis decisions can be influenced by the observation of the data.24 In order to stimulate researchers’ interest in trying out preregistration, COS launched the Preregistration Challenge25, 26 a contest in which 1,000 researchers will earn US$1,000 each for publishing their preregistered studies. The Challenge is designed as an education campaign to increase awareness and knowledge about preregistration, and initiate the behavior to possibly instill preregistration as a habit. In sum, COS develops and evaluates a variety of initiatives to strengthen the incentives for open and reproducible practices among active researchers.
Infrastructure
Supporting all of COS’s initiatives is the infrastructure to make transparency easy and practical. The Open Science Framework27 is an open source framework to connect the services that researchers use, and provide an easy means of storing, archiving, preserving, and sharing one’s research data, materials, and workflow. Researchers use the OSF as a virtual workspace to manage their projects with their collaborators. By default, the projects and materials associated with them are private – available only to those individuals that the researcher designates. At any point, the researchers can choose to make parts or all of their project publicly accessible. The flexibility integrates the researchers public and private workflows and removes the practical barriers to openness, replacing them with only the question of whether one wishes to share. Simultaneously, the OSF offers tools to preregister studies, share pre-prints of articles or other research objects, and connect the storage services (e.g., Dropbox, GitHub, institutional repositories) and other tools that researchers use (e.g., citation managers, analysis tools, data collection mechanisms, publication systems) into a single environment. Coupled with COS projects like SHARE28 − an effort to create an open dataset of all research content – we are creating public goods infrastructure that supports the entire research lifecycle.
Cultural Change
COS aims to provide support for evidence-based changes that will align scientific practices with scientific values, and ultimately improve the efficiency of knowledge accumulation and its application to advancing the social good. COS has big goals, but cannot accomplish them itself. Collaboration and collective action are essential. The design of COS as a non-profit developing exclusively free, open-source tools means that COS has no competitors. COS can collaborate with and support others working toward similar ends without concern about competitive disadvantage.
Cultural change is a coordination problem. Many stakeholders need to contribute to shifting cultural norms in concert; initiatives pursued in isolation will falter. All of COS’s initiatives depend on the broader research community supporting, embracing, and even driving changes to research practices and the culture of incentives more broadly. To the extent that COS has succeeded so far, it is largely a consequence of leaders and upstarts in the community supporting the efforts. For example, during the leadership of Eric Eich, Psychological Science initiated a variety of new initiatives including adopting badges for open practices. Also, via leadership of Alan Kraut, Sarah Brookhart, Bobbie Spellman, and many others, the Association for Psychological Science has supported COS efforts – such as adopting Registered Reports at Perspectives on Psychological Science – and other cultural changes. More generally, many researchers share our desire for change, and have been acting on their own, in collaboration with others, and in connection with COS.
The only competition that COS faces is the power of inertia and the status quo. Cultural change is hard, even when everyone agrees that change is needed. However, as 2016 begins, my simple assessment is not ‘Change is possible’ or ‘Change is coming,’ it is ‘Change is happening.’ Across the sciences, funders, journals, societies, universities, and researchers are taking on the task of identifying, testing, evaluating, and implementing changes to research practice to improve the quality and efficiency of research. Incentives are shifting toward embracing transparency and reproducibility as evidence of good practice.
But change is not complete. Researchers, particularly early-career ones, may find the uncertainties in the shifting culture unsettling. I find it exciting and liberating. The culture of incentives had previously been stacked against what most researchers believe is good practice and best for science. Now, the door is open for change and stakeholders across the scientific community are supporting that change.
We worry about kids losing interest in science as a cost to advancing knowledge and having an informed citizenry. To solve the problem, we look for ways to change their minds, and most efforts aren’t working. Perhaps we instead need to focus on changing ourselves. I almost killed Haven’s budding interest in science by describing how it actually works. As those realities shift, I can cultivate Haven’s interest by showing how those scientific ideals that she is learning in 3rd grade are borne out in the daily practice of scientists around the world.
Acknowledgements
Preparation of this chapter was supported by the Laura and John Arnold Foundation, John Templeton Foundation, and the National Institute of Aging.
1 Anderson Martinson & Vries, 2007.
2 Iqbal, Wallach, Khoury, Schully & Ioannidis, 2016; Miguel et al., 2014; Wicherts, Borsboom, Kats & Molenaar, 2006.
3 Button et al., 2013; Ioannidis, Munafo, R, Fusar-Poli, Nosek & David, 2014; Open Science Collaboration, 2015; Simmons, Nelson & Simonsohn, 2011.
4 Nosek, Spies & Motyl, 2012.
5 Button et al., 2013.
6 Implicit Harvard, n.d.
7 Dataverse, n.d.
8 Open Science Framework, n.d.
9 Open Science Collaboration, 2012.
10 Open Science Collaboration, 2015.
11 Errington et al., 2014.
12 Ebersole et al., 2016; Klein et al., 2014.
13 Silberzahn et al., 2016.
14 Kidwell et al., 2016.
15 Open Science Collaboration, 2016.
16 Kidwell et al., 2016.
17 Open Education Badges, n.d.
18 Open Education Reports, n.d.
19 Chambers et al., 2014.
20 Nosek & Lakens, 2014.
21 Nosek et al., 2015.
22 TOP Guidelines, n.d.
23 Wagenmakers et al., 2012.
24 Gelman & Loken, 2014; Simmons, Nelson & Simonsohn, 2011.
25 Preregistration Challenge, n,d,
26 COS, n.d.
27 Open Science Framework, n.d.
28 SHARE, n.d.
References
Anderson, M. S., Martinson, B. C., De Vries, R. (2007). Normative dissonance in science: Results from a national survey of U.S. scientists. Journal of Empirical Research on Human Research Ethics, 2, 3–14. DOI: https://doi.org/10.1525/jer.2007.2.4.3
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafo, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 1−12. DOI: https://doi.org/10.1038/nrn3475
Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. J. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1, 4−17. DOI: https://doi.org/10.3934/neuroscience2014.1.4
Dataverse. (n.d.). Available at https://dataverse.harvard.edu/dataverse/bnosek
Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski. B., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A., Capaldi, C., Chartier, C. R., Cicero, D. C., Coleman, J. A., Conway, J., Davis, W. E., Devos, T., Dopko, R. L., Grahe, J., German, K., Hicks, J. A., Hermann, A., Humphrey, B., Johnson, D. J., Joy-Gaba, J., Juzeler, H., Klein, R. A., et al. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68−82.
Errington, T. M., Iorns, E., Gunn, W., Tan, F., Lomax, J., & Nosek, B. A. (2014). An open investigation of the reproducibility of cancer biology research. eLife, 3:e04333. DOI: https://doi.org/10.7554/eLife.04333
Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102, 460. DOI: https://doi.org/10.1511/2014.111.460
Ioannidis, J. P. A., Munafo, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014). Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention. Trends in Cognitive Sciences, 18, 235−241. DOI: https://doi.org/10.1016/j.tics.2014.02.010
Iqbal, S. A., Wallach, J.D., Khoury, M.J., Schully, S.D., Ioannidis, J. P. A. (2016). Reproducible Research Practices and Transparency across the Biomedical Literature. PLoS Biology, 14(1), e1002333. DOI: https://doi.org/10.1371/journal.pbio.1002333
Kidwell, M. et al. (2016). Offering badges increases availability of research materials and data. Unpublished manuscript.
Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Jr., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M., Brooks, B., Brumbaugh, C., Cemalcilar, Z., Chandler, J. J., Cheong, W., Davis, W. E., Devos, Theirry, Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., Hicks, J. A., Hovermale, J. F., Hunt, S. J., Huntsinger, J. R., Ijzerman, H., John, M-S., Joy-Gaba, J., Kappes, H., Krueger, L. E., Kurtz, J. (2014). Investigating variation in replicability: A “many labs” replication project. Social Psychology, 45, 142−152. DOI: https://doi.org/10.1027/1864–9335/a000178
Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B. A., Petersen, M., Sedlmayr, R., Simmons, J. P., Simonsohn, U. Van der Laan, M. (2014). Promoting transparency in social science research. Science, 343, 30−31. DOI: https://doi.org/10.1126/science.1245317
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Bucj, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Fresse, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., Ishiya, J., Karlan, D., Kraut, A., Lupia, A., Mabry, P., Madon, T., Malhotra, N., Mayo-Wilson, E., McNutt, M., Miguel, E., Levey Paluck, E., et al (2015). Promoting an open research culture. Science, 348, 1422−1425. DOI: https://doi.org/10.1126/science.aab2374
Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137−141. DOI: https://doi.org/10.1027/1864–9335/a000192
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615−631. DOI: https://doi.org/10.1177/1745691612459058
Open Science Collaboration. (2015). Estimating the Reproducibility of Psychological Science. Science, 349(6251), aac4716. DOI: https://doi.org/10.1126/science.aac4716.
Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7, 657−660. DOI: https://doi.org/10.1177/1745691612462588
Open Science Framework. (n.d.). Available at http://osf.io/
Open Science Framewordk Badages. (n.d.). Available at https://osf.io/tvyxz/wiki/home
Open Science Framework Reports. (n.d.). Available at https://osf.io/8mpji/wiki/home/
Project Implicit. (n.d.). Available at http://implicit.harvard.edu/
Preregistration Challenge. (n.d.). Available at http://cos.io/prereg/
SHARE. (n.d.). Available at http://osf.io/share/
Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E. Bahnik, S., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., Fong, N., Gamez-Djokic, M., Glenz, A., Gordon-McKeon, S., Heaton, T. J., Hedros Eriksson, K., Heene, M., Holfelich Mohr, A. J., Hogden, F., Hui, K., Johannesson, M., et al. (2016). Many analysts, one dataset: Making transparent how variations in analytical choices affect results. Retrieved from https://osf.io/j5v8f/
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
TOP Guidelines. (n.d.). Available at http://cos.io/top
Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638.
Wicherts, J.M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726–728. DOI: http://dx.doi.org/10.1037/0003–066X.61.7.726
How to cite this book chapter:
Nosek, B A. 2017. Opening Science. In: Jhangiani, R S and Biswas-Diener, R. (eds.) Open: The Philosophy and Practices that are Revolutionizing Education and Science. Pp. 89–99. London: Ubiquity Press. DOI: https://doi.org/10.5334/bbc.g. License: CC-BY 4.0