Your post was fascinating, as it brought up many points of direct interest to me. I can imagine sharing a pleasant coffee (or something harder) with you discussing these. So, this response may be a touch eclectic.
I remember when the Science (“Estimating the reproducibility of psychological science”, 2015) article came out last August and posted a note about it in the chat forum. The lack of replicability of 65% of psychology studies seemed worrying. My take then was that educationalists may need to be careful over which theoretical studies to use on which to base their research as a lot of educational research has roots in psychology. I also felt that this allowed educational research more freedom theoretically because a lack of theoretical base gives educational research the ability to create its own bases.
However, now I’m not so sure. These 100 studies were quantitative ones. Replicating 100 qualitative studies is practically and theoretically impossible (as we never stand in the same river twice, to paraphrase Heraclitus). If educational research is to be taken seriously by the end users (c.f. Harker, 2015), social scientists should perhaps be wary of undertaking unreproducible qualitative studies. Although Bryman, Becker and Sempik’s (2008) produce a criteria for quality in qualitative research, I wonder if much of the general public are able to distinguish between the general notion of science (Harker, 2015) and what social scientists do. For us in our narrower environments, perhaps this isn’t an issue. But personally, I’d feel awkward telling people that my doctoral thesis was in ‘The Beatles’ (Michaels, 2011).
You comment that “even the most hardened naturalist struggles to maintain their view when confronted with the classroom environment” (Simon, 2016). My introduction to quantitative research was through a biology Ph.D. who taught English at my current school. Over many beers, he would dissect the day’s classroom experiences into possible areas for empirical, quantitative questioning. His view was that it’s an intellectual cop-out to go qualitative before the quantitative issues are settled. What I think he meant by this was that (after Lund ) as qualitative questions are more appropriate for the generation of hypotheses, it appeared that enough hypotheses were already derivable from the messiness of classroom interaction. Qualitative research that looked for other hypotheses was a waste of time. I suspect that this reflected his biology background, but I also suspect that his attitude is shared by many both inside and outside education.
I loved the diagramme of cognitive biases. They are all based on quantitative research models. How would such a diagramme look for education studies?
Bryman, A., Becker, S., & Sempik, J. (2008). Quality criteria for quantitative, qualitative and mixed methods research: A view from social policy. International Journal of Social Research Methodology, 11(4), 261–276. doi:10.1080/13645570701401644
Harker, D. (2015). Uncertainty and bias in science and society. In Creating scientific controversies. Cambridge: Cambridge University Press.
Lund, T. (2005). The Qualitative–Quantitative Distinction: Some comments. Scandinavian Journal of Educational Research, 49(2), 115–132. doi:10.1080/00313830500048790
Michaels, S. (2011). Canadian woman is world’s first Beatles graduate | Music | The Guardian. Retrieved January 19, 2016, from http://www.theguardian.com/music/2011/jan/27/worlds-first-beatles-graduate
Simon, S. (2016). Naturalist and Constructivist Inquiry [online forum post]. University of Liverpool.