The term evidence-based1 is often used in discussions about the use of various digital technologies in education, i.e. "show me the proof, evidence that this works or does what you claim it does". In the current climate of evidence-based policy, or as some critics have suggested, policy-based evidence, it is difficult to imagine that the practices commonly found in formal education settings are without evidence.

But do we have evidence that age-based classrooms are a good thing? Do we have evidence to support the various testing and teaching regimes that are deployed? What about learning styles? What about the assumptions made about human learning and cognition? If education is to aspire to be "scientific", then the position on all of these things must be that they are able to be falsified, i.e. proved wrong. That is how science, albeit imperfectly, works.

Papert2 in 1972, parodied attempts to be "science-like" in the evaluation of computer-based learning. He suggested that the failure to find significant differences in favour of computer-based approaches was like the failure of a 19th Century engineer who failed to show that engines were better than horses.

This he did by hitching a 1/8 HP motor in parallel with his four strong stallions. After a year of statistical research he announced a significant diference. However it was thought that there was a Hawthorne effect on the horses … the purring of the motor made them pull harder.

In other words, somehow proving that electronic whiteboards are better than chalk boards is a rather silly enterprise.

A long time ago, I described a somewhat different way to think about what is going on3:

Here is a model for the cycle that appears to operate.

1 A new technology appears on the market.

2 Arguments are made concerning the improvements the technology will make to existing teaching/learning practices. This is an important and necessary step in terms of recruiting support. The technology has to be positioned so it appears as a solution to a particular problem. So the initial story has to be constructed around current practices. It would make little sense to claim an outcome that was unfamiliar and impossible to foresee the unexpected outcomes. It’s only justification can be in terms of what is already known, but importantly it has to be in terms of a current problem of some kind or other. This is what Sproull and Kiesler4 call first level effects— “the planned efficiency gains or productivity gains that justify an investment in new technology”.

3 The justification for acquisition is successful and the new technology is put in place5

4 Then one of two things happen. In the process of adoption, interesting things happen that bear little relationship to what was imagined, what Sproull and Kiesler call second level effects, “..people pay attention to different things, have contact with different people, and depend on one another differently” (Sproull and Kiesler 1991, 4). There is little point or interest in evaluating the original claims.

Alternatively, step 1, a new technology appears and new efforts go into making a new case for its acquisition.

The cycle repeats itself with the regularity of the seasons, except that the seasons for the information technology industries are more likely to be of the order of weeks rather than months. I am not suggesting that participation in this loop need necessarily be construed as evidence of mindless consumption. Though I do believe that operating in the loop with a view that it is all inevitable is worrying to say the least.

It is not all a matter of out of control consumption by schools. There is a potential constraint on continuing to cycle in the loop. The human costs associated with implementing and then maintaining each innovation are high. It is not simply a matter of putting a particular piece of technology in place and there being no or little further costs. Even the simplest piece of classroom “edutainment”6 takes some time and effort to install and then requires ongoing support for its ongoing use in classrooms. Suchman7 points to the importance of what she terms “articulation work”, the human effort required to engineer a particular technology into a particular setting and keep it working harmoniously. The more complex the software or hardware the more effort is required. If a particular configuration of hardware, software and people continues to work, then these human resources are unavailable for a newer piece of technology. It is simply not possible to continue adding technologies without discarding ones adopted earlier. The simplest and least controversial example of this is an upgrade to a piece of application software. In a class situation, teachers know how much simpler it is to support a single operating system, common hardware configuration and standard application software. These considerations are familiar to all who juggle the resources of a school or classroom.