Drop Testing EduTech

A researcher I know has devoted three years to following a group of low-income students in the Baltimore area who have been learning geometry with the help of an innovative online program. Her paper (which isn't published yet) is a marvel of careful observations and statistical analysis. Its conclusion, however, is poignant: not only did the students who used the computer program not learn more geometry than the ones taught the old-fashioned way--they might have learned less.

The program was thoughtfully designed and took advantage of the latest and greatest learning algorithms. If any program should be able to help students learn geometry, one might be tempted to conclude, it should be this one. That kind of logic could give ammunition to those who declare that computer-assisted learning is bunk.

But there's more behind the story.

The researcher told me (and is writing in the paper) that she observed even the most well-intentioned teachers really struggled to figure out how to use the technology. The program wasn't well integrated into the regular classwork. The "protocols" for use, carefully constructed by the developers, weren't followed because, as every teacher knows, stuff just happens. Students moved out of town; new students showed up. Teachers came; teachers went. The list goes on.

It was, in short, a pretty good reflection of how technology gets implemented in most classes -- hardly in the precise and careful way designed by those who have sweated over the program.

The trial was a flop -- not because the technology failed but because there was a mismatch between how the designers believed it should be used and how the teachers wound up using it. Was that the teachers' fault? Nope. Every day, in every class in the world, teachers come up with workarounds to cope with the unexpected. Most technology, however, isn't yet as resilient.

We drop test hardware before we send it into the field. Seems like it's time to start drop testing software programs before sending them into the classroom.