I remember pretty clearly the day that the February 2000 issue of Presentations Magazine arrived in our Kalamazoo, Michigan office. It contained the results from a Presentations/3M study on the effectiveness of multimedia presentations. Prior to this, I had heard rumblings of “data” from the 1980s studies but had never actually read those research reports. For the next few years, I confidently told audiences and clients that “multimedia presentations could increase comprehension 78% over static visuals.” Like many others, I was eager to use any data that seemed to validate the services we were offering to clients.

So what did this study really tell us about the effectiveness of PowerPoint multimedia presentations? Unfortunately, the answer is: Not much at all.

Background
Presentations Magazine funded this study in partnership with 3M. The researchers hired to design and conduct the investigation were two assistant professors at Portland State University, Hayward Andres and Candace Petersen. I have not been able to track down what Andres is doing these days, but as of January 2005, Petersen is Vice President of Corporate Strategy at InFocus. (Update: now COO of Learning.com)

The only source of information I currently have on this study comes from the February 2000 Presentations Magazine article written by Tad Simons. Tad joined Presentations (now defunct) while this study was underway. As far as I can tell, this research was never published in a peer-reviewed journal. If I can get my hands on a summary report or some of the raw study data, I will update this post with additional information and/or corrections as needed.

The study’s stated purpose was to compare the overall retention and comprehension effectiveness of presentations when information is delivered using text only, overhead transparencies or animated PowerPoint visuals. In addition, the researchers wanted to compare the degree of personal engagement felt by audience members as well as their perception of the presenter and presentation quality when the three different mediums were used.

Study Design
The study involved three different presentation scenarios. The first consisted of information on the workings of a photocopier. The second scenario presented information on the greenhouse effect. In these first two scenarios, three groups of 30 to 40 Portland State University students received the information as: 1) text only - no presenter; 2) presenter with overheads; or 3) presenter with PowerPoint visuals that included graphics and animation. The third scenario involved presenting information about two fictional banks with identical services. Two sessions were conducted, one comparing multimedia with text only, and the second comparing multimedia with overheads. The multimedia in this case consisted of an animated PowerPoint presentation that included a video testimonial.

Subjects in all scenarios filled out multiple-choice tests and short essay questions to measure recall of information as well as a survey using a 10-point Likert scale to assess the audience’s perception of their own engagement, presenter effectiveness and overall presentation quality. I do not have copies of the tests or surveys used.

Significant Design Problems
There are numerous flaws in the design of this study, but two in particular make it impossible to trust the results of this research. One of the main problems with this design is that it does not measure a change in information recall or comprehension. In other words, there is no possible way to tell if some people just knew more about photocopiers and greenhouse emissions than other people did to begin with. If people who knew more about these topics to begin with ended up in the multimedia group for example, the results would be significantly skewed in favour of multimedia presentations. Doug Vogel was careful to avoid this problem in his UM/3M study design. He administered a pre-test 2 weeks before the actual presentations were given and then measured the difference between the pre- and post-test data. This design flaw in the Presentations/3M study invalidates the recall and comprehension results from the first two scenarios.

In addition, the Presentations/3M researchers did not control for speaker variability. In fact, the text only group had no presenter involved at all. It is impossible to know if any difference between this group and the others was due to the media used, or the fact that a presenter was introduced into the equation. Vogel carefully controlled for the presenter by videotaping the presentation so each group saw the exact same person giving the exact same presentation. Only the support visuals were changed in his study. This design flaw also invalidates the results from the fictitious bank scenario. The Presentations/3M researchers claim to have trained the presenters to give the exact same presentation every time, but it is hard to imagine how this would be possible. There is no way to know if any results seen were due to the media or due to presenter differences.

Results
The Presentations/3M researchers reported an increase in fact recognition and comprehension when multimedia was used. Unfortunately, due to the problems outlined above, it is simply impossible to have confidence in any of the results reported for this investigation. The results from scenario 3 are interesting, where multimedia was compared to overhead transparencies to present information regarding the fictitious banks. Most of the audience (79%) selected the multimedia bank in this scenario. The audience generally found the multimedia presentation more credible, more professional and felt the multimedia bank offered better services and lower fees, even though the services and fees were identical to those presented with overhead transparencies. Unfortunately, without carefully controlling for presenter variability, it is hard to know what caused these differences. I also have questions regarding how well claims of preference by students in the bank scenario would mirror a real world business result. Researchers have argued for and against the use of students as surrogates for business people (Alpert, 1967 and Cunningham, 1974 for example). When the topic is bank services, and no real action is required, I am not sure a stated preference by students really indicates meaningful persuasion has occurred.

Discussion
The problems with this study come as no surprise to Tad Simons, editor of Presentations Magazine. In September of 2004, Tad published an informative article about the challenges inherent in studying the effectiveness of media. I would argue, however, that it is possible to design productive research programs in our field.

Well done studies should build on the work done by previous researchers. Study designs should be consistent whenever possible so that the results can be compared to previous work and our cumulative knowledge can grow. It is difficult to see how the Presentations/3M researchers could have carefully analysed even the two primary studies conducted in the 80’s by Oppenheim and Vogel. Vogel’s work, for example, provided a solid model of a well designed trial. If the Presentations/3M researchers had used Vogel’s concepts, their results might have added to our body of knowledge.

It is not our goal at Presentation Facts to simply criticize the work of others. Our goal is to try our best to determine which research is credible … and which research isn’t.
Our credibility as Presentation Professionals depends on knowing the difference.