This blog post contains spoilers for “Avengers: Infinity War.”
A big part of my job here at Civis is to test the effectiveness of advertising. I often find myself hypothesizing about advertisements I encounter in my day-to-day life watching TV or scrolling through Instagram. A few weeks ago, I saw an ad online reminding me to pre-order tickets to “Avengers: Endgame.” I texted my favorite movie companion (my fiancé MJ) and told him I would buy tickets. I didn’t even ask if he wanted to see it, I already knew. We’ve both seen every Marvel movie, some of them multiple times, and MJ just finished rewatching every Marvel Cinematic Universe (MCU) movie in order.
This got me wondering: for folks who’ve already seen Marvel movies like me and MJ, does Marvel even need to advertise? In my mind, I’ve already converted. “Thor Ragnarok,” “Black Panther,” “Captain America: Winter Soldier,” and “Captain Marvel” (to name my favorites) were all the branded messaging I needed to get super hype about seeing “Endgame!”
Because I work at Civis, I didn’t need to leave these questions unanswered. So, I generated some testable hypotheses:
Then I turned to Creative Focus, Civis’s application to quickly test text, video or display advertising with methodological rigor. It took me about five minutes to set up my test with our user-friendly application and get my test out into the field. From there, Creative Focus kicks off a randomized control trial survey (which we data nerds consider the most scientific), where a control group sees no trailer, and, in my case, three treatment groups see one trailer each. The system runs statistical models on the dataset, which shows the average treatment effects for the population (all US adults in this case) and specific subgroups (I was particularly interested in whether or not someone has already seen a Marvel movie).
Spoiler Alert: Much like Tony Stark in “Avengers: Age of Ultron,” I was very wrong. I love when science dispels bad assumptions, especially when they’re my own! Let’s go hypothesis by hypothesis and see just how wrong I was.
All of the trailers tested increased awareness, though the Honor spot was least effective. The Big Game spot increased awareness by five percentage points and has near-zero probability of backlash (i.e. which would make someone less likely to see the movie). The full trailer did not perform best, which is atypical in our experience — usually the longer the video, the more persuasive it is.