Skip to Main Content

This blog post contains spoilers for “Avengers: Infinity War.”

A big part of my job here at Civis is to test the effectiveness of advertising. I often find myself hypothesizing about advertisements I encounter in my day-to-day life watching TV or scrolling through Instagram. A few weeks ago, I saw an ad online reminding me to pre-order tickets to “Avengers: Endgame.” I texted my favorite movie companion (my fiancé MJ) and told him I would buy tickets. I didn’t even ask if he wanted to see it, I already knew. We’ve both seen every Marvel movie, some of them multiple times, and MJ just finished rewatching every Marvel Cinematic Universe (MCU) movie in order.

This got me wondering: for folks who’ve already seen Marvel movies like me and MJ, does Marvel even need to advertise? In my mind, I’ve already converted. “Thor Ragnarok,” “Black Panther,” “Captain America: Winter Soldier,” and “Captain Marvel” (to name my favorites) were all the branded messaging I needed to get super hype about seeing “Endgame!”

Because I work at Civis, I didn’t need to leave these questions unanswered. So, I generated some testable hypotheses:

  • Awareness for the film is already so high, the trailers will have no effect on awareness.
  • Those who have already seen MCU films have already made up their mind to see “Endgame,” advertising will not have an effect on their likelihood.
  • Those who have not seen an MCU movie will be unmoved by advertisements for “Endgame.”

I picked three videos from the Marvel YouTube page: Honor TV SpotBig Game TV Spot, and the most recent full theatrical trailer.

Then I turned to Creative Focus, Civis’s application to quickly test text, video or display advertising with methodological rigor. It took me about five minutes to set up my test with our user-friendly application and get my test out into the field. From there, Creative Focus kicks off a randomized control trial survey (which we data nerds consider the most scientific), where a control group sees no trailer, and, in my case, three treatment groups see one trailer each. The system runs statistical models on the dataset, which shows the average treatment effects for the population (all US adults in this case) and specific subgroups (I was particularly interested in whether or not someone has already seen a Marvel movie).

Spoiler Alert: Much like Tony Stark in “Avengers: Age of Ultron,” I was very wrong. I love when science dispels bad assumptions, especially when they’re my own! Let’s go hypothesis by hypothesis and see just how wrong I was.

Hypothesis 1: Creatives will not affect awareness.

How much awareness effects backlash or persuasion

All of the trailers tested increased awareness, though the Honor spot was least effective. The Big Game spot increased awareness by five percentage points and has near-zero probability of backlash (i.e. which would make someone less likely to see the movie). The full trailer did not perform best, which is atypical in our experience — usually the longer the video, the more persuasive it is.

Hypothesis 2: Marvel movie viewers will not be persuaded by the ads, because they already have high intent to see it in theaters.