Research on Research:

Rival Platform and Qualtrics

The next generation of research is here, and we wanted to put the platforms to the test! To compare Rival to traditional survey platforms, Reach3 conducted a proof-of-concept test in June 2021 to evaluate the Rival platform performance against a more traditional survey software.

The Rival platform is mobile-first, with a messaging format that makes the survey experience feel more conversational, like texting or chatting with a friend instead of taking a test. The platform also allows for easy video submissions for richer qualitative data.

What did we learn?

Rival outperformed Qualtrics on survey experience, data quality, data richness, and the ability to collect videos. Video collection does come with a trade-off though – a lower response rate and slightly longer survey. We believe those trade-offs are a small price to pay for bringing the learnings to life through video.

Methodology:

Mobile-only A/B testing to compare the Rival platform to a traditional survey platform, Qualtrics.

minute survey

targeted completes per survey

? Exact same survey questions programmed on Rival and Qualtrics in an A/B testing format

? The Rival platform allows for video responses to open-ended questions

? All open-ended questions were required, allowing us to review them for quality control purposes

? Click-balanced incoming sample

? Participants routed at random to either the Rival or Qualtrics platform

? Ensures an apples-to-apples sample pool (any variances in the final dataset can be attributed directly to the participant experience and not to a difference in sample)

Key Metrics:

Survey Experience

Completion Rate

Data Quality

Data Richness

Survey Length

Video Responses

Winners on Key Metrics:

wins on:

Survey Experience
Data Quality
Data Richness
Video Responses

wins on:

Completion Rate
Survey Length

Detailed Findings

Survey Experience:

Participants overwhelmingly felt that the Rival platform was more fun and enjoyable than other surveys they’ve taken in the past, much more so than those who took the survey on Qualtrics.

Much more fun 59% 59%
36% 36%

Much more enjoyable 56% 56%
39% 39%

Honestly going through this survey, I was like wow! It is so much easier and not to mention it looks more like the 21st-century.

The message responses were cool! Other surveys don’t do that. Plus it feels like you’re communicating with someone.

I like the casualness of it more. It didn’t feel like all the other surveys I’ve taken before.

I felt like it was more personal. It seems to me like I was actually talking to someone rather than just answering questions on a survey.

It seems the survey is communicating with a real person not a bot.

This is much easier than most because it doesn’t need to continually click ‘Next.’ They automatically move to the next stream. Multiple choice questions are much easier.

I felt like it was more personal. It seems to me like I was actually talking to someone rather than just answering questions on a survey.

The interactive chat feel was great. The option for video instead of typing responses was even better.

Hear directly from participants on why they prefer Rival! ??

Completion Rate:

Rival completion rate

Qualtrics completion rate

Qualtrics performed better on completion rate, which is to be expected. Most drop-offs in the Rival platform fell at open-ended questions where participants were given the choice to submit video responses, likely due to bandwidth limitations and/or user error.

The Rival team is continually working on the platform, including streamlining the video submission process to reduce dropouts. Since this study was fielded, the Rival team has already added a variety of updates to the video upload experience for respondents, which should reduce technical barriers to completion and improve completion rates.

It should also be noted that not every open-ended question is an ideal candidate for a video selfie – the platform allowed a video response option for every open end in order to keep the A/B testing format, but in an actual study, a mix of text and video responses would reduce the survey length and improve completion rates.

Data Quality:

Rival pass rate

Qualtrics pass rate

Upon reviewing all responses for junk data, the final Rival platform completes were higher quality. Video response questions tend to weed out poor respondents who may otherwise get by with typing a junk response (i.e., a video response is a much harder quality check to pass than a typed response). In this regard, video submissions can help serve as a natural, more stringent quality check for responses while simultaneously bringing survey data to life with video.

Data Richness:

The Rival chat format encouraged richer qualitative data responses. Respondents on average wrote 5 or 6 more words per open-ended response on the Rival platform than Qualtrics.