Quant+Qual = Agile User Research
WEVO’s CEO & Co-founder Nitzan Shaer was recently a guest on the Data Gurus podcast. He and host Sima Vasa discussed several topics including how WEVO’s unique combination of qualitative and quantitative data provides rapid insights that anyone within an organization can leverage to validate new concepts and ideas within an agile research process.
Below is a transcript of their conversation, edited for readability. Listen to the complete episode here.
Sima Vasa: Welcome to another episode of Data Gurus. I’m excited to welcome Nitzan Shaer, the co-founder and CEO at WEVO. I’m so glad that you are here. As I often interview people in the core analytics space, it’s always exciting to understand what’s happening in adjacent spaces. Your company straddles multiple segments of the industry. Before we get into that, share with our listeners a little bit about your background and how you founded WEVO.
Nitzan Shaer: I’m more than happy to. I think some of the biggest innovations come when you bring together ideas from different industries to create completely new opportunities and disrupt the landscape of opportunities in a certain space. That’s very much how our customers think about the WEVO solution.
WEVO is the fourth company that I’m either starting or joining early on. I just love the thrill of joining a team of like-minded people early on and swinging at the fences and trying to build something where nothing existed before. I joined early on with the mobile team at Skype, and we sold the company successfully. I did that two more times before starting WEVO, always at companies in the space of trying to help consumers solve a very big problem.
An Unmet Demand for Fast, Affordable Customer Insights
The context for WEVO was really formulated while I was at Skype. We disrupted an industry that hadn’t been changed for probably dozens of years — international calling. It used to be about $10 to call internationally per minute. I remember my parents would gather all the kids around to speak to Grandma, and we all had to be at the phone at a scheduled time. When you reduce $10 a minute to $0.02 or to completely free, then suddenly people can speak unlimitedly.
As a result, the amount of international calling minutes skyrocketed. I started thinking about other industries where there was unmet demand for something faster and cheaper. One of the problems that we faced at Skype was that it was just too expensive to run focus groups and usability studies that we needed to understand more about our customers. We were selling to 147 countries and territories around the world with different use cases and different age groups. It was a nightmare to collect user research, to know what products to build, and what features to improve. It’s so costly to build a product and then A/B test it. We founded WEVO because we really wanted to listen to our customers more, and there was an opportunity to reduce costs of doing so dramatically. We now take the process that’s about 200 hours to run a proper usability study or focus group and reduce it to half an hour.
Understanding the “Why” of Customer Behavior
Sima Vasa: How does WEVO work?
Nitzan Shaer: Fundamentally, WEVO is a tool to listen to your customers and hear what their needs and challenges are with your digital user experience. It provides you with validated insights on what the hurdles are as well as which things are working really well. Instead of just answering the question of ‘where are the problems,’ we answer the question of, ‘why are people experiencing a challenge in your user experience,’ thus enabling your team to generate a better user experience. We also enable our customers to validate their ideas. If you have a new concept you want to try out, but your organization doesn’t have the time or money to build every single idea, you could test those ideas with WEVO. We’ll tell you if those ideas are better than the existing solution. If you have five ideas, WEVO will tell you which one is going to do best in the market. And we do this with the help of human-augmented AI specifically.
Sima Vasa: Tell us a little bit about how that actually happens with WEVO. What’s going on in the background?
Nitzan Shaer: In traditional research you’d have to come up with a plan, select the panel, interview the panel, and analyze the feedback. You’d have to have experience in user research to do it properly. With WEVO, all of that goes away. The user only needs to define the target audience that they’re focused on and identify the experience that they are trying to optimize. It could be a landing page. It could be a set of three steps in an online process. It could be a part of a mobile app. It could even extend across an entire customer journey. Some of our customers, Procter & Gamble, for instance, test an experience starting with their websites through the fulfillment on Amazon.
Effortless Insights with Human-Augmented AI
You define the experience. You define the target audience. That’s all you have to do. It takes less than 10 minutes. Then WEVO does all the heavy lifting. We have a global panel of 30 million people. We identify a lookalike audience, auto-generate a survey for those people, watch as they go through the experience, and ask them a whole lot of quantitative and qualitative questions. Then we have AI that goes through those results. First, it cleanses the data. Then it themes together the different ideas and looks at them through multiple, benchmarked lenses. When that process is complete, you get a report with all the information you need to pinpoint why people aren’t engaging. If you launched multiple experiences, it’ll also tell you which is the best. And finally, it’ll tell you if there are any gaps between expectations people had and the reality of the experience itself. So you can identify things that aren’t on the page or on the experience.
Sima Vasa: It’s really exciting. At what point is there human intervention in that process?
Nitzan Shaer: That’s an excellent question. There are two levels. First, there are the participants themselves that we’re following, and what it is that they’re doing. The second is in the ultimate analysis of the Key Findings, as we call them. Most of the work of the theming, of the identifying and prioritizing and benchmarking is done by machine. But we do have analysts who look at all the insights. They specialize in conversion and engagement and success of experiences. They create a one page report at the end that summarizes everything. We trust the AI, but we are big believers in human-augmented AI, which is a partnership between human insights and the rigor of a machine.
Three Lenses for a Holistic View of the Customer Experience
Sima Vasa: Very cool. Tell me about the lenses that WEVO uses to analyze insights.
Nitzan Shaer: When we started WEVO, we brought together a group of thought leaders in the UX, usability, CX, and design space. And we asked them, how do you go about evaluating if a user experience is good or has room for improvement? Each one has their own set of tools. So through a process that started about four years ago, that we continuously refine and compare to how people behave in the real world, we’ve come up with three primary ways to look at each user experience.
The first lens is geographical. What is happening on the pages of a user experience? What are the areas that are roadblocks to conversion, and what are the areas that are actually helping the engagement? It looks a little bit like a traditional heat map, but it’s very, very different. We call it a sentiment map. The WEVO sentiment map shows you, by intensity of color, the biggest problems in the experience and what are the biggest successes. A problem could be an image that is offensive to people. It could be a headline that is confusing. We identify from an intensity perspective, what are those areas that are highest priority to be fixed? And because everything we do is quant and qual, we also explain why it is a problem.
Sima Vasa: So then you ask some sort of sentiment questions related to that?
Nitzan Shaer: That’s right. We call that combination of quant and qual “qualt”. Everything we do is qualt because we believe that in order to save time for our users, we have to point them in the right direction. What’s the most important thing? Then qualitatively, we explain what exactly is the problem. So many systems today lack that partnership between quant and qual. You can run A/B testing as much as you want. A/B testing is important, but it tells you nothing about what the problem is. A did better than B, but why, and what can you do to create C that’s twice as good as A.
The second is the lens of diagnostic drivers, the five main areas that we look at in order to understand how the experience is doing from a benchmark perspective. It looks at the gestalt of the whole experience together. We look at clarity. Do people understand what’s on the page? We look at appeal. Is the product or offering appealing to them? We look at credibility. Do they believe the company can deliver on the promises that it’s making through this experience? We look at relevance, and we look at overall experience. We do this with benchmarking to the industry and to the product. This is really important, because we can tell you how you’re doing against the benchmark. Are you above or below? What do you need to fix and why?
Why Benchmarking Matters
Sima Vasa: Do you have benchmarks by industry? Because I would imagine consumers tolerate different experiences depending on where it fits into their life. For example, healthcare is a durable industry. It’s needed to live and a consumer might have a higher tolerance for a less enjoyable experience versus a beverage company.
Nitzan Shaer: That’s right. I’ll even build on what you said. There’s a difference by industry, and there’s even a difference by product in the industry. I treat the experience of choosing a credit card differently from choosing a retirement plan. WEVO recognizes that and has benchmarks built per product, per industry. We’re building our database all the time. There’s no technical integration needed with WEVO; we can WEVO any website on the planet. We are continuously WEVO-ing more and more experiences, and expanding our benchmarking. It’s evolving all the time as customer expectations are changing.
Sima Vasa: That relative understanding for a client is so important. You could get good, bad or indifferent, but what does that really mean relative to competition or to other players in the industry?
Nitzan Shaer: We all want to know if we are in the top 10th percentile and the margin to improve is minimal, or we at the bottom 10th percentile and there’s still tons of room to improve? It’s really important to know that, so we can focus our efforts.
The third lens is about expectations versus reality. We ask people what their hopes and concerns are when considering purchasing this product or service, even before they’ve seen the site. Then once they’ve experienced it, we ask them whether their expectation was met. I can’t emphasize how important human-augmented AI is in this process because traditional user research would say, out of the following three things, what are most important to you? Well, what happens if you got those things wrong? What happens if those things changed since the last time you created that model of personas or priorities? They change all the time.
WEVO doesn’t ask it that way. We don’t assume that we know the answer to what things are important. We ask an open-ended question. In real time, the AI themes the responses with a well-understood algorithm. Then we ask, relative to this hope and concern that you expressed, to what extent was it met in the product? That helps our customers understand where they have to focus.
Sima Vasa: Exciting. Give us an idea of how long a project takes?
Nitzan Shaer: The time it takes from the user is about 30 minutes: 10 minutes to get a project started and 20 minutes to review the results. And you don’t have to prepare any PowerPoints. It’s all ready to be shared. There’s a sharing button that you can share with your colleagues. They don’t even have to log in. The time it takes to collect these insights depends on your target audience, it can be anywhere from three to 10 days.
The Disruptive Power of WEVO
Sima Vasa: Who in the organization gravitates to WEVO’s solution?
NItzan Shaer: Currently we’re seeing a lot of adoption with UX researchers, designers, marketers, and product managers. We’re seeing engineering teams starting to use it as well. Here’s a very important thing: It used to be that you need a research group. You needed to reach out to the research group and ask them to test with your users. They would put you in the queue and you’d wait a month until they get to you in the queue, and they take another month or two to execute it. Then you’d get the results. Then you’d schedule a PowerPoint presentation.
WEVO compresses that time and makes it such that anybody in the organization, if they’re an engineer, a designer, a UX researcher, can push a button, no expertise needed. In three to 10 days, you’re going to get feedback and you don’t have to invest tons of time in it. It’s validated. So you don’t have to do multiple iterations on this. So the learning of the organization starts to accelerate. The ability of the organization to take bigger risks because now they can test everything, accelerates. And that’s the fundamental disruptive power of WEVO.