WEVO’S Jenni Bruckman, VP of Customer Success & Strategic Partnerships, recently joined the expert trio of Moe Kiss (Data & Analytics, Canva), Michael Helbling (Owner, AJL Analytics), and Tim Wilson (Director of Analytics, Search Discovery) on the Analytics Power Hour Podcast.
They took a deep dive into one of the hottest digital analytics topics of the day: the vast and varied world of qualitative research and how it is the perfect partner to quantitative data.
Jenni’s primary focus has always been to drive experience optimization forward. However, the demand for UX research — and the ability to scale it in a way that makes insights more accessible across cross-functional teams — has never been greater.
Listen to the full podcast here and read on for key takeaways.
Demystifying Qualitative Data
“You have your UX design, your development team, your project management, your strategic direction, and then you’ve got your analytics. What was always missing was being able to answer the ‘why’. . . The benefit of qualitative to me was answering that why. The challenge I saw over and over again was that qualitative is typically a very small sample size. To do it well, it’s typically very labor and expertise intensive. That’s very hard to scale.” – Jenni Bruckman, WEVO
Jenni believes there has to be a tried and true way to ask the right questions in the right way to the right audience. The ‘why’ needs to be reliably viewed and responsibly collected across a greater volume of people in order to be a reliable driver for the roadmap’s impact.
Qualitative: Truth vs. the Noise
Tim wanted to know how qualitative is consistently defined.
“The qualitative gets used and sometimes there’s the thought that – behavioral is quantitative, attitudinal is qualitative. Or if it’s objective, it’s quantitative. If it’s subjective, it’s qualitative. Is there a hard black line that defines whether something is, say, a Likert scale on a survey? They’re providing a hard number that we can perform math on, but it’s an attitudinal subjective measure. Is there a definition, or is it kind of a squishy threshold as to whether something is quantitative or qualitative?”
-Tim Wilson, Director of Analytics, Search Discovery
A consistent qualitative definition all comes down to sample size. Jenni believes you take a lot of that variability or ‘squishiness’ out when you collect the attitudinal data in a quantifiable way, at a great enough scale that you can say – here are the real signals of truth, and here is the noise.
“I think there are a variety of different methodologies and they all kind of have a strength for different use cases. Card sorting and tree testing, for example, are enormously valuable for early prototyping. It’s kind of an idea generation that asks – how are we architecting this experience and are we building it the right way?” – Jenni Bruckman, WEVO
But having a large sample size must be organized in a thoughtful way. You can do that by identifying some themes, then seeing which themes of data are statistically significant.
“I think the teams that are doing qual and quant really well are those that have super experts like a UX researcher embedded in a center of excellence, the same way that we saw that trend and optimization with data science and analytics early on. Then, they have a tool set that makes insights more democratized across the team, in a way where they trust what’s being collected as a reliable method.” – Jenni Bruckman, WEVO
She goes on to say it isn’t just about uncovering the way people like the design of the page or the experience. It’s about knowing whether they believe in the brand and the product. All of those layers are part of one holistic understanding for the customer.
Seeing the bigger picture
In digital analytics, we’re used to just answering whatever questions the business comes up with, even if they may not be great quality questions.
“It seems to me that because of the need to set up experiments and design them carefully, you have to be really careful about what questions you try to get answers to because they need to have a life cycle that lasts longer than the research itself. We do this in analytics all the time. By the time you do the analysis and come back with the answer, nobody cares anymore because it wasn’t actually that meaningful. (Shout out to all my overlooked analysts out there.) But how do you work with businesses and stakeholders to help them think bigger picture or ask better questions?
Michael Helbling, Owner, AJL Analytics
Jenni believes the best question businesses can start to ask is – what insight and data drove this priority?
“I think the best models are those that bring about a top-of-mind awareness. Once you become aware that you can test and iterate, that becomes the path forward. As soon as you understand that you need quant and qual, it’s impossible to separate them.” -Jenni Bruckman, WEVO
Jenni sees the organizations that are driving value the best through qual and quant are those that have an executive level stakeholder who is all about the customer.
“You need a maverick to start to push that narrative upwards. Find that really magnetic or charismatic stakeholder at the ground level who will say, we’re in this together and this is our vision – answering both the what and the why – and we’re going to solve that with data and insights to drive change. This creates that culture of asking what data and insight drove this priority over and over again, because it’s happening at multiple levels within the organization. That’s really where the organizational change comes from.” – Jenni Bruckman, WEVO
What organizational precursors are effective for qualitative research and experimentation?
Moe Kiss refers to the need to be highly cross-functional, with embedded resources across every product level, business unit, or marketing function. She believes there are advantages to everyone being able to responsibly gather insights with a tool that everybody depends and relies on.
“I think it’s about having a team that can not only build a vision, the next experiences, and hear a hypothesis or a problem, but also participate collectively and begin from validating. That’s going to be the foundation of the roadmap. If we can build that way, we’re going to build those cross-functional resources responsibly. That really is a game changer.” -Moe Kiss (Data & Analytics, Canva)
And Jenni couldn’t agree more.
To do that at scale across democratized teams, you need a lot of resources to be able to go a mile wide, but then the UX researchers need to go a mile or 30 miles deep — while knowing that they’re going a mile or 30 miles deep in the right direction. You need to know you’re putting the ladder against the right house before you start to paint.” -Jenni Bruckman, WEVO
Types of UX research
Formative and generative research is sorting the UX research expert-level stuff — giving you the ability to go that 10 miles deep.
Evaluative research is easier to tie to quant because it’s typically later stage, once the experience is prototype level or later. There’s value in performing evaluative research of competitors or existing flows that are out there beyond just your own when you’re creating something that new. Evaluative research can exist throughout the process, but typically generative, informative research falls into that expert-level user.
“Once you understand the mile wide, now you can spend more valuable time going into the 10 miles deep, otherwise you’re just kind of throwing darts and going 10 miles deep without any validation. That’s what broader evaluative research at scale can deliver. You get that validation along with a ton of discovery and light bulbs, then you know where to spend time and investment.” – Jenni Bruckman, WEVO
Neither quantitative or qualitative data on its own tells the whole story, but customer insight is the critical (often missing) element in the development of conversion test plans.
Qualitative research wasn’t as popular and critical as it is now, and I think that scale will take off with a trajectory that we can’t keep up with. With that, it drives the need for a solution where every team member can access insights that the experts trust. Then, you can answer the why in a much more powerful way.” – Jenni Bruckman, WEVO