Quantitative UX Research, When to Measure and When to Ask Why

Which quantitative UX research methods fit which question?

Quantitative UX research methods include surveys, product analytics, A/B testing, quantitative usability testing, tree testing and first-click testing, and each one counts how many users do or prefer one thing.

UX research methods as a whole split by data type, qualitative or quantitative, and by data source, attitudinal or behavioral, and the full map of UX research methods places the qualitative ones too. Much UX quantitative research reuses data you already hold, such as web analytics in Google Analytics, while the rest needs recruited participants or live traffic; eyetracking also counts but needs special equipment.

The table compares seven methods, six of them from NN/g's overview of quantitative user-research methodologies, by question, typical sample, the metric each returns and whether it needs a live product.

What is quantitative usability testing?

Quantitative usability testing has dozens of participants attempt the same tasks, so the team can report task success rate, time on task and errors with a margin of error.

NN/g describes it as close to a controlled experiment built to capture metrics, while a qualitative test sets out to find interface problems, so tasks and structure differ. Repeating the same tasks release after release is usability benchmarking. Our usability testing services have every participant answer the System Usability Scale and the Single Ease Question (SEQ), and run unmoderated studies with 20–50 participants in Maze or UserTesting. Give each task one clear success state before a usability benchmark starts.

What are 5 examples of quantitative UX research?

Five common examples are a survey, a funnel report, an A/B test, a usability benchmark and a tree test, shown here on the illustrative shift-swap feature.

• Survey: five questions to every account admin about how easily their staff change shifts.
• Funnel report: how many sessions open the swap screen, and how many send a request.
• A/B test: two button placements, compared on swap requests per active user.
• Usability benchmark: 40 recruited staff attempt the swap task, scored on success and time.
• Tree test: where 30–50 participants look for shift swapping in a text-only menu.

Run the one whose number would change your next design decision.

Which qualitative UX research methods explain each number?

Qualitative UX research methods, such as user interviews, contextual inquiry, moderated usability testing and diary studies, each explain a specific kind of number.

Send the method to the users behind the number: a funnel drop-off, for example, calls for watching recorded sessions, then interviewing people who left at that step.

When the number comes from work away from the screen, such as staff who arrange swaps on a break-room whiteboard, contextual inquiry shows the real routine. A retention rate that sags after the first month calls for a diary study, in which the same participants log each time they use or skip the feature over several weeks. Pin each metric on your dashboard to the method that would explain it.

Which UX metrics does quantitative UX research track?

Quantitative research in UX measures behavior with UX metrics such as task success rate, time on task and conversion rate, and attitudes with SUS scores and Net Promoter Score.

The scorecard lists seven metrics with what each measures, its scale and how to read it.

Two of these are misread most. MeasuringU's Jeff Sauro reports that the average SUS score across 500 studies is 68, the 50th percentile, so 68 means average, not 68% or a pass; a score above 68 is above average. Task times skew right, and MeasuringU's analysis of average task times found the geometric mean less biased than the median in samples under 25.

For product-level tracking, the HEART framework from Kerry Rodden, Hilary Hutchinson and Xin Fu of Google, presented at the 2010 Conference on Human Factors in Computing Systems (CHI), names five categories: happiness, engagement, adoption, retention and task success. It ties each category to goals, signals and metrics. Retention shows up in real product results too: our TradeZella trading journal case study reports gains in user interaction and user retention after its dashboard revamp.

Pick one primary metric per research question and record its baseline before the change ships. In our work, the usability testing service scores SUS and SEQ for every participant, and our research program's card sorts and tree tests report grouping and findability scores.

How many participants does quantitative UX research need?

Quantitative UX research needs about 40 participants for many studies, per Nielsen Norman Group, against 5 per round of qualitative usability testing, because a measurement needs a small margin of error.

NN/g's 40-participant guideline rests on stated assumptions: a binary metric such as task success, a 15% margin of error, so the true score sits within 15 points of the observed one, and a 95% confidence level. That 15% is the precision target; the confidence level says how sure you are that the true value sits inside it. Together they set a confidence interval, the range the true value probably falls in, and the margin of error is half its width.

The ruler compares the worst-case margin of error for a task success rate at five sample sizes. We computed it with the normal approximation, 1.96 × √(0.25 ÷ n), widest at a 50% success rate; below 40 participants the approximation is rough, so read those rows as rough guides.

Other methods size differently: card sorts and tree tests run with 30–50 participants in our program, and surveys are sized by the margin you need. For A/B testing, the traffic needed is enough live visitors to reach statistical significance, a result unlikely to come from chance alone, which many low-traffic business-to-business (B2B) products do not reach in a useful time.

Count each user group separately, because admins and staff each need a representative sample. Software as a service (SaaS) products make this concrete, and our B2B SaaS design research covers administrators, end users and security reviewers separately. That recruiting effort comes first: in our program, participant recruitment takes 3–6 days of screening through a recruitment panel or your customer list. Count the matched users you can reach in each group before you promise anyone a percentage.

When to use quantitative UX research instead of qualitative

Choose quantitative studies when the question asks how many, how often or which version wins and you can reach enough users; choose qualitative UX research for why and how.

Not yet knowing what to count is the other reason to start qualitative. The decision tree routes a study through four questions, in this order.

Product stage shifts the odds: discovery leans qualitative, usability benchmarks need a working prototype or live product, and A/B tests need live traffic after launch, which mirrors the split between generative and evaluative research. A low-traffic B2B product still gets numbers without an A/B test; the FAQs list three ways.

The question word picks the method, as three illustrative rewrites from the scheduling app show:

• "Do staff like the new swap screen?" becomes "How many staff send a swap request unaided?", a usability benchmark.
• "Why do managers reject swap requests?" stays a why question for user interviews.
• "Is the button in the right place?" becomes "Which of two placements gets more requests?", an A/B test if traffic allows.

Rewrite your next study request around its question word before you choose a method.

How to combine qualitative and quantitative UX research

You combine qualitative and quantitative UX research with one of three mixed methods research designs: measure then explain, explore then count, or run both at once and compare.

The designs come from John Creswell and Vicki Plano Clark's mixed methods work, and research centers such as Harvard Catalyst teach the basic mixed methods designs, adapted from Creswell, where the convergent design appears as a convergent parallel design.

• Explanatory sequential design: quantitative first, then qualitative to explain the results; use it when a number moved and nobody knows why.
• Exploratory sequential design: qualitative first, then a quantitative instrument, such as a survey that tests a hypothesis from interview themes; use it when you do not know what to count.
• Convergent design: both at once, compared at the end; use it when time is short and both kinds of data exist.

Triangulation is the payoff: when you triangulate, two kinds of evidence that agree make a finding safer to act on. In the illustrative scheduling app, the explanatory order would read analytics for how many staff ever open the swap screen, interview staff who never use it to separate those who missed it from those who swap by text message, then measure the redesigned flow with a 40-person usability benchmark.

Our program follows the exploratory order: generative fieldwork (7–15 days) of user interviews and contextual inquiry, then evaluative studies (4–8 days) in which card sorts and tree tests with 30–50 participants check the early information architecture. Our guide on how many participants a card sort needs explains that range.

When a metric moved and nobody can say why, the explanatory follow-up is interviews and contextual inquiry with the users behind it, the fieldwork step of our program. The price of a 4–8 week research program turns on user groups, participants, methods and recruitment difficulty, so count your user groups first.

Which quantitative UX research traps create false certainty?

Quant UX research creates false certainty when a measurement looks precise but is not, as with an A/B test stopped at its first statistically significant result.

Each trap has a fix you can set before the study runs.

• Peeking at an A/B test. Stopping when a result first crosses statistical significance inflates false wins, so fix the sample size or duration up front.
• Statistical without practical significance. A real but tiny lift is not worth shipping, so set the smallest change that matters before reading results.
• Novelty effect. Returning users click anything new for a while, so run the test until that fades.
• Double-barreled survey questions. "How easy and fast was swapping?" asks two things, so ask one neutral question per item.
• Averaging skewed task times. A few slow participants drag the mean up, so report the geometric mean or median.
• A research panel that is not your users. People who never schedule shifts rate the task differently, so screen on behavior.
• Renamed analytics events. A renamed event looks like a drop in use, so check tracking before reading a trend.
• Net Promoter Score (NPS) read as usability. A loyalty score cannot locate a usability problem, so pair it with a task metric.

A list does not replace a second reader, so have someone outside the study review its plan before the first participant joins.

Which roles run quantitative and qualitative UX research?

Teams with steady traffic and an analyst can run quantitative research in UX in-house; explaining why a number moved, and recruiting the users behind it, calls for a qualitative researcher.

Three roles cover the work:

• Product analyst or data scientist: analytics dashboards and A/B tests on live traffic.
• Quant UX researcher: surveys, usability benchmarks and experiment design centered on users.
• Qualitative researcher: user interviews, contextual inquiry and moderated usability tests.

The missing role decides what to bring in. The program fit for a team with numbers but no reasons is our research program, which includes participant recruitment through a recruitment panel or your customer list, user interviews and contextual inquiry, card sorts and tree tests with 30–50 participants in Optimal Workshop, and synthesis in Dovetail ending in a 90-minute readout. One sign you need it is feature adoption that stays low while analytics show usage but not the reason.

The two reports from the start stop clashing once each answers its own question, and neither says how many new staff get lost, which is the next study. If a number on your dashboard moved last quarter and nobody can say why, note the date it moved and the user group it covers. With those two facts in hand, send us the metric that moved while the change is recent enough for the users behind it to remember what happened. Every project begins with a free consultation and a fixed-scope proposal.

FAQs