Qualitative Signals
The chart tells you what happened; five conversations tell you why. How to find the people, ask about the last time instead of the hypothetical, keep your solution out of the room, and turn what you hear into something the team can act on.
The ask, the obvious response, and how it goes wrong
Every lesson starts where the work starts: someone asked for something, and the first response that comes to mind has a problem.
The chart shows people leaving at the shipping step — how would you find out why without asking them a question that already contains your answer?
The funnel shows a clear drop at the shipping step. The PM says: "Let's send a survey asking if people would like more delivery options." The designer wants to redesign the step.
Run a survey or ask a few customers whether they would use faster delivery options. If most say yes, build them. Talking to users is research's job anyway.
"Would you like more delivery options?" gets a yes from almost everyone, because more options sounds good and costs the respondent nothing. The team builds three delivery tiers, and the drop at shipping does not change.
- "Would you like more delivery options?" gets a yes from almost everyone, because more options sounds good and costs the respondent nothing. The team builds three delivery tiers, and the drop at shipping does not change.
- The real reason turns out to be that the shipping cost appears for the first time at that step, and people leave because the total is higher than they expected. No question about delivery options could have found that.
- The survey reaches people who completed an order, because they have an account and an email. The people who left at shipping — the ones you wanted to hear from — are not in the sample.
- The engineer who could have spotted "the cost appears here for the first time" from the code was not in any conversation, because talking to users was assigned elsewhere.
What is really going on
- Numbers answer what and how much; conversations answer why. A funnel can say that people leave at shipping and how many; it cannot say what they were thinking when they left.
- People are poor predictors of their own future behaviour and good reporters of their recent past. "Would you use…?" invites a guess and a polite yes. "Tell me about the last time you bought something online and didn't finish" gets a real story.
- Leading questions carry the answer. "Was the shipping page confusing?" plants "confusing". People are cooperative in conversation and will agree with a plausible frame, especially when the interviewer obviously built the thing.
- Five conversations are not a sample for estimating a rate. They are enough to find reasons you had not thought of. When the same reason comes up unprompted in several of them, you have a hypothesis worth measuring.
- Qualitative and quantitative work in a loop: the chart points at where to ask, the conversations propose why, and a new measure or small test checks whether the why is common (Instrumentation First).
The chart says where, the conversation says why
The funnel is unambiguous about where people leave: shipping. It is silent about why. Every reason the team proposes — too few options, slow delivery, a confusing form — is a guess, and each guess leads to a different project. Five conversations with people who actually left are the fastest way to find out which guess is wrong, and sometimes to find a reason nobody proposed.
Not leading the witness
The interviewer's hypothesis leaks into questions in small ways: an adjective, an option list, a "would you". People are polite and cooperative, and when the person asking clearly built the page, they are more so. The fix is mechanical: ask about the past, ask open questions, and stay quiet longer than feels comfortable.
A customer who reached shipping last week and left has agreed to a fifteen-minute call. You suspect delivery options are the problem.
"We're thinking of adding express and next-day delivery. Was the shipping page confusing? Would more options have helped you finish your order?"
"Can you tell me about the last time you got to the shipping part of our checkout? What were you buying? … What did you expect to happen on that page? … And what did you do next?" — then silence, and "What made you decide that?" when they mention leaving.
A guide you can follow on a bad day
A short written guide is what keeps the conversation honest when you are nervous, excited about your idea, or running late. It is not a script to read verbatim; it is a list of what to ask and what not to say.
1## Conversation guide — left at shipping (15 min)2 3Before: know their session (what was in the cart, how far they got). Do not mention it first.4 51. Tell me about the last time you shopped with us. What were you looking for?62. Walk me through what happened once you went to check out.73. When you got to the delivery part — what did you expect to see?84. What happened next? What did you do instead?95. Is there anything else about that visit that stuck with you?10 11If they mention a reason: "Tell me more about that." Then wait.12 13Do not:14- describe anything we are planning to build15- ask "would you use / would you like"16- offer options in the question ("was it the price or the speed?")17- correct them or explain how the page is meant to work18 19After: notes within the hour. Left column what they said, right column what I think it means.The "Do not" list is the valuable half. Each item is a way the interviewer's idea gets into the answer.
Reporting five conversations honestly
The output of five conversations is a small number of reasons, each with how many people raised it unprompted and one quote. It is not a percentage, and it should not become one on a slide.
80% of users want shipping costs shown earlier. Recommend redesign of the cart.
Four of five people who left at shipping brought up, without prompting, that the total went up when shipping was added ("I thought it was free until the last page"). None mentioned delivery speed. Next: count how often the total rises at the shipping step, and test showing estimated shipping in the cart.The statistic claims a precision five conversations cannot have and skips straight to a solution. The finding says what was heard, how it was heard, what was not heard, and how to check it — so the team can trust it the right amount.
How to do it
Most important first.
- Start from the chart: know exactly which behaviour you are trying to explain — "left at the shipping step" — before talking to anyone.
- Recruit the people who did the thing, not the people who are easy to reach: recent sessions that reached shipping and left, via a short on-site prompt, a follow-up to logged-in users who abandoned, or support contacts about shipping.
- Ask about the last specific time, in their words: what they were buying, what they expected, what happened at the step, what they did instead. Then ask to be shown, if they can.
- Keep your hypothesis and your solution out of the room. No "would you like…", no describing the fix. When you want to test a reason, ask an open question that allows it without naming it.
- Write notes the same day, separating what they said from what you think it means. After five, look for reasons that came up unprompted more than once.
- Turn the finding into a measure or a small test: if the reason is "surprised by shipping cost", count how often the total rises at the shipping step, and try showing the cost earlier (The First Version That Teaches You Something).
How to explain the decision
The sentences, the order, and what to lead with — for someone who did not make the call.
- Lead with the pattern and how you found it, not the anecdote: "In five conversations with people who left at shipping, four brought up the total going up when shipping was added — none of them mentioned delivery speed."
- Say what it is and is not evidence of: "That doesn't tell us how common it is. It tells us a reason we hadn't considered, and it's specific enough to check."
- Propose the check and the smallest fix: "We can count how often the total rises at the shipping step today. If it's common, showing estimated shipping in the cart is a small change we can test."
- Use one quote, well chosen, to make it real: a customer saying "I thought it was free until the last page" does more in a review than a paragraph of summary.
- "Five people isn't statistically significant." Correct, and we are not measuring anything with them. We are looking for reasons we had not thought of; the counting comes afterwards.
- "Engineers shouldn't be talking to customers." Engineers shouldn't be selling to customers or promising features. Listening, with a script that forbids pitching, is where they spot what only the code explains.
- "We already know why — it's delivery options." Then the conversations will confirm it quickly and cheaply. If four of five bring up something else unprompted, we have saved the delivery-tiers project.
What can go wrong
- Leading the witness: every conversation confirms the hypothesis the interviewer came in with, because the questions contained it.
- Pitching instead of listening: the engineer describes the planned fix, the user says it sounds great, and it is recorded as validation.
- Treating five conversations as a vote: "three out of five wanted X" becomes a 60% statistic in a slide.
- Talking only to fans — power users, friendly customers, colleagues — whose problems are not the ones in the funnel.
- Over-correcting: refusing to act on anything qualitative until it is proven at scale, and missing obvious fixes that three people described identically.
- "Qualitative is for when you don't have data." It is for the why, which data does not contain at any volume.
- "Just ask users what they want." Users are experts on their problems and their recent experiences, not on your solution space. Ask about what happened, not what to build.
- "Surveys are the scalable version of conversations." Surveys are good at counting answers to questions you already know how to ask. Conversations are how you find the questions.
Knowing whether it worked
- The team can name the last time a conversation changed what they built, and what the chart looked like before and after.
- Conversation notes live somewhere the whole team can read, and engineers are in some of the conversations, not just reading summaries.
- The reasons found in conversations show up as new events or tiles within weeks, and the numbers either confirm them or kill them.
- Support contacts about the step drop after the fix informed by the conversations.
- At small scale, the team can talk to most users directly and conversations may be the primary evidence. At 10x users, they become the way to explain what the numbers show, not the way to measure it.
- At 10x team size, a research function may own recruitment and methods. The engineer's job becomes showing up to some of the sessions and bringing the questions only they would think of.
- In B2B, each conversation is with a customer who may also be an account; what you say can be heard as a commitment. The rules about not pitching become stricter, not looser.
- Conversations are slow to arrange and hard to schedule with the people you most need — the ones who left. Recruiting takes longer than talking.
- They are easy to do badly, and bad ones produce confident wrong answers, which are worse than no answers.
- Five conversations cannot tell you how common something is. Acting on them without a follow-up measure risks fixing a vivid minority problem.
Where this applies
Product advice is context-sensitive. These labels say what each claim is specific to, and where a different stage, team or product would differ.
- GENERALAsking about past behaviour rather than hypothetical preference applies to any conversation with users. It differs for expert B2B users configuring a tool, where asking about their workflow in detail is also asking about their preferences, and they know it.
- TEAM-SPECIFICWith a research function, recruitment and methods are theirs and engineers join sessions. Without one, the engineer or PM runs them directly — and should use a written guide, because the default instinct is to lead.
- PRODUCT-SPECIFICIn a consumer store, the people who left are anonymous and hard to reach; on-site prompts and support contacts are the realistic route. In B2B, every customer has a name and an account manager, so reaching them is easy and pitching by accident is the larger risk.