← Back to the blog

How to write survey questions that don't tell you what you want to hear

A badly written survey is worse than no survey. Writing unbiased survey questions is what separates a real signal from a comfortable confirmation. Without one, you know that you don't know. With a biased one you have numbers, and numbers give you a confidence you haven't earned.

The problem is that the bias is almost never intentional. Nobody writes trick questions on purpose. You write questions that sound natural because you already have a hypothesis, and that hypothesis leaks into the vocabulary without you seeing it.

These are the four mistakes I've made most, and how I correct them.

1. The loaded question

The most common one, and the hardest to spot in your own writing.

❌ Does it bother you when shipping costs are hidden at checkout?

That question carries the verdict inside it. It says "hidden", it says "bother", and it hands the respondent a script to complete. They'll say yes almost every time, and that yes means nothing because they didn't choose it: you suggested it.

✅ Tell me about the last time you abandoned an online purchase. What happened?

This one mentions no costs, no shipping, no annoyance. If the cost problem is real, it will show up on its own. And when it shows up on its own, it's data.

That's exactly what happened in the research I did for Mexx: without asking about costs, 100% of respondents brought them up spontaneously as a reason to abandon. That number is worth something because nobody put it in their mouth.

2. The double-barrelled question

Two questions packed into one, with only one answer available.

❌ Was it easy and quick to find the product?

If it was easy but slow, what do they answer? Whatever they pick blends two things, and you'll read an average that describes nobody.

✅ How easy was it to find the product? ✅ How long did it feel like it took?

Practical rule: if there's an "and" in your question, stop and check whether it's two.

3. The hypothetical question

People are terrible at predicting their own behaviour. Not because they lie: because imagining a situation isn't being in it.

❌ Would you use a feature that saves your cart across devices?

Everyone says yes. Saying yes to something that doesn't exist is free. That question has a very low ceiling of usefulness and a high risk: it gives you a green light to build things nobody ends up using.

✅ Have you ever built a cart on your phone and ended up buying on a computer? How did you handle it?

The past is verifiable. The future is a fantasy shared between you and the respondent. At Mexx this question revealed that 80% researched on mobile and bought on desktop — behaviour that was already happening, not behaviour someone imagined they'd have.

4. The impossible-memory question

❌ In the last six months, how many times did you abandon a purchase?

Nobody knows that. They'll invent a rounded number, and you'll average it as if it were data.

✅ When was the last time it happened? Tell me what you remember about it.

One concrete, recent case is worth more than an invented statistic. And it comes with detail: the context, the brand, what they were doing. That's where findings come from.

Two rules of form for unbiased survey questions

Go from general to specific. If you open by asking about shipping costs, you've contaminated everything that follows: you already planted the topic. Open questions first, directed ones last.

Ask about behaviour, not opinion. "What do you think of X?" produces politeness. "What did you do last time?" produces data. People want to be nice to whoever is asking, and you can't switch that off by asking them to be honest.

How many people is enough

Fewer than you think. For qualitative research — understanding what happens and why — 8 to 12 people already show you the patterns that matter, in line with what Nielsen measured about sample sizes. At Mexx it was 10 people and 12 questions.

If what you want is to measure — what percentage of your base does something — that's a different discipline and needs real samples. Confusing the two is how you end up with headlines like "80% of users prefer X" resting on nine responses.

I use the survey to discover the what and the why, then confirm with usability testing, which shows what people do instead of what they say they do.

The final check before sending it

Read each question and ask yourself: can I guess what they'll answer?

If you can, the question is badly written. A good question is one that can surprise you. If none of your questions can surprise you, you're not researching: you're looking for permission.