I called eleven people who cancelled. The survey had been lying

Eleven people cancelled last quarter. I phoned all of them and nine picked up. Nine calls, twenty minutes each, three hours across a fortnight.

Nine sounds arbitrary. It isn't, and there's a formula that tells you roughly where to stop.

How many conversations you actually need

If a given problem has probability L of showing up in any one conversation, then after n conversations you'll have found this share of the distinct problems:

found  =  1 - (1 - L)^n

That's from Nielsen and Landauer's work on usability testing, and the L they measured across a pile of studies averaged about 0.31.

At L = 0.31, five people get you most of the way and the curve flattens hard after that.

Five conversations find about 84 percent of what's there. Nine find 96. The tenth call adds roughly one percent, which is why my nine wasn't diligence, it was just how many answered.

Read the L = 0.15 column too, because that's the honest case for anything subtle. Rare problems need many more conversations, and if you're hunting something that only bites one user in seven, five calls will probably miss it. Five is enough for the big stuff. It is not enough for everything, and people quote the five figure as though it were.

Why the survey was worse than useless

I had a churn survey before this. Multiple choice, four options, on the cancel screen. Sixty percent picked too expensive and I believed it for eight months.

Two problems, and the second is the fatal one.

Too expensive is the polite exit. It ends the conversation without insulting the person who built the thing. On the phone, only two of nine mentioned price at all.

And I wrote the options. Four choices, every one of them something I already suspected. A multiple choice survey cannot return an answer you didn't think of. Eight months of data confirmed my existing theories, and it could not physically have done anything else. That's not a survey. It's a mirror with a response rate.

Top is the survey. Bottom is the nine calls.

Four of eleven were an onboarding problem I could have fixed. Two were positioning. Five had nothing to do with the product.

The one who genuinely mentioned price meant something different by it. He reconciles twice a month, so paying monthly for something he touches 24 times a year felt wrong. That's not a pricing objection, it's a packaging one. He's back, on a per reconciliation plan. On the survey he'd have ticked too expensive and I'd have filed him with everyone else and considered dropping my price, which would have fixed nothing and cost me a third of my revenue.

The four who left without a word

This is the group that changed how I work.

Four people hit a wall in week two, decided it wasn't for them, and never sent a message. All four had spreadsheets with merged header cells, which we handled badly and now handle.

None were angry. That's what I'd got wrong. I'd been picturing churn as a decision, somebody weighing you up and concluding no. It's almost never that. It's somebody with ten minutes on a Tuesday who hits friction, puts it aside for later, and later never arrives.

The cancellation is paperwork on a decision made six weeks earlier.

Which means the cancel survey fires at the wrong moment by design. You're asking about a decision at the point where the person stopped thinking about you a month and a half ago. They aren't being evasive when they tick a box. They genuinely don't have that Tuesday in mind any more, and nothing on your form reminds them.

Running the calls

I asked one question. What were you trying to do on the day you signed up. Then I shut up.

That isn't a personality note, it's the method. My first two calls I explained things, described features they'd missed, offered to walk them through the import. Both produced nothing usable, because the moment you explain, the other person switches into being polite about your explanation. The seven calls where I said almost nothing produced all seven findings.

Getting people to answer was easier than expected. Six of nine picked up a cold call from an unknown number, and two told me afterwards they'd assumed it was about a refund.

What I do now is email first. One short paragraph saying I'm not trying to win them back and I'd like fifteen minutes to understand what went wrong. Say the words not trying to win you back, because that's exactly what they're bracing for. Offer a specific time rather than asking for one.

And don't offer them anything. I gave the first two a discount to return, both said no, and it soured the rest of both calls, because from that moment they were managing me instead of telling me things. A discount converts an interview into a negotiation and you can't run both.

What replaced the survey

Fewer answers, all of them usable.

One open text field, labelled tell me what went wrong, with a line saying I read every one of these personally, which I do. Response rate fell from 60 percent to about 35, and the 35 is worth incomparably more.

The personal line is doing real work. We tested the open field without it and got around 20 percent. People don't write paragraphs into a box they assume nobody opens, and they're usually right.

The change that mattered more than all of this

I stopped waiting for the cancel screen.

Anyone who's been active and then goes quiet for a fortnight gets a short email asking what they got stuck on. That catches the four who'd otherwise vanish silently, and it catches them while they still remember the friction.

About one in five replies. Worse than the survey on paper, far better in practice, for two reasons. They're still customers, so the answer can still change something. And they're describing a specific Tuesday rather than summarising a quarter, which is the difference between a usable answer and a category.

Roughly a third of the people who reply come back into active use with no intervention beyond me acknowledging the problem. I still find that slightly odd. My best guess is that being asked is itself the intervention, because it moves an abandoned task back onto a list.

If you do one thing

Call five people who left. Not a survey. A call.

Ask what they were trying to do on the day they signed up, rather than why they left. Five gets you 84 percent of what's there, it costs an afternoon, and you cannot unhear the answer once you have it.