Lessons learned from 17,000 user interviews

Okay, the title's a bit of a misnomer. I didn't conduct 17,000 user interviews myself, but rather for the customers of an AI tool I created called Resubscribe.ai.

I spent around six months working on the tool with two other co-creators, going from idea to launch to paying customers ($500 MRR) before inevitably shutting it down in late 2024 due to lack of product-market fit.

I'm no longer actively working on the project, but I wanted to share a few learnings from my experience.

What was the product

Resubscribe was a user feedback tool that allowed B2C founders to gather continuous qualitative feedback at scale. More concisely, it was an AI tool that conducted user interviews.

e.g. a B2C app founder might want to understand why users were deciding not to pay when presented with a paywall

As a previous app founder myself, this was scratching my own itch.

Why was this a pain point?

During the 0→1 phase, founders need to listen to their users in order to iterate towards product-market fit. However it's extremely difficult to get users to talk to you in B2C.

Think about it from your own perspective - when was the last time you saw an email in your inbox asking for feedback and you responded?

Often, B2C founders try to solve this problem by offering incentives like $100 gift cards to get users on the phone. This solution sometimes works, but it doesn't create a continuous discovery loop.

From our own experiences and market research, we believed this was a hair on fire problem.

Why did the product idea fail?

Ultimately, the product failed due to a lack of product-market fit with our ICP (Ideal Customer Profile) of early-stage founders.

This was a poor beachhead market to go after for a few reasons:

1. Many early-stage founders don't intuitively value user discovery. Just look at how vigorously accelerators like Y Combinator try to incept the idea "talk to your users" into their portfolio company's brains.
2. Some early-stage founders don't have the aforementioned pain point. They have direct connections with their early customers, even going so far as having cell phone numbers of their customers. This is far less common in B2C.

We felt that we'd seen enough signal of #1 and #2 to warrant abandoning the idea. I still believe this is an interesting problem area, but perhaps for a different set of customers.

Unintuitive learnings

Even though we failed to create a successful product, we did learn a bit about the psychology of user feedback in a B2C context.

Many early-stage founders don't understand The Mom Test

If you've never read The Mom Test, I would strongly recommend that you go check it out. It's tough to summarize its dense contents, but here's a brief summary:

"The Mom Test" teaches entrepreneurs how to conduct effective user interviews and why user discovery is important. The core principle is that you should ask questions so objective that even your mom couldn't lie to you with polite, misleading answers about your business idea.

Our hypothesis was that founders intuitively understood the teachings of The Mom Test.

However, when pitching our product, some founders would ask questions like "Well, we already have an open text box for feedback. why is your product better?". This surprised us.

As a follow up, we'd ask "How often do you conduct user interviews?". Commonly we would hear something like "probably 6 months ago".

For product teams that are launching consistently, this is way too infrequent. However, it wasn't our job to change opinions or move the market - our job was simply to understand it.

Alternative feedback channels were "good enough"

Another trend we saw from B2C founders was that they believed that their current feedback channels were good enough. Many of them were already:

• collecting feedback via an open text field in a form
• collecting feedback from support channels
• collecting feedback from review channels (app stores, etc)

We received a clear signal that they didn't value the additional channel we were providing.

Feedback from free users can be weird (and useless)

After we onboarded a few customers and started conducting feedback for them, we noticed something interesting.

You can reach a local maxima of feedback relatively quickly, especially for free users.

For one of our customers, we asked end users "why didn't you pay?". The vast majority of responses said something like 'it's too expensive' or "I don't use it enough".

In B2C, most users won't care enough to pay, so interviewing them doesn't often yield much valuable information.

People love to share feedback with AI

LLMs are great at affirmation and making people feel heard. In the graphic here, you can see:

• On average, 83% of users who sent one message, sent at least one other message
• The number of responses peaks at 5 messages. That's a lot in B2C where users notoriously don't engage!
  • (The big spike at the start was due to lots of garbage data from free users)

I'll leave you with this

More than ever, I believe in the importance of talking to your users.

If you haven't already, read The Mom Test!