Why Most Innovation Fails — And How Better Research Fixes It

Why Most Innovation Fails — And How Better Research Fixes It

Laptop and report paper in office wooden desk.

Innovation – an important and broad topic. Of course, you need to stay innovative as a company to stay competitive. But still, we see fewer and fewer real “innovations” in the market.

In a recent BCG survey, 83% of executives ranked innovation among their top three priorities, yet only 3% said their organization is fully ready to innovate.

McKinsey adds another layer: 84% of leaders believe innovation is critical for growth, but only 6% are satisfied with their own results.

Did we reach the top? Is there nothing left to innovate anymore?

I believe there are two core ingredients to innovation:

  1. A visionary mindset: the ability to look past “what is” and break free from our assumptions.
  2. Deep listening: the ability to really see the current “what is,” to connect deeply with barriers and struggles, and to understand what truly deserves to be solved.

And I believe that there is one big “innovation enemy”: pressure.

Innovation doesn’t happen by toggling a switch on or off. And it’s not even the path to finding a solution that takes the most time — it’s the path to understanding the problem. And that’s exactly where we give ourselves less and less time. Having access to an enormous amount of data (the internet) and speeding up how we process it (with AI) is actually not helping. On the contrary: we think we understand problems better and faster because we can get to answers quickly.

The Innovation Gap: Why Companies Struggle to Innovate

If you’re a manager, you probably know the situation: Your company wants to be more innovative. Your leadership team might even say: “This year, innovation is a priority.”

Innovation and team motivation
Designed by Freepik

So you start running brainstorm sessions, send out customer surveys, and maybe you’re organizing a hackathon or design thinking session. Weeks pass—a few “nice ideas,” little traction. You’re collecting ideas; they look good in presentations, but the roadmap doesn’t change.

Why does this keep happening? I see three main patterns with most of my clients:

  • Vague top-down mandate results in scattered effort. Saying that we want to be innovative as a company is not enough. There needs to be a strategy in place, a broader vision to set a focus for our efforts. Otherwise, we are just throwing darts in the dark.
  • Shallow inputs result in shallow bets. An essential ingredient for innovation is understanding your current situation – which means: your customers, your market, technology and developing trends. Simply running some quick interviews or sending out broad surveys will not provide the deeper insights you need – they will mostly just confirm what you already know.
  • Time pressure results in skipping problem definition. When we act under pressure, we want to see changes quickly. We want to “deliver”. The tendency is to grab the first idea that comes to mind, develop and ship it. But is it actually solving a problem? Will it drive innovation? In most cases I see customers shrug – because we didn’t take the time to understand the real need.

How Deep Research Actually Helps

If there’s one lever you can pull as a manager to drive more innovation, it’s how you do research. Done correctly, it can help to:

  • define a clear vision (of what “being innovative” should mean for your company)
  • understand the current problem (and therefore defining the focus of your team’s effort)
  • getting actionable feedback (before you invest too much time and effort into developing and shipping).

To be clear: “research” doesn’t only include understanding your customers. With research, you learn about your market & competitors, technology & trends, and, yes, also about your customers and their behaviour.

Designed by Freepik

This is not a new epiphany. Most teams already know they need customer insights.

A study by Kantar Worldpanel (2022) reports that 61% of new product launches fail within their first two years, mainly due to insufficient customer understanding. Academics across business disciplines also see customer insights for innovation as a high-priority research area.

The problem is how companies do research. The study “Customer insights for innovation” (which includes a survey of 305 managers) shows that the most commonly used research methods are:

  • Surveys, which provide answers to the questions you’ve already thought of.
  • Interviews, which capture what people say they do, but not what they actually do.
  • Data analytics, which show click-through rates, but not the emotions behind decisions.

None of these methods are “bad.” However, if this is all that’s being used, it creates a narrow view. They give you fragments of reality — enough to feel informed, but not enough to spark true innovation.

There are more research methods than just those three. In this study, methods were connected to the impact they have on innovation – and the result? Companies underuse the methods that deliver the deepest insights. These are either methods that require active customer involvement (like crowdsourcing or co-creation) and methods that require a deliberate insight-gathering effort (observing, testing, imagining).

Graph that shows prioritized underused research methods
Customer insights domains sorted by degree of underuse (N = 305). Note. The asterisks represent the p-values for t-tests comparing the difference in the mean scores between ‘should use’ and ‘current use’ (i.e., the Δ) for each domain of customer insights for innovation to zero (which means that managers perceive the ‘current use’ as equal to the normative ‘should use’): *** p < 0.01; ** p < 0.05; * p < 0.1; n.s. = non-significant. All t-tests are two-tailed (Image from original study)

In a correlation study, they further found out that especially crowdsourcing, imagining and co-creation have the biggest impact on innovation (yet are rarely used).

So if your research feels shallow, it’s not because you’re doing it wrong — it’s because you’re only using part of the toolbox.

From Insight to Innovation: a 4-Step Framework

So what can we do about it to drive more innovation?

Here is a simple, 4-step plan that you can apply to almost all projects:

1) Start by defining the decision, not the method

  • Write one sentence: “We must decide whether to ______ for ______ (segment) because ______ (risk/unknown).”
  • If you can’t write it, you’re not ready to research.

2) Diversify your research toolbox

  • Observe: shadow 3 users doing the task in their real environment (45–60 min each).
  • Test: put a rough prototype in front of 5 users (paper or clickable).
  • Co-create: run a 90-min session with customers + cross-functional team to sketch solutions.
  • Trend/competitor scan: collect 10 screenshots of how others solve the same job; annotate what’s working and why.

3) Turn data into insights

  • Synthesize on one page: Top 3 patterns, Top 3 friction points, Top 3 opportunities.
  • Add a “So what” next to each: what we’ll try, what we’ll stop, what we’ll watch.

4) Act, measure, iterate

  • Define one small shippable experiment you can accomplish in 2–4 weeks.
  • Define one success metric (outcome focused, what should change for your customers).
  • Review, keep/kill/iterate. Repeat.

Take Away

Innovation doesn’t start with ideas. It starts with understanding.
If you remember one thing, remember this: expand your research beyond interviews and surveys into observing, testing, and co-creating—then take action/ make a decision. That’s how insight becomes impact.


Over to you

If you want to make changes, here is a task for you to try:

Run a 5–3–1 Sprint in the next 14 days:

  • 5 real-world observations (shadow customers in context)
  • 3 basic prototype tests (paper or clickable)
  • 1 explicit decision (what we’ll build/stop next, with a metric)

This is small enough to do now, big enough to change the conversation.