Data-driven and Wholly Unscientific

How our modern tech and data led approach to marketing often doesn't represent anything close a scientific one.

We like to believe that modern marketing is more science than art through the complexity of our technology and abundance of data, but that’s simply not true in many cases.

In reality, we’re often ignoring - and even losing the skills for - the crucial first parts of what makes up the scientific method.

We conduct experiments, produce lots of data, make definitive claims of impact that we defend with our technology, talking of statistical significance, and yet can be claiming wildly incorrect things that wouldn’t even pass a test of logic.

Where did we go so far wrong?

The scientific method - applied to marketing

It’s an important starting point to actually understand that it means to approach something scientifically, because it’s not about how much data you bring to the table but rather how sound the hypothesis that leads into the experiment is.

Source: Britannica

Good hypotheses come from good questions informed by data and observations.

In marketing, that means an understanding of the buyer, the category, and the conditions surrounding a purchase.

How does someone buy a product like this?

How do the make purchasing and vendor decisions? Who’s involved? When do they get involved? What would change their mind on vendors?

When would things that cause that change that need to occur?

Good marketing hypotheses comes from market research - an area we’re sorely lacking in across industries.

But it’s more than just the available research that leads to bad hypotheses. We often apply poor logic about how marketing even works on audiences and products that resemble what we’re selling.

And that matters a lot when it’s how we’re accepting and pursuing targets that our jobs often depend on.

---

A B2B foundational hypothesis

Here’s a very typical scenario in B2B SaaS that I’ve seen and experienced countless times, and which I’ll bet would relate to a great many businesses:

1. The average sales cycle for the product is 90 days

2. Research says 70% of buying journey happens before the sales process

3. That would imply an average buying journey of about 300 days

4. 80% of typical deals are won by the preferred brand in mind at the beginning

5. 95% of deals are won by at least one of the brands in mind at the beginning

6. Marketing is asked to launch a campaign to create more pipeline this quarter

Before going any further, this set of variables should result in a set of foundational insights if we were to be truly scientific:

Any brand new potential buying processes that we could be considered in would likely not come to fruition for at least 210 days from now.

Therefore, to win incremental sales opportunities this quarter it will have to come from the 20% of deals that were already in process, and in which we were a considered vendor, or from the 5% of deals where we weren’t.

Before going any further, this is a pretty bleak starting point for our objective, especially when you half the odds even further with the reality that only about 50% of deals will go to any vendor.

But, this is the mandate being handed to many marketing teams all the same, and so it should lead to a series of initial questions:

• What sways people away from their initial consideration set?

• What types of companies or people are most easy swayed?

• How expensive or profitable is it to steal a late-stage deal versus working our way into new ones with other marketing approaches?

These, plus a great many more questions, should be asked before decisions are made. But it rarely works like that in marketing.

Much more often, it’s a matter of a leadership message of “figure it out” which is later met with blame for not magically doing so.

---

Working backwards from the conclusion

The foundational hypothesis, though, isn’t the only core problem with how we conduct our unscientific marketing.

The other big problem is that we begin with the final conclusion metrics and fit all hypotheses into it with the assumption that they will all show up that way if true.

A designed experiment is one where we ask the very simple question of:

What would logically happen, based on our understanding, if this marketing campaign were to work?

Sticking with the above B2B example, what would happen if we were to focus not on the current quarter impact but on getting into those day-one consideration sets? This is the basis of most of our brand investments, so it’s an important one to think through.

If we were to run a big brand campaign, what might logically occur that would indicate that something effective and incremental was happening as a result of our work?

What typically happens in B2B is that we are simply asked to draw a straight and measurable line from those efforts to pipeline and revenue outcomes, often through attribution tools.

Anything that doesn’t deliver pipeline this way wasn’t effective, we deem.

But this isn’t a designed experiment. This is retrofitting our marketing into an experiment designed for something entirely different.

Attribution is a tool that works really well for analyzing the paths in at the end of a journey, but it simply terrible at tracking the entire one and for a great many reasons. A truly scientific experiment would include pipeline, of course, but it would need to be thoughtfully considered.

That means thinking about what might happen in the near, medium, and long term that could indicate impact.

This doesn’t have to mean complex modeling and incrementality tools, by the way, but rather applying our understanding to what we can observe and measure:

• Might some of those deals we’re being considered in result in a higher win-rate to an observable degree - and could we ask it in the sales process?

• Might we observe more deal consideration than usual in the medium-term that could indicate a correlated lift to examine?

• Can we observe increased behavior indicative of more consideration for the medium-term, such as branded search volume?

• Can we survey those nowhere near in-market behavior to see if there are changed perceptions that can indicate long-term success?

This is far from a complete list, and yet is likely more grounded that most of the situations that marketers find themselves in when they’re given a metric to make it all show up in, and that’s the whole point.

---

Better insights and questions over better data

The good thing about science is that it works even if we can’t measure it.

The bad thing about modern marketing is that things often aren’t actually happening because of us despite our measurement ‘proving’ that it did.

Being data-driven can sound like a very impressive trait, especially with all the negative perceptions about marketing being the ‘arts & crafts department’ but having lots of data does not make one scientific.

And, it’s worth noting, that marketing itself is one of the soft sciences where even the best understanding can often result in an ability to ever be able to show clear, indisputable, repeatable experiments.

We’re dealing with the messy reality of memories, emotions, decision making and the noisy world that we all live in.

Perfect measurement is not really possible, but much better thinking behind our experiments absolutely is.