Be data informed, not data driven

When I interviewed Jeff Sauer in 2022, his company was called Data Driven U. He'd found the name the proper way, with a naming book and a notebook full of options, and everyone agreed it was the winner. Then a member of his team asked him a question:

"Isn't data informed a better way to put what we do than data driven?"

Jeff's answer: "100%." Driven, he said, "means that you're driven for something without really looking at the reality."

I liked that he told the story against himself. The guy who teaches thousands of marketers analytics had put the wrong word on the front door, and one of his own people caught it.

He wasn't the first guest to draw that line. At MozCon in 2015, Ian Lurie passed on something a speaker had said that morning: "Don't be data driven, be data informed." Seven years apart, two very numbers-heavy people, same sentence. It's been on my mind again this year, because the dashboards have started talking.

The hippo and the $17 million intern

The best explanation of the difference I've heard came from Jeff, and it involved two characters every agency has met.

The first is the hippo, the highest paid person's opinion. People who "make decisions without any data at all, just based on their instincts, and they think that's what they get paid for." Mike King described the same animal in a persona workshop: everyone does the affinity mapping, then "the highest paid person says, well, you know, this is who our audience is". I wrote about that one in people are more than personas.

The second character is the one I think about more:

"I've also had interns who look at a Google Analytics account and they make a conclusion that you know we just made $17 million in the site because the goals counting says $17 million. And they don't realize that that's not actually money in the bank and that it was misconfigured."

That's the data driven mistake, in Jeff's words: "only looking at the data and not applying the human element." Neither is right, he said, "It's somewhere in the middle." You need "a certain amount of data to make a decision, not too much, not too little", and working out how much is the part that takes a career.

Most of us have been that intern, or sat next to one. A goal fires on the thank you page and the page reload, conversions double overnight, and for about ten minutes you're a genius.

Numbers don't know why

Ian's worry, even in 2015, was that clients "can't always walk it back to their business goals". His example still makes me wince when I look at old reports: "Great, we got more links. Great, we moved up in the rankings. What is that?" If you're looking at Google Analytics, he said, "you still have to be able to draw conclusions and then come up with creative new directions".

Dr. Pete Meyers built MozCast, which measures how much Google's results move each day. He knows more about measuring search than almost anyone, and he was frank about the limits. Daily ranking flux has a "signal noise ratio" that is "really bad". Showing that something happened was the easy bit. The hard bit was the next question: "what caused that? What does that break down into? That is an extremely difficult task."

Mike King had a sharper version, with money attached. A client of his made about 30% of its revenue, over a billion dollars, from organic search, and organic was still "the last on the totem pole", because the reports gave the last click all the credit. Nobody had to fake a number for that to happen. The story was wrong. Under last click, paid can say "we spent $10 and we got a million dollars", and nobody asks about the seven or eight organic visits that came first.

Jeff had the same warning about ad platforms. If Facebook says 50 conversions and Google Analytics says 30, he'd trust Analytics, because "every ad platform will take credit". Each one's honest about what it counts, they're just not counting the same thing.

Now the dashboard writes the story for you

GA4 has been flagging "unusual changes or emerging trends" for a while through its automated insights. Since December 2025, Google has been rolling Analytics Advisor into English language GA4 accounts: a Gemini chat inside Analytics that can investigate why a metric changed and suggest what to do about it. Google's own help page still calls it a beta and says it "may have some capability limitations". Most of the reporting tools we use have added something similar.

I use these, and they're good at the first half of the job. They'll find the spike in seconds and write a tidy paragraph about it.

But go back to the intern. If the goal is misconfigured, the AI reads the same $17 million the intern did, and now it explains it in fluent, confident sentences with a recommendation at the end. It can't know the tag broke on Tuesday, or that the spike was one bot, or that sales rang every one of those leads and none were real. The old dashboard needed a person to turn numbers into a story. The new one writes the story for you, so a person has to check it.

That's the bit of Jeff's job he said will never be finished: the "ability to weigh the qualitative versus quantitative". The tools have changed, but that part hasn't, and there's a lot more confident text to check now. Same reason I keep saying you can't automate empathy.

The common sense filter

Jeff's favourite tool is the one I've borrowed most. When someone comes to him with a very technical question or an obscure finding:

"I apply a common sense filter and it stops them in their tracks because the reality is: is this real for real people? This is humans."

My version, for reading any dashboard, whether a human or a machine wrote the insight:

1. Would the bank notice? If the dashboard says revenue jumped, check it against actual sales or the CRM. This is the $17 million test.
2. Did the tracking change? New consent banner, site release, tag edit, a goal someone "tidied up". Check change history and annotations before you believe a big swing.
3. Who's claiming the credit? Put the ad platform, Analytics and your sales numbers side by side. If they add up to more than you sold, someone's overclaiming.
4. Is it bigger than normal noise? Compare it with the same week last year and the usual range. Dr. Pete's point about signal and noise applies to your site as well as Google's.
5. Can you find one real person in it? Watch a session recording, read the enquiry, call the lead. Jeff's rule was that one customer who asks a question stands for "10 to 100 times that many people" who think it.
6. What would we do differently? If the answer is nothing, don't put it in the report. Jeff called himself "very anti-data hoarding" for a reason.
7. For AI insights, check one number yourself. Ask the tool which report, date range and filters it used, then open that report and see if you get the same figure.

Where this leaves the hippo

None of this is an argument for going back to instinct. Jeff was clear that most businesses start on faith, and data is the waypoint that tells you whether it's working. Ian's advice for getting a sceptical team on board was to make one small change that "we can't measure", map it to real business value, and build trust from there.

Being data informed means the numbers get a vote and a person makes the call. Dr. Pete's advice on presenting data still works as the last step: decide what the story is first, then pick the simplest way to show it. It's telling a story, just with numbers.

If you want more, Jeff's full answer is in his written interview, and these days he teaches clean measurement at MeasureU. Ian's is in our MozCon interview, Dr. Pete's MozCast story is in his interview from SMX Sydney, and if you're still getting your head around GA4 itself, start with our plain English guide to GA4.