When I was a teen, I used to love lying on the floor of my bedroom listening to Rick Dees Weekly Top 40 and filling in my own colored chart tracking the rise and fall of pop songs in the charts. I’d try and predict how songs would go in the following week and look at different patterns that songs followed. I didn’t have internet access to be able to look at the chart archives, so it was vital to listen in or tape the show to keep my records updated. I’d get annoyed if I was out and my taped copy ran out before the show finished and I never learnt who was #1! (I learnt today that his show is still going. Wow!)
I’ve always been a bit of a geek for collecting and plotting data. More recently, I’ve plotted everything from my weight loss after giving birth, to live election results and digging deep into Google Analytics for myself and clients. I’m planning on getting an UP3 fitness tracker when it’s released.
I love statistics. I love the stories they can help us tell and the data-driven decisions they can help us make. I spent years lecturing in statistics at The University of Auckland and in particular, statistical literacy.
My favorite quote about statistics is not the one which is most often quoted (“Lies, damned lies…”). Instead it’s this:
“Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.”
– H.G. Wells
Each year, I feel that H.G. Wells’ prediction is even more a reality. Whether it’s being able to critically consume numbers in the news, evaluate risks and research studies to make health decisions, assess marketer’s claims or understand traffic behavior on your website, statistics are everywhere.
With the explosion of wearables and the quantified self, there will be more types of data and more data being collected than ever before.
When it comes to evaluating how your website is doing, what’s the right statistic to tell the story of how it’s doing?
One of the answers that statisticians like to give most often is “it depends”. If you asked someone running a website which earns its income via advertising, they might say “page impressions per month”. A media website may say “unique visitors per month”. An online store might say “revenue per month”. A B2b may say “leads per month”. I might say “new design surveys completed per month”. A blogger may say “signups per month”. Medium might say “total time reading per month”.
Truthfully, no one measure will ever capture the whole story. Ev says he’s increasingly frustrated with this “one-dimensionality” reporting of success. It’s well-worth the time reading.
One of first questions to ask when seeing a statistic is what the definition is. How do we really define success? Measures are imperfect and proxies for what we’re really interested in.
In this post, Ev gives some good examples where statistics can hide the real story of what’s going on and discusses what does it really mean that “Instagram is bigger than Twitter”?
And whatever statistic is seen as to be the most important can be gamed.
Buzzfeed’s Jonah Peretti discusses this in an excellent, very lengthy interview:
“When you game it, you essentially are creating a fake version of that metric.
So page views are a metric of how many stories people want to read—and then you split the story in two. You essentially are doubling your pages for that story, or not quite, probably, because not everyone will click. But you create pyrite page views.
If you’re looking at clicks, if there’s a story that says “Malaysian airliner, new developments,” and you want to know about new developments in that story and you read the story and there’s a new development, then you’re happy and the click rate is actually pretty accurate. If you click and there’s actually no new developments or the story is actually about something else…
So the bigger issue is not which metric is the most important metric. You’ll see some sites talking about time as being the metric.
The challenge there is that, like you said, if you create a long, meandering, boring story that’s just good enough to keep people reading, they might spend more time on that story than the short, condensed one that just tells you what you need to know. If you use time on that one, it will tell you to do the wrong thing.
So there’s not going to be one metric that you look at. I love metrics and I love thinking about optimization, but I think that the optimal state is being slightly suboptimal because as soon as you try to actually optimize, particularly for a single metric, you end up finding that the best way to optimize for that metric ends up perverting the metric and making the metric mean the opposite of what it used to mean.”
Bigger is not always better.
To properly understand a statistic, more of the story needs to be uncovered. Whether it’s what the definitions are, when they were collected, how they were collected (and by whom) or some other such questions, statistics invite detective work.
I recently looked through an automated traffic report sent to a client that came from their SEO “expert” who was concerned about recent changes to their website and a corresponding drop off in traffic at the same time. A quick dig through their statistics showed a simpler explanation: their Adsense campaign had concluded at the same time as the site changes and that was the main cause of the traffic decline.
So many clients have Google Analytics but rarely log in or look beyond basic traffic figures. As we start the new year, I want to encourage you to learn to love data exploration and uncover stories about your site that you can use to make improvements and base decisions on. If that sounds daunting, try starting with the questions you’d like to get answers to and get help with the rest. I hope you are get more out of your statistics this year by looking at data in depth and in multi-dimensions rather than solely through single simplistic figures neatly presented in a monthly report.
If you run a website, what kinds of questions do you wonder about? Please share them in the comments, or email me directly. I’d love to hear them!