Analytics and Inside Pool

You may or may not have been hearing about a debate going on in web analytics.

To most, it might seem like a lot of inside pool. And I suppose most of these things are.

I want to talk a little bit about some of that inside pool.

Over the course of my WAA Research Committee work last week, I stumbled upon a paper entitled “Assumptions, Explanations, and Prediction in Marketing Science: “It’s the Findings, Stupid, Not the Assumptions” by Eric W. K. Tsang.

In it, he replies to a debate that’s been going on for a long time, but what natural scientists had settled a hundred years ago. Richard Staelin back in 1998 said that there’d always be debates about whether analytical models needed to have realistic assumptions or not. Shugan came out in 2007 and argued that it wasn’t about the assumptions. I can remember reading that paper back then. It had an effect on me. Let’s fast forward to 2009.

I don’t quite know how it happened, but I ended up sitting at a table with the megastars of Marketing Science research at an informs conference. Dr. Lehmann was there – as was Dr. James Lattin. From what I gather – they’re pretty distinguished researchers. I didn’t know it at the time, and I doubt that it would have changed my behavior much. Maybe only outliers would ever dare sit with that group. That’s how I met Alex.

Two outliers at a table of high insiders.

Alex is an economist out France. I won’t go so far as to call him a French Economist, but, regardless, there it is. 🙂 We got into a discussion about how irritated I was with stupid assumptions.  I understood that without invalid assumptions that the math wouldn’t work: but maybe there’s no value in the math that doesn’t work. That unless I could use the model to understand the world, or at very maximum: predict the future in some way – that wasn’t of any use to a practitioner or to a scientist. Alex explained to me that the Math unto itself could help science chip away at the edges of complexity – and if something adds understanding, then it is of value: but maybe not to a practitioner.

I still accept where Alex comes from. I think there’s a role for trying to understand complexity by way of deliberate simplification. How those assumptions get selected still bothered me, and I continued to want to shout down anybody who had selected, in my judgement, a stupid assumption for such little gained value.

Back to Tsang, in his paper, where he takes Shugan on. Apparently there’s an entire school of thought that dismisses my belief that science should have at least a goal in making accurate predictions about the future. Tsang carefully deconstructs Shugan’s 2007 arguement, and in the end concludes that “although Shugan (2007) rightly stresses that it is inappropriate to dismiss a model or a theory based only the realism of its assumptions, realism does matter, and it matters a great deal for model building and theory development.”

And I happen to agree with Tsang. He’s helped me immensely in being able to reconcile some of that inside pool.

A lot of the inside pool going on right now in Web Analytics is very similar to Tsang-Shugan and Christopher-Alex. There are huge disconnects between what many web analytics practitioners want analytics to be, what some of the industry titans want it to be, what customers of web analytics outputs want it to be, and even within the broader analytics community (data miners, revenue managers, and market researchers are in the same neighborhood) want it to be.

All this – within an industry that couldn’t possibly employ more than 50,000 people in total.

Inside pool is important because it’s about values and refining the definitions that are in use by a community.