Why Your Big Data Project is Failing and How to Save It

Big data. Magical business analytics. Such topics are all the rage these days.

I hate to be the one pointing out the fact that the emperor has no clothes (well, not really, I kinda like that role actually). But the fact of the matter is that the vast majority of big data and magical business analytics projects fail. Not in a great big system-won’t-work way, but in a more insidious and hidden way. They fail because the users don’t use them.

Yup. That’s the truth. Much of the great promise of business intelligence (BI) goes unrealized because decision makers aren’t using the decision support systems in any meaningful way. Yes, I know there are a few wonderful examples that get all the press. But the vast majority of big data and business analytics projects implemented by normal companies suffer from chronic underuse.

This has been my personal experience from working in the field for more than 20 years and seeing it firsthand. I also hear it from colleagues and clients all over the world. Think I’m especially unlucky? Try reading through the data warehousing and analytics forums on LinkedIn and elsewhere. It’s a rampant problem. With some variation depending on the situation, the litany of complaints goes like this:

• “The system doesn’t tell me what I need to know.”
• “I can’t quite reconcile the numbers to what I used to get from Sue.”
• “I just want the data in Excel.”
• “I don’t have time to learn to use a big analytics system, I just want a report.”
• “I don’t want a fixed report, I want more ad-hoc freedom.”
• “I don’t want to use an ad-hoc tool, I just want the answers to my questions.”
• “Can’t you just do it for me?”

Sound familiar?

When the truth about the lack of business use slowly emerges, usually some six to nine months after your multi-million dollar business intelligence initiative goes live, CIOs and business leaders turn to the IT group and demand: “How could you let this happen? What went wrong?”

A bit unsure themselves of just why things are the way they are, despite the fact that they believe they did everything right, the IT team immediately launches its barrage of anti-business missiles:

• “The users signed off on everything. They got what they asked for.”
• “Of course they don’t use the system; they aren’t willing to learn how to use it.”
• “The users change their minds every time we see them.”
• “It’s not the system that’s the problem, it’s missing the data that they want.”
• “The users were never behind the effort in the first place. They just went along with it because it was a corporate effort.”

Usually it’s neither IT nor the business that is entirely to blame when BI projects fail. The truth lies somewhere in the middle. But knowing the truth and being able to learn from it (and prevent it from repeating itself) are very different things.

So, to avoid this problem I would like to propose we create The BI Professionals and Users Charter Principles of Implementation Excellence. It is my belief that if both sides of the BI equation (i.e., the business users and the IT builders) adhere to this charter and its principles, the problem of business intelligence underuse will go away.

The Business Intelligence Charter

We, the members of the BI program at our organization, both IT professional and business professional alike, do solemnly swear to adhere to and uphold the following implementation principles:

• No report or other front-end screen will go into development without the user community approving an actual mock up depicting the front-end user experience exactly as it will look after it is developed. (With real data!)
• No report will be developed without it being specifically identified as being used and reviewed in a particular and named business forum or meeting.
• No report will be developed without it being directly tied to a specific and tangible business question that is being asked today and that cannot be adequately answered.
• No report will be developed without the business community specifically identifying, in writing, what decision they will be making with this report.
• No report or system component will be released to any user without said user completing the training program.

For a moment I thought about explaining why each of these principles is so critical to solving the underuse problem in BI. But I figure that you get it.

Two final thoughts:

• 1.No, I am not playing favorites here. It may seem on the surface like I am coming down hard on the business but the onus for asking the questions embodied in these principles (and getting the business to provide satisfactory answers) falls squarely on the IT group.
• 2. I know that some of you are already thinking about a few more principles you would like to add to the charter. Of course you can add to it if you must. But my counsel is to keep it as short as possible. Short and sweet is critical to driving adherence.

Now off you go. Figuring out how to use that 900 TB database of cellular call location coordinates to identify your customer’s ideal location for presenting offers. Just make sure you have the exact business question that it will answer and the actual business decision it will drive firmly in hand before you spend too much time and money building it.