Why 85% of Your Internal Big Data Programs Fail or Get Stuck

(Plus, an Industry-Proven Way to Shift Your Success Odds to 90%+)

Big Data Presents Big Payoffs but NOT Before Big Problems Need to Be Solved — preferably BEFORE the project begins. But since that’s not always possible, the sooner expert mentorship can intervene (once issues first appear) — the quicker the project can be put — not just back on track — but rather, back on the fast-track.

The data tells the story. Plus, the big reason WHY most initiatives get stuck, stall, struggle, suffocate:

What’s the difference between the 15% of Big Data initiatives that avoid all the challenges/impediments, and the 85% who don’t?

(Or, why is one company successful — while 9 out of 10 fail at least 4-6 ways on average, before getting it to work right?)

Most of those 15% winning companies received high-level, masterful mentoring, hands-on expert help throughout the process — to hit that “success mark.” Some openly talk about that help and most understandably don’t.

At Big Data Institute, we provide a unique, exceptionally-successful, hard-won and well-proven approach to helping Big Data teams (and their companies) transition smoothly (and optimally) into the Big Data projects they need — to succeed competitively. We work directly with your existing team — through a powerful knowledge transfer guidance dynamic, to give them the missing skills, identify the critical nuances, problems, and the staged support they need. We call this our “Big Data Rollout.”


There are six steps to a Big Data Rollout. Simple sounding when you hear them. But they ARE the keys to producing the performance successes required – when integrated and activated together under out expert mentorship. Using these six steps, we have helped companies reduce their risk (and accelerate their timelines) and create successful Big Data projects on their very first try! These steps are:

1. Start off by assessing the team and project

2. Use case evaluation

3. Management and technical training

4. Architecture review and evaluation

5. Project planning and creation of crawl, walk, run phases

6. Programming and architecture support while coding

Assessing the Team and Project

What are the team’s technical strengths? What are the project’s complexities and weak points? Has a specific metric been created to say a project is successful or not?

These are 200 questions that need to be answered before starting a Big Data project. To get a well-rounded view of the team and their abilities, we have each member of the team fill out an in-depth survey. This gives us a starting point of where the team is and where there needs to be improvement. It allows us to focus on the parts that must be fixed before starting the project or be addressed during the training.

Use Case Evaluation

We start helping your team at the very beginning of the project. We help your team understand how designing a Big Data solution is a very different animal than small data. Before you start writing a line of code in a Big Data project, you need to understand the use case deeply. We go over the use case(s) in great depth with team and stakeholders. We ask the questions that your team doesn’t know to ask yet. We look for potential blockers and portions where the difficulty spikes. Planning at this stage pays off by not having to rewrite later on.

Management and Technical Training

Next, the teams are trained in Big Data technologies. This doesn’t just apply to the technical teams. We also train the management team.

Management training is a key piece of the puzzle. Poor management of Big Data projects and teams is one of the common recipes for failure. Managers need to know that a Big Data project is vastly different and complex than a small data project. They need to learn what is possible and what’s available in the Big Data ecosystem. They need to know the basic steps to creating a Big Data solution and the iterative approach that’s required.

Next, the technical team needs the knowledge of how to create data pipelines with Big Data technologies like Hadoop and Spark. This comes in the form of an in-depth and hands-on class on Big Data technologies.

Architecture Review and Evaluation

Now that both the management and technical teams are trained in Big Data, we move on to choosing the right technologies for the job. We do this by leading a meeting to talk about the use case and choose the technologies. Changing a box or technology on a whiteboard is vastly cheaper than changing it in code. Because the teams have been trained, they are active participants in the meeting; they’re using their domain knowledge and newly acquired Big Data skills to think through the solution.

Project Planning and Phases

Once the technologies have been chosen, the project plan can be created. This is another time where a Big Data project is very different than a small data project. Taking your team from no Big Data skills to creating a complex data pipeline is a common recipe for failure. You have to break the plan down into smaller and more manageable pieces. This allows the team to build momentum and build upon previous successes or the next phase of the project. We call this planning crawl, walk, and run.

Programming and Architecture Support

At this point, the team is ready to start coding. We don’t abandon the team at this point; they still need a safety net to ask the pressing questions and get expert help in a timely fashion. We provide the team with email and virtual support for their programming and architecture questions. Getting stuck for days, weeks, or months is costly for a new data engineering team. It’s also an unfortunately common scenario due to the complex nature of Big Data.

Why the millions you’ve already invested (or already allocated) aren’t going to give you the “Big Data Rollout” ROI you expect, UNLESS…….

While your internal engineering team represents millions of dollars of cumulative investment – the expertise your team gained was/is domain knowledge of your existing products and processes. That’s been invaluable for the level of segmentation, analysis, and quantification you’ve asked of them.

But tackling Big Data uber “challenges” for the multitude of “high-yield” project payoffs definitely has the capacity to provide – – is recognized by experts, institutions and technologists alike, as being at least 10X more complicated – full of unfathomable integration/compatibility issues. Yet the multiple-opportunity yielding payoff it can continuously (and progressively) serve up, done right, IS unimaginable – frequently exceeding initial targets by double, sometimes redoubling, even doubling outcomes again.


Here’s a list of the most common reasons why your Big Data rollout can easily become a Gordian knot:

  • Novices to a new and highly complex domain make the most mistakes due to a lack of experience in the domain.
  • Organizations will start or completely staff their Big Data initiative with their data warehouse or DBA team.
  • The team is never given the time or resources to learn the brand-new technologies and concepts.
  • Teams will drastically underestimate the amount of time, complexity, and resources needed for the project.
  • No one understands the problems that happen after the initial deployment that have to be done during the initial development.
  • The project is too ambitious for the team’s abilities or use case and the team never makes any progress.

If you’re reading so far, odds are exceptional that YOUR team has been “snared” or stymied by more than one of these traps already. In all probability, they are still deeply engulfed trying to extricate the project from this technological miasma – – trying desperately to break free from the clutches of a well-intentioned initiative somehow turned bad. Our job is to come in and assess, identify, then resolve those restrictions. Then, we step everyone on the team UP to the next level of challenge, then the next – until the entire project “challenge battle” is won – with your team becoming victorious and your company gaining the rich rewards!

Trying to solve (and resolve) it on their own takes your team hundreds of combined hours, weeks, or months – if not multiple teams – plus enormous time, effort, frustration, lost opportunity cost, resets, and non-stop trouble-shooting – – while your critical business timetable gets pushed further and further behind. And your budget gets bloated fatter and fatter. But you obviously already realize this from your problematic experiences so far.


Your goal in doing this project IS more than worthy; it’s competitively long-term “mission-critical” and essential. All the more reason your team MUST get it right the first time!

Please don’t feel guilty, angry, or think your people incompetent. Variations of this scenario are playing out in 9 out of 10 current Big Data initiatives all over the country (and in industries of every kind) — and that fact is confirmed to be the “truth” in a Gartner research article:

Only 15 percent of businesses reported deploying their big data project to production, effectively unchanged from last year (14 percent).

But we overcome the impediments, constraints, obstacles, and self-defeating issues, as well as those deeply-seated, totally undervalued “monkey wrench” factors that get unknowingly thrown into the gears throughout the rollout process — unrecognized and unbeknownst to everyone on the team — team leader(s) included.

Doing this in continuous collaboration with us as your teams’ and leaders’ “Masterful Big Data Thinking Partner,” while following the six-part process described above — and teaching everyone proactive, critical and consequential Big Data Project thinking as we progress through the project — transforms your team into Big Data “Black Belts” fully capable of sustaining and successfully expanding the continuous Big Data growth initiatives you’ll next want to achieve.

We produce the absolute corollary to all this negative data (and derailment) — We represent the flip side of the coin: one out of ten Big Data initiatives are powering through the maze magnificently, and proficiently — and producing outcomes for their companies’ that are true game changers. These are the caliber of companies (and teams) our company is brought in to mentor both in the beginning, or when trouble first starts to mount — to get that project going and growing, and get the program producing the kind of spectacular performance results we both know are achievable (done right the first time or course-corrected early in the game).


We solve Big Data “Gordian knots.” We re-orient the entire team’s strategic mindset. We transfer knowledge permanently through demonstrating higher/better thinking and action/ solution-based case studies — full of detailed, problem-causing, identified real-life horror stories and turnarounds — focused squarely on the critical elements, facets, factors and processes that might have gone wrong, or the high probability it will go wrong. Then we help guide, direct, strategize and mastermind the technical and integration cures needed.

This way we can capably solve, fix, rework, replace — or teach the team to proactively avoid faulty thinking, erroneous assumptions, problem-fraught aspects of the initiative — together, working together under MY guidance — but with YOUR team confidently and masterfully executing, and taking ownership of the knowledge and methodology forevermore.


Not infrequently I’ll find that one, if not THE biggest, problem is failure from an erroneous assumption, or failure to see the inherent complexity jof Big Data.

Also, for many teams we have to audit, assess, reevaluate performance (functional roles and skillsets) AND hold/validate and evaluate the entire project — this lets us recommend reallocation of resources and team members to their optimal performance advantage, if required — for higher achievement of your outcome goal. We correlate the assumptions on which your project is based vs. the processes and progress (or lack) you’re producing. Oftentimes we must reset each phase or advise shifting different team positions; because the current execution strategy is either stuck, or its trajectory and team roles is far off-target. If the project is not advancing or achieving the outcome your company is after — I fix it so your goal WILL be achieved (or overachieved) in the most timely, cost-effective, sustainable manner possible.

While none of this is easy — it’s all quite doable, and in doing it, we frequently deliver to you a far more profitable, proficient, capable, competent and formidable Big Data team to go forward with — in a monumentally more successful way for years and critical projects to come.

Again, when we collaborate with your team, we implement / integrate:

  • Team project no-nonsense validity assessment
  • Use case study examples to demonstrate the most effective transformative methodologies in successful Big Data project makeovers (so everyone enthusiastically “buys in!”)
  • Accelerated management and technical top-to-bottom retraining or rethinking
  • Architectural Catscan 360 Review and probability outcome evaluation — with basis and content fully provided
  • Project planning/reporting
  • Program and ongoing progress/course correction support


That’s the big picture. Now let me get more granular, so your tech leaders can comfortably and confidently feel good at least — conferring with me (without cost or obligation, and in confidence) about the issues you’re currently facing. Please note: everything I do with an organization is not meant to be static. My involvement is constant and consistent; it can include every team member. It includes both strategic as well as tactical implementation. My purpose is to transfer my expert knowledge permanently over to your team — so they/you won’t need another Big Data Mentor again AFTER we bring in the successful project at hand. It all is strategically designed to turn struggling technology teams into Big Data (sustainable) superstars.

Reality Checking 3-Way Process You MUST Use

Do you have an effective/successful method for assessing your team, their progress, and proper position assignment of the project itself? Plus, interventional strategies if any of these critical elements become out of sync or impeded?

When we talk about your situation — I’ll share the basis and reasoning for our methodology, along with a dozen of the factors we access; you’ll see how differently we think for you and your team/company.

I’d like to share ten with you here:

  • Do you have the right people and positions for the project?
  • Do you have all the core skills necessary for a data engineering team?
  • Does the team have any skill gaps?
  • Does the team have any ability gaps?
  • Does the management team have reasonable expectations for the outcomes and deliverables for the project?
  • Has the team clearly and deeply studied the use cases?
  • Were the technologies chosen before the use case was fully understood?
  • Does the team have all of the resources they’ll need to develop, test, and deploy the project?
  • Does the team already have difficulty just working with small data?
  • Does the management team have a sane timeline for the project’s completion?


Hopefully, this provides a great indicator of WHY my mentoring produces such rapid results and team turnaround success.

Now you know what/who to assess on your team without the project protocols (but why are these factors so important)?

Let me give a simple — but common — example of how mentoring helps teams get unstuck. It’s very common for organizations to start their Big Data practice in their data warehousing or database teams. Look around and you’ll see others expousing this view, but they’re wrong. The vast majority of Big Data project failures happen initially because the project was started in the wrong department with the wrong people. With our experience and data, we’ve shown that database- and SQL-focused positions are the wrong positions for the job. With our mentoring, we undercover the unknown unknowns that are making you fail.

We specialize in working with enterprises and specific issues that enterprises face like handling multiple projects and teams – with results that are ALL proven and validated in scenario-based, real-world Big Data environments like your team is facing – We are brought in to monumentally enhance success, diminish problems, accelerate timelines, align team collaboration – deliver out-sized results (frequently far exceeding your projected expectations).

It was frustrating. We were failing and the Vice President was watching us do it. We had all the books and tried to work through the steps, but our use of relationship database systems like MySql and Oracle meant we were approaching Hadoop in a fundamentally incorrect way. Our RDBMS solution was slowing down the Big Data solution.

Within two days of finishing the course we had rewritten major components of the database architecture. Over the next six months we completely overhauled everything we had done before we worked with Jesse. The 14 billion rows of data that used to take 16 hours to run were running in 2 hours, and the upgrades saved our company millions of dollars. If you take into account the lost time, and the salaries of our team while we were failing we could have saved a lot more — we really should have worked with Jesse sooner.

Steve R.

Publicly Traded Digital Marketing Company

Here for your benefit are actual case studies from our client files that clearly demonstrate how I help your team bring the project in – and a big/winning way. You can see all the areas a project can go wrong – plus exactly and precisely how we get it back on track on time and on point.

The wrong ratio of data engineers to data scientists


One company was stuck and couldn’t figure out why. They had 50+ data scientists, but they had all sorts of trouble getting things done. The data scientists were running at 20-30% efficiency because they were doing jobs they were suited for. We came in and helped the company establish the right ratio of data engineers to data scientists. The data scientists went from from 20% efficiency to 90% efficiency. They company was able to do significantly more with the same number of people.

Stopping what was started


One company would start projects and then hit a wall. They could execute up to a point, but when things got too complicated, they stopped. By their own estimate, they were losing millions of dollars in business because the team couldn’t do the complex parts of the data pipeline. We came in, identified the issues in the personnel on the team, and helped them sort out their architecture. The team is now making the company millions and starting to back through their abodoned projects to restart them.

Creating the data engineer classification and first teams

One enterprise lacked the job classification for a data engineer. This made them lose out on qualified candidates because it was the wrong title and pay scale. The wrong job descriptions led to resumes and hiring of unqualified people. We worked with the human resources department to establish a correct job description of a data engineer. We showed them why a data engineer is not a data scientist or a DBA. The company started recruiting the right people with the right titles and the right salaries.

Stopping the wrong technology

One enterprise was going to make a company-wide rollout of a Big Data technology. They didn’t research the technology in the depth and breadth they should have. We helped the teams understand why the technology was the wrong choice and which technologies were a better choice. This saved the company millions of dollars from having to rewrite code and transition to a new technology right after having deployed it.


A successful Big Data rollout makes your business/company more profitable, productive, preeminent, competitive. (A failed attempt wastes millions AND puts your business well behind the competitors you’re trying to outpace.)

The Critical Difference Between A Static Big Data Consultant, A Big Data Project Coach, A Big Data Elevated Success Mentor, and a 3D Masterful Thinking Partner

Which one, or combination, will most effectively impact your ultimate outcome, timeline, cost containment, performance capability/expandability? Why don’t YOU be the judge?

When Big Data projects falter, fizzle, fail or merely get seriously stuck — and preferably prior to even beginning — you have four choices to help ensure optimal success with minimal problems, impediments, constraints:

Hire a Static Consultant to “parachute” in to try fixing the singular problem at hand. Assuming they do — they “helicopter out” and you’re on your own to deal with the next, and the next, and the next wave of integration challenges your Big Data project will progressively face. I call this Band-Aid technology — addressing a current symptom, not the macro, long-term issue.




Hire a Big Data Project “Coach” (or anyone of a dozen other titles they go by). This is certainly a better choice than a one time / one issue quick fix — but it, too, comes with unique limitations your project — itself helps create. To wit:

A coach focuses on coaching the leader and/or team to reach a current outcome (your leadership has decided) is a competitive advantage for the company. And successfully achieved — in a timely enough way and within the budgeting tolerances you’ve allocated — that project can be a mammoth, temporary competitive advantage. But — and don’t shoot the messenger here! — sooner or later ALL your viable direct and indirect competitors WILL have to follow suit — in order to merely remain reasonably competitive. What your company may initiate first, and lead by a mile doing — SOON BECOMES THE STANDARD operating procedure for everyone. That means your coach needs to not let you build a Big Data system that can’t explain, adapt, and preemptively evolve as your competitive position and technology breakthroughs require. Ending up with the best 3G or 4G system, when 5G is around the corner — or choosing technology solutions that may, actually, connect well for the current requirements but become obsolete for the next growth phase is a pricey, short-term competitive pyrrhic victory.

Hire a Big Data Mentor. This is someone possessing extensive first-hand, supremely successful Big Data project completion history. It’s someone who has a vast experiential background dealing with just about every conceivable challenge, issue, problem — technological or otherwise — that gets Big Data projects stuck.

This expert knows precisely how to tackle far more than the problem at hand. He/she understands how to expand the critical thinking knowledge base of each team leader. They know most probable sequences of problems that will arise next …….. Thus, they can teach your team how to engineer (and pre-test) many little understood elements of the system — before another major breakdown can occur in the project.

They can reduce your “problem factor” by as much as 85-90% — which, of course, supports timeline compliance and budget fiscal success. More importantly, coming in on time — (or early) and within (or under) budget allows your Big Data initiative to provide maximum competitive/profit advantage for the maximum period of time — multiplying your ROI both in percentage terms and longer duration before it needs upscaling, expansion, or replacement.

But, BETTER than all that — a Big Data mentor clearly grasps a far-sighted understanding of what will most likely evolve in the future, technologically as well as capability-wise. So they won’t allow you to go forward with a flawed project in either intent or content.

The final category of expert Big Data specialist is what I call a Masterful Thinking Partner. They are definitely rock-solid, outrageously successful mentors — BUT, they take their responsibility to a far higher level of collaboration, advisement and project leadership guidance: preemptive thinking training, consequential as well as Big Data-based critical thinking, and integrated insight coupled with foresight. This ensures your team ALWAYS remains cutting edge, well ahead of the herd, critically on top of viable technology and application development that both sustains and geometrically expands competitive positioning of technological advantage.

I combine both Big Data Mentoring WITH a Masterful Thinking Partnership. To my knowledge — no other successful specialist provides this combination in the integrated and elevated way I do to ensure maximum paradigm/expanse in your entire team.

Do you know what your team and team leader really need to permanently sole the problem? We don’t treat every Big Data client project the same. Why? Because one-size can’t possibly fit all the differences, variables, factors, and issues your Big Data team and project need. Some teams are highly advanced; thus, their issues are far more sophisticated and nuanced, and require exceptional technological mastery. Other teams and projects are more foundational and require a far different nurtured form of knowledge transfer.

We evaluate your case to find the exact technology you need plus the interviews, instructions, and intentions are more comprehensive.


Our track record is significant enough, successful enough, and results certain. If we accept you as a client, we guarantee we will refund our fee pro rata up to 100% if we don’t hit our promised deliverables’ timeline/budget. 

Obviously, this guarantee IS predicated on your team and leader collaborating fully, executing our directives in a timely manner, and allowing us to have technical influence over the entire team strategy. That part, followed on your side — and if we accept you and your project — we fully expect a monumental success outcome — or we’re the biggest loser, not you! We’ll be accountable to you in progressive, pre-agreed-upon performance stages.

A few final points that we think are important.

When you collaborate with us (whether at the very beginning of the project — or to successfully turn around a project gone wild) we ALWAYS make certain everyone on your Big Data project team (from leader to technical individual contributors) gain clarity in the nuances and variances we grasp. The ways a team is run, the makeup of the team, the skills, the use case, political atmosphere all make a difference in the successful outcome; so we make certain we transfer expert understanding of it all — if needed.

I wrote one of the more validating books on Big Data, entitled:

Data Engineering Teams

The book explains my proprietary strategies and systems for how to manage and create Big Data teams, the exact steps I provide organizations/clients to take when creating Big Data solutions.

Finally — here’s proof, testimonials, references, success stories, and everything else needed to make you comfortable, confident, and committed to get on the phone with me either before you start your Big Data project — or before any current problem gets and worse.

Think of us as a Big Data Successful Rollout Assurance or think of me.

The Ways We Can Contribute and When Each Is Most Appropriate


Why it makes sense to at least talk through your current or forthcoming Big Data project and get my or our well-reasoned, extensive experience and expert perspective on either the strategy, the process, the people, the purpose, the outcome expectations or all of the above. 

To my knowledge there is no one more knowledgeable and experienced and capable of either course correcting, fast tracking, mentoring or overseeing the success of your big day to rollout.

That may sound a bit bold and arguably audacious. But facts ARE facts and I’ve been brought in to turnaround (or totally re-direct) entire multimillion dollar Big Data rollouts — and my success rate is 100% (of course that’s predicated on you providing a collaborative, directable and knowledge thirsty team for me to remold.)


The only way you and your team can verify/confirm that statement is to schedule a comprehensive and deeply probative confidential discussion with me and see what I see that, perhaps, you and your team aren’t perceiving.

In summary — please know that, there’s no negative in seeking expert collaboration to enhance, advance, course-correct and ensure optimum Big Data rollout success. The payoff is incalculably substantial, the dangers of not executing successfully are unimaginably profound.

It’s (arguably) one of the most complex and challenge-fraught processes and programs your company will ever pursue.


It’s also (arguably) one of the most game-changing, outcome-shifting and success-propelling achievements you could accomplish. And I’m here to help your team and your company make the process more successful, the outcome more effective and the impact to your business far more stratospheric.


At very least let’s talk through your goals, current progress, intended process, and the profiles of the people assigned to the team. Let’s see whether I can make a big difference and my perspective has value or not within a one-hour-long conversation. 

I wrote the book on fixing and preventing Big Data roll out problems. I’d like to buy you a copy to introduce my thinking and expertise to you and your team. At worse, it’ll answer many questions you’re probably struggling with, it’ll give you directions you’re probably searching for, it’ll answer and solve some meaningful problems, or challenges you are your team are grappling with. And — hopefully, it’ll compel you to feel comfortable and confident discussing these problems and challenges with me risk free.

If you want blessed relief for a Big Data rollout that’s in trouble — let me help by scheduling a time to talk!

5 + 6 =