%
Why 85% of Internal Big Data Programs Fail or Get Stuck
While our mentoring clients have a 90%+ success rate
Customer Example: A Completely Overhauled Project in Six Months
“It was frustrating. We were failing and the Vice President was watching us do it. We had all the books and tried to work through the steps, but our use of relationship database systems like MySql and Oracle meant we were approaching Hadoop in a fundamentally incorrect way. Our RDBMS solution was slowing down the Big Data solution.”
“Within two days of finishing the course we had rewritten major components of the database architecture. Over the next six months we completely overhauled everything we had done before we worked with Jesse. The 14 billion rows of data that used to take 16 hours to run were running in 2 hours, and the upgrades saved our company millions of dollars. If you take into account the lost time, and the salaries of our team while we were failing we could have saved a lot more — we really should have worked with Jesse sooner.”
STEVE R.
What’s the difference between the 15% of Big Data initiatives that avoid all the challenges/impediments, and the 85% who don’t?
Or, why is one company successful — while 9 out of 10 fail at least 4-6 ways on average, before getting it to work right?
Most of those 15% winning companies received high-level, masterful mentoring, hands-on expert help throughout the process — to hit that “success mark.” Some openly talk about that help and most understandably don’t.
At Big Data Institute, we provide a unique, exceptionally-successful, hard-won and well-proven approach to helping Big Data teams (and their companies) transition smoothly (and optimally) into the Big Data projects they need — to succeed competitively. We work directly with your existing team — through a powerful knowledge transfer guidance dynamic, to give them the missing skills, identify the critical nuances, problems, and the staged support they need. We call this our “Big Data Mentoring.”
Six Critical Steps to Success with Big Data
There are six steps to a Big Data Mentoring. Simple sounding when you hear them. But they ARE the keys to producing the performance successes required – when integrated and activated together under our expert mentorship. Using these six steps, we have helped companies reduce their risk (and accelerate their timelines) and create successful Big Data projects on their very first try! These steps are:
- Start off by assessing the team and project
- Use case evaluation
- Management and technical training
- Architecture review and evaluation
- Project planning and creation of crawl, walk, run phases
- Programming and architecture support while coding
Let's talk and schedule a time to go over your project and needs
Step 1: Assessing the Team and Project
What are the team’s technical strengths? What are the project’s complexities and weak points? Has a specific metric been created to say a project is successful or not?
These are 200 questions that need to be answered before starting a Big Data project. To get a well-rounded view of the team and their abilities, we have each member of the team fill out an in-depth survey. This gives us a starting point of where the team is and where there needs to be improvement. It allows us to focus on the parts that must be fixed before starting the project or be addressed during the training.
Step 2: Use Case Evaluation
We start helping your team at the very beginning of the project. We help your team understand how designing a Big Data solution is a very different animal than small data. Before you start writing a line of code in a Big Data project, you need to understand the use case deeply. We go over the use case(s) in great depth with team and stakeholders. We ask the questions that your team doesn’t know to ask yet. We look for potential blockers and portions where the difficulty spikes. Planning at this stage pays off by not having to rewrite later on.
Step 3: Management and Technical Training
Next, the teams are trained in Big Data technologies. This doesn’t just apply to the technical teams. We also train the management team.
Management training is a key piece of the puzzle. Poor management of Big Data projects and teams is one of the common recipes for failure. Managers need to know that a Big Data project is vastly different and more complex than a small data project. They need to learn what is possible and what’s available in the Big Data ecosystem. They need to know the basic steps to creating a Big Data solution and the iterative approach that’s required.
Next, the technical team needs the knowledge of how to create data pipelines with Big Data technologies like Hadoop and Spark. This comes in the form of an in-depth and hands-on class on Big Data technologies.
Step 4: Architecture Review and Evaluation
Now that both the management and technical teams are trained in Big Data, we move on to choosing the right technologies for the job. We do this by leading a meeting to talk about the use case and choose the technologies. Changing a box or technology on a whiteboard is vastly cheaper than changing it in code. Because the teams have been trained, they are active participants in the meeting; they’re using their domain knowledge and newly acquired Big Data skills to think through the solution.
Step 5: Project Planning and Phases
Once the technologies have been chosen, the project plan can be created. This is another time where a Big Data project is very different than a small data project. Taking your team from no Big Data skills to creating a complex data pipeline is a common recipe for failure. You have to break the plan down into smaller and more manageable pieces. This allows the team to build momentum and build upon previous successes or the next phase of the project. We call this planning crawl, walk, and run.
Step 6: Programming and Architecture Support
At this point, the team is ready to start coding. We don’t abandon the team at this point; they still need a safety net to ask the pressing questions and get expert help in a timely fashion. We provide the team with email and virtual support for their programming and architecture questions. Getting stuck for days, weeks, or months is costly for a new data engineering team. It’s also an unfortunately common scenario due to the complex nature of Big Data.
We'll help you do a reality check on your Big Data project
Do you have an effective/successful method for assessing your team, their progress, and proper position assignment of the project itself?
When we talk about your situation — I’ll share the basis and reasoning for our methodology, along with a dozen of the factors we access; you’ll see how differently we think for you and your team/company.
During our conversation, we can perform reality checks on these topics:
- Do you have the right people and positions for the project?
- Do you have all the core skills necessary for a data engineering team?
- Does the team have any skill gaps?
- Does the team have any ability gaps?
- Does the management team have reasonable expectations for the outcomes and deliverables for the project?
- Has the team clearly and deeply studied the use cases?
- Were the technologies chosen before the use case was fully understood?
- Does the team have all of the resources they’ll need to develop, test, and deploy the project?
- Does the team already have difficulty just working with small data?
- Does the management team have a sane timeline for the project’s completion?
Unprecendented Success Guarantee
Our track record is significant enough, successful enough, and results certain. If we accept you as a client, we guarantee we will refund our fee pro rata up to 100% if we don’t hit our promised deliverables’ timeline/budget.
Obviously, this guarantee is predicated on your team and leader collaborating fully, executing our directives in a timely manner, and allowing us to have technical influence over the entire team strategy. That part, followed on your side — and if we accept you and your project — we fully expect a monumental success outcome — or we’re the biggest loser, not you! We’ll be accountable to you in progressive, pre-agreed-upon performance stages.
Data Engineering Teams
Creating Successful Big Data Teams and Products
When you collaborate with us (whether at the very beginning of the project — or to successfully turn around a project gone wild) we ALWAYS make certain everyone on your Big Data project team (from leader to technical individual contributors) gain clarity in the nuances and variances we grasp. The ways a team is run, the makeup of the team, the skills, the use case, political atmosphere all make a difference in the successful outcome; so we make certain we transfer expert understanding of it all — if needed.
I wrote one of the more validating books on Big Data, Data Engineering Teams. The book explains my proprietary strategies and systems for how to manage and create Big Data teams, the exact steps I provide organizations/clients to take when creating Big Data solutions.
Think of us as a Big Data Successful Rollout Assurance or think of me.
At the very least - let's talk!
At very least let’s talk through your goals, current progress, intended process, and the profiles of the people assigned to the team. Let’s see whether I can make a big difference and my perspective has value or not within a one-hour-long conversation.
I wrote the book on fixing and preventing Big Data roll out problems. I’d like to buy you a copy to introduce my thinking and expertise to you and your team. At worse, it’ll answer many questions you’re probably struggling with, it’ll give you directions you’re probably searching for, it’ll answer and solve some meaningful problems, or challenges you are your team are grappling with. And — hopefully, it’ll compel you to feel comfortable and confident discussing these problems and challenges with me risk free.