An engineering team needs more than just 4 days of training to be successful. We mentor and guide teams through the entire process of creating the solution.
You have an engineering team that you’ve spent millions building and maintaining. They have the domain knowledge of your products and processes.
But, you’re realizing Big Data is a very different animal. It’s 10x more complicated than small data. You’ve read articles about how 85% of Big Data projects fail. You seen other executives talk about their Big Data initiative going nowhere.
At the same time, you’re seeing other companies really pulling ahead of the competition by using their Big Data projects effective.
What’s the difference?
Why is one company successful while another fails?
Most of those companies received expert help to be successful. Some talk about the help and most don’t.
At Big Data Institute, we provide a unique approach to helping teams and companies transition to Big Data. We work directly with your existing team to give them the skills and support they need. We call this our “Big Data Rollout.”
There are six steps to a Big Data Rollout. Using these steps we have helped companies reduce their risk and create successful Big Data projects one their very first try. The steps are:
- Start off by accessing the team and project.
- Use case evaluation
- Management and technical training
- Architecture review and evaluation
- Project planning and creation of crawl, walk, run phases
- Programming and architecture support while coding
Accessing the Team and Project
What are the team’s technical strengths? What are the project’s complexities and weak points? Has a specific metric been created to say a project is successful or not?
These are 200 questions that need to be answered before starting a Big Data project. To get a well-rounded view of the team and their abilities, we have each member of the team fill out an in-depth survey. This gives us a starting point of where the team is and where there needs to be improvement. It allows us to focus on the parts that must be fixed before starting the project or be addressed during the training.
Use Case Evaluation
We start helping your team at the very beginning of the project. We help your team understand how designing a Big Data solution is a very different animal than small data. Before you start writing a line of code in a Big Data project, you need to understand the use case deeply. We go over the use case(s) in great depth with team and stakeholders. We ask the questions that your team doesn’t know to ask yet. We look for potential blockers and portions where the difficulty spikes. Planning at this stage pays off by not having to rewrite later on.
Management and Technical Training
Next, the teams are trained in Big Data technologies. This doesn’t just apply to the technical teams. We also train the management team.
Management training is a key piece of the puzzle. Poor management of Big Data projects and teams is one of the common recipes for failure. Managers need to know that a Big Data project is vastly different and complex than a small data project. They need to learn what is possible and what’s available in the Big Data ecosystem. They need to know the basic steps to creating a Big Data solution and the iterative approach that’s required.
Next, the technical team needs the knowledge of how to create data pipelines with Big Data technologies like Hadoop and Spark. This comes in the form of an in-depth and hands-on class on Big Data technologies.
Architecture Review and Evaluation
Now that both the management and technical teams are trained in Big Data, we move on to choosing the right technologies for the job. We do this by leading a meeting to talk about the use case and choose the technologies. Changing a box or technology on a whiteboard is vastly cheaper than changing it in code. Because the teams have been trained, they are active participants in the meeting; they’re using their domain knowledge and newly acquired Big Data skills to think through the solution.
Project Planning and Phases
Once the technologies have been chosen, the project plan can be created. This is another time where a Big Data project is very different than a small data project. Taking your team from no Big Data skills to creating a complex data pipeline is a common recipe for failure. You have to break the plan down into smaller and more manageable pieces. This allows the team to build momentum and build upon previous successes for the next phase of the project. We call this planning crawl, walk, and run.
Programming and Architecture Support
At this point, the team is ready to start coding. We don’t abandon the team at this point; they still need a safety net to ask the pressing questions and get expert help in a timely fashion. We provide the team with email and virtual support for their programming and architecture questions. Getting stuck for days, weeks, or months is costly for a new data engineering team. It’s also an unfortunately common scenario due to the complex nature of Big Data.
A Big Data rollout makes you more profitable, productive, and preeminent.
A Big Data rollout is for companies that want to save millions. They realize that expert guidance is the key. One of our clients found this out halfway into their Big Data project. We’ll let them tell the story of failing for six months and before coming to us for help:
It was frustrating. We were failing and the Vice President was watching us do it. We had all the books and tried to work through the steps, but our use of relationship database systems like MySql and Oracle meant we were approaching Hadoop in a fundamentally incorrect way. Our RDBMS solution was slowing down the Big Data solution.
Within two days of finishing the course we had rewritten major components of the database architecture. Over the next six months we completely overhauled everything we had done before we worked with Jesse. The 14 billion rows of data that used to take 16 hours to run were running in 2 hours, and the upgrades saved our company millions of dollars. If you take into account the lost time, and the salaries of our team while we were failing we could have saved a lot more — we really should have worked with Jesse sooner.Steve R.
We want to help you become more profitable, productive, and preeminent with a successful Big Data project. Fill out the box below to get your company started.