What Is Big Data?

You’re starting to learn about Big Data or you’re wanting to learn more about Big Data. You start of by googling “what is Big Data?” You get an answer that doesn’t quite makes sense. The site talks about 3 Vs or sometimes they’re 4 Vs or even 5 Vs. These 3 Vs are...

On Cheating with Big Data

To achieve the scales of Big Data, you have to cheat in some way. Sometimes people call these tradeoffs. In Big Data, I prefer to call them cheats. A tradeoff makes it sound like a small thing, but the reality is that Big Data tradeoffs can make a use case possible or...

When You Have the Wrong Team for Big Data

In my book, Data Engineering Teams, I talk about the right skills and people to be on a data engineering team. The right skills and people are incredibly important to the success, or failure, of a Big Data project. Sometimes it’s easier to understand this point with...

Integration Testing for Kafka

We’re creating more and more complicated data pipelines and systems with Kafka. These interactions are becoming even more complex as we create microservices. As we create these complex systems, we aren’t thinking about how to test, debug, or fix them. These 3 parts...