The Data Analytics sector has seen immense growth since the early days of the internet. The original startup boom of the 90’s put data analytics solutions on the map and drove the advancement of database technologies. This was followed by the “Big Data” movement of the 2010’s and the current promotion of the “modern data stack”. As a data practitioner for the past decade, seeing this evolution and investment in the field has been a welcomed sight. However, I feel that most recent companies vying to be considered a crucial part of the modern data stack are more hype than substance.
Driven by large IT/Data budgets and VC-generated buzz, new tools are competing to become market leaders in newly created verticals within the sector. a16z captures this recent tool evolution with the diagram that follows.
If you felt overwhelmed by the diagram, then don’t worry here is another diagram of the modern data stack.
Still lost? How? And you call yourself a data practitioner!?! Don’t worry, even the most seasoned data person will not understand the whole map. These chaotic diagrams do the best job they can to capture the complexity of defining the “modern data stack”. With so many new tools and areas to now consider it makes you wonder what tools are an advancement and what are just a substitute for existing technologies.
To the naked eye, the branding and website copy make you feel that their business could be revolutionary to the data industry. However, upon closer inspection, many of these tools provide functionality that are abstractions of individual features of the foundational data tools we use today. They might be quality enhancements to these features, but in the wise, soul-crushing words echoed by many VC’s, “it's a feature, not a product”. Usually in these cases, the startup does not have a long lifespan, making it a poor candidate to invest time and money in. There is nothing more humbling than vouching for the new, hyped tool, only to have to explain to the company that you are back to square one after a few years of usage.
Beyond anticipated lifespan of new tools, you should also evaluate the functionality of your current tools to determine if the need for a new tool is legitimate or just recency bias? We all love new. It is sexy and carries none of the baggage we have developed with existing tools over time. Sometimes this makes us blind to the fact that existing tools have these capabilities and justifies the decision against buying a new tool.
Building on this is the idea of cost and value. It easy to go on a spending spree if given the budget, but usually at the expense of business objectives. In order to prevent wasterful spend and resources, there needs to be a process and strategy in place. By having this structure, you should understand where the business is currently and where it would like to go and a roadmap that will help you understand the incremental steps to get to the goal. At each step of the way you can reevaluate if objectives can be achieved with the current tools or another tool is needed to achieve that goal or help you get there faster. This allows you to keep your stack small and scalable in a manageable way.
A better approach to tooling is to think about “your data stack” rather than worry about the “modern data stack”. A few questions to ask yourself before you go off and adopt a new tool for your data stack are:
- What are the problems that your data stack faces today?
- Have you done enough research to confirm that those problems cannot be resolved with the tools you have today?
- Is it that your current tools can’t resolve this problem or is it really that you are lazy and don’t want to be bothered with the work?
- What is the cost/benefit of solving this issue? Is it worth it?
- Which tool helps you beyond the immediate need? What can provide incremental value in the future?
This is by no means a complete list of questions you should ask yourself, but the intent here is to ask yourself questions that fight natural biases and hype before spending time, money, and resources on a new data tool. Rather than fear falling behind competitors who are using the hot, new tools, focus on what is best for your business and the tools that fit its needs.