Harry Stebbings
, an independent podcast focused on venture capital.
More posts from Harry Stebbings
“We are entering the most exciting time for big data,” according to Matt Turck, in our latest interview. In 2010 only 2.5 percent of the Series A market was committed to big data. Today, the sector amounts to more than 7.5 percent of total venture investments. So where are we in the world of big data, and is the recent obsession with AI still fundamentally related to big data?
Ecosystem maturation
The evolution of big data can be viewed in three distinct timeframes. The formative years of the big data landscape were dominated by a small set of large Internet companies (LinkedIn, Facebook, Google). As a result of their traction, these companies attained mass datasets, had no legacy infrastructure and had the ability to hire the best engineers. So they set out to build the technology the world needed.
The second phase saw the fragmentation of top technical talent away from large Internet companies to form their big data own startups. The rise of many “budding unicorns” provided these startups with a client base that similarly had no legacy infrastructure. As Turck stated in our interview, “their lack of legacy infrastructure was key to their innovation.” So they became early clients of these big data startups.
The final phase takes us to the present and the most challenging. Many big data technologies have been embraced by a broader range of companies; however, we still have a long way to go.
Unlike the “budding unicorns,” most companies do have a legacy infrastructure and a lot more to lose. To a large extent, their existing infrastructure is sufficient, presenting the most challenging hurdle. How can the big data community convince companies to leave legacy infrastructure that has been core to their business, in favor of big data technologies?
Turck stated, “it is the job of startups to show that data allows you to be inherently smarter about your business.” Whilst suggesting that there must also be willingness from larger companies “to begin playing with big data.”
AI
The recent obsession and progression of AI would not have been possible without big data. Turck even suggests that “AI is the child of big data.” Despite the fact that the algorithms behind deep learning were created decades ago, it was not until they could be applied to mass datasets quickly and cheaply enough that their potential was reached. This led Turck to state, “AI allows big data to deliver on its promise.”
Ultimately, the continuing maturation of the ecosystem will see AI act as the catalyst toward the rise of the application layer of big data. However, we are still in the very early days of the big data landscape. With that in mind, and the continuing discovery of dataset applications, the big data opportunity is bigger than anticipated.
Comment