Mohit Aron
As we witness what may be the final days of Yahoo as an independent business, consider how just a decade ago it was running neck-and-neck with Google, now one of the world’s largest companies by market value.
It would be silly for anyone to claim they could have predicted how these two businesses would compare today, but we can still learn something from examining what sent their fortunes in such different directions.
I began working for Google in 2003, at a time when the two tech giants were competing vigorously to dominate the rapidly growing territory of World Wide Web. So many factors influenced the ultimate outcome, but one in particular — the way Google and Yahoo differed in their approach to core infrastructure — seems especially telling.
Perhaps my perspective is affected by the fact that I worked closely on the underlying Google File System, but I still believe Google’s sharp contrast with Yahoo on infrastructure offers powerful lessons about building a sustainable business, especially in the rapidly transforming technology landscape.
Building fast and building to last
At the beginning of the new millennium, Google and Yahoo started down very different paths to attain the enormous scale that the growing size and demands of the Internet economy (search, email, maps, etc.) required. For Yahoo, the solution came in the form of NetApp filers, which allowed the company to add server space at a dizzying rate. Almost every service that Yahoo offered ultimately ran on NetApp’s purpose-built storage appliances, which were quick to set up and easy to use, giving Yahoo a fast track to meet market demand (and soon made the company NetApp’s largest customer).
But in nearby Mountain View, Google began work on engineering its own software-defined infrastructure, ultimately known as the Google File System, which would function as a platform that could serve a diverse range of use cases for all the services Google would offer as part of its future ecosystem. Instead of using the latest storage appliances as a foundation, the Google File System used commodity servers to support a flexible and resilient architecture that could solve scalability and resiliency issues once and for all, simplifying and accelerating the future rollout of a wide range of web-scale applications, from maps to cloud storage.
Scaling complexity
It took four years of ongoing development, and enormous amounts of engineering resources, before the Google File System reached the point where the company used it for mission-critical operations. Meanwhile, Yahoo had been able to add NetApp filers almost immediately to keep up with growing demands for its services. In the race to dominate the Internet landscape, it appeared Yahoo had pulled far ahead.
However, Yahoo’s rapid go-to-market approach also began to show some cracks. As demand continued to expand and diversify, downsides to the appliance-based infrastructure emerged in the form of redundant engineering work, increasingly complex and inefficient environments and finally, mounting vendor costs. When Yahoo added a new service, it needed to re-engineer the NetApp platform for that specific use case.
As a result, identical challenges for separate services, such as Yahoo Search and Yahoo Mail, had to be solved multiple times on different infrastructures. The fragmented infrastructure also exposed greater resource inefficiencies, as each use case required separate server space and compute power that couldn’t be shared across the platform. On top of that, the cost to run NetApp appliances grew as fast as Yahoo did, taking a significant bite out of the company’s revenue.
On the other hand, Google built its file system in anticipation of these challenges, so that adding new use cases or fixing underlying architecture challenges could be done efficiently. After the purchase of YouTube, for example, Google could simply say, “throw away your back-end and we’ll put you on our platform.” Engineers could make upgrades to the underlying architecture once, and the solution would apply across all of Google’s services.
Finally, the flexible platform allowed resources and compute power to be shared across different use cases, so that when servers weren’t busy on search they could be used to process email. It didn’t hurt that all this was built on commodity hardware, which offered costs that decreased in line with Moore’s Law.
As the cost and complexity of Yahoo’s underlying infrastructure mounted, the company simply could not afford to match Google’s pace in developing and deploying major new applications.
The importance of a fresh start
This could be a simple story about the importance of flexible architecture, but I believe the lessons here extend beyond infrastructure or application engineering, and offer insight into what it takes to build a sustainable business. It speaks directly to one of the most important things I’ve learned from my time at Google: the need to completely understand the problem before even considering the solution.
When you visualize a problem, start from scratch. Whether you’re an engineer or an entrepreneur (or both), close your eyes to existing solutions and ways of doing things, ignore what has been done before and build your ideal solution. Once you have that, you can determine which existing solutions should be used and what needs to be rebuilt.
This has been a key aspect of the success of many startups that have displaced legacy enterprises (consider Amazon’s decision to sell infrastructure-as-a-service, which disrupted Comdisco’s previous success outsourcing IT infrastructure on behalf of enterprise clients). It’s also increasingly common at larger companies that don’t want to lose their place to the next young upstart (similar to how Facebook is increasingly building its own infrastructure, from server racks to cameras).
Of course, there are times when the “start from scratch” approach means sacrificing immediate growth for long-term sustainability, which can be a hard pill to swallow, especially in the fast-moving world of Silicon Valley. But quick fixes bring greater risk in the form of growing complexity and inefficiency. Google built a broad platform that extends across the entire web by focusing on simplicity and flexibility, while the complexity of Yahoo’s infrastructure may be the reason it ends up as a small part of another business.
Comment