“We want this to be the database of the future and to last for many decades to come,” Microsoft Technical Fellow Dharma Shukla told me when we talked about Cosmos DB, the new globally distributed database the company is launching at its Build developer conference today. And for Shukla, this is a project that started seven years ago when he started prototyping what a globally distributed (or “planet-scale,” as Microsoft often likes to call it) database would look like. This project (which at the time was called “Project Florence”) first turned into DocumentDB, Azure’s NoSQL database service, which launched in 2015 and is now morphing into Cosmos DB.
Cosmos DB is, in Shukla’s words, “a major leap forward” from what DocumentDB was able to offer. DocumentDB only offered a subset of the capabilities of what is now Cosmos DB. While DocumentDB was essentially a store for JSON data, Cosmos DB goes much further. It extends the idea of an index-free database system and adds support for various new data types that allows Cosmos DB enough flexibility to work as a graph database or key-value database, for example. And for those who are looking to store more traditional columnar relational data, Cosmos DB will also offer support for those.
All of this is driven by a mantra that you’ll almost inevitably hear when you talk to somebody from Microsoft’s developer division these days: “We want to meet developers wherever they are.” So while you could also use the MongoDB APIs to access your data in DocumentDB, Cosmos DB also features support for SQL, Gremlin and Azure Tables — and the team plans to launch a large number of similar driver and translation layers in the near future.
“No data is born relational,” Shukla told me. “In the real world, nobody thinks in terms of schemas — they think graphs or maybe JSON document if you’re an IOT device. […] We want to make sure that the systems we build have a common engine to efficiently map different data models.”
Given its heritage, it’s no surprise that Cosmos DB takes many of its cues from DocumentDB. One of those is the availability of tunable consistency models. If you don’t spend your days thinking about globally distributed databases, then consistency models may not exactly be all that important to you, but most competing database systems (including Google’s recently launched Cloud Spanner), only feature two consistency models: strong consistency and eventual consistency. With strong consistency, whenever data is written to the database, all the different nodes (which could be spread around data centers across the globe) have to agree on a new value before it becomes visible in an app, for example. That comes with some obvious performance trade-offs given the added latency of this. Eventual consistency is essentially a more lenient system where, over time, all the nodes don’t update simultaneously and instead come to agree on a value only after there hasn’t been any recent update for a while.
“What is unusual about Cosmos DB is that it provides different consistency models where the user gets a trade-off between how much consistency he gets over how much of a performance hit he takes,” Leslie Lamport, the Turing Award winner whose work underpins many of these concepts (and who also wrote the LaTeX document preparation system) and who joined Microsoft Research in 2001, told me. Cosmos DB offers three different flavors of consistency for different use cases. “Those kind of intermediary consistency guarantees sort of have been around in academic systems that people build to write papers around,” Lamport explained. Cosmos DB is among the first commercially available database systems to offer this variety of consistency models.
For Cosmos DB (and previously for DocumentDB), that means you can choose between a consistency model where reads are allowed to lag behind writes for only a certain amount of milliseconds, for example, or a model that focuses on offering consistency for a specific client session (in a Twitter-like app, for example), but where it’s not all that important that every user sees every write at the same time (or even in the exact same order).
As Shukla noted, though, his idea here was to build a database system that could last decades. To do this, he also brought in Lamport to teach the team TLA+. Lamport has long had a special interest in how developers spec out their applications. TLA+ is essentially a formal language for doing just that. “When we started out in 2010, we wanted to build a system — a lasting system. This was the database of the future for Microsoft,” Shukla told me. “We try to apply as much rigor to our engineering as we possibly can. […] TLA+ has been wonderful in getting that level of rigor in a team of engineers to set the bar high for quality.” TLA+, Lamport noted, allows you to do the high-level design of a system in a completely formal way — and because it’s done formally, it can be checked for correctness, too (and to be fair, AWS and others also use TLA+ to spec out their distributed systems). “I don’t want to give the impression that TLA+ is great and I’m brilliant. It’s great because it’s almost entirely based on mathematics,” Lamport added.
The Cosmos DB feels strongly enough about its engineering chops that it is also offering a number of SLAs that are somewhat unusual in the database space. While it’s typical to guarantee a certain degree of uptime, Microsoft also offers SLAs for throughput, consistency and latency. “The thing I’m super proud of — and we feel this is the legacy — it’s done with careful craftsmanship,” said Shukla.
It’s worth noting that, as is now a tradition at Microsoft, Cosmos DB has long been used internally at Microsoft. In its previous incarnation as DocumentDB, Cosmos DB currently serves “tens of thousands of customers,” Microsoft says, and stores petabytes of data.
Cosmos DB is now live in 34 Azure regions and is generally available, with all the SLA promises that entails. All existing DocumentDB customers (and their data) will automatically become Cosmos DB users.
Comment