Dean Peterson
Using Amazon for everything feels wrong to me — a bit like the rebels in Star Wars renting a Star Destroyer from Darth Vader rather than recruiting the Millennium Falcon to fight the Empire.
“Raaaw, don’t run stateful apps in a container, Raaaw.” That is what currently stands for sound advice in the development/operations world. I hear it parroted over and over. I get it; running stateful applications in a container is hard. However, nobody should be paying you to spin up 1,000 stateless containers to print “hello world” then click a button in Amazon for the hard stateful parts.
You give yourself an out by saying you are following the guidelines of the “12 Factor App.” That tells you writing applications born in the cloud should attach stateful services separately. All you really need to own is dynamically scaling that amazing stateless “hello world” app you wrote using React and Node.js.
Amazon, MongoLabs, Rackspace, Microsoft; they all whisper sweet nothings in your ear. “Run along child, go play with Angular 2, Node.js, React; give Spring Boot, Rat Pack and Polymer a try. Leave the hard stateful parts to us.” You agree and start putting everything in Amazon’s cloud. They get you hooked on their drug. Then, when it is too late, you learn the real cost of “all your data are belong to us.”
Let me back up a minute and tell you why I am writing this and what I want (short version: I want Google’s Kubernetes, Docker and OpenShift to step up their game). I am working on a rewrite of a large-scale application at my place of employment. We are fed up with the complexity that has grown inside a monolith and have decided to pursue a micro-services architecture. The promise of smaller, more maintainable apps keeps me working all hours day and night.
This is how things progress. I learn Angular 1, Bootstrap, Grunt, the JBoss stack: REST services running in Wildfly’s JEE7 app server, and I secure everything with a central OAuth/OpenID Connect provider to make it easy to share user credentials in a number of separate applications.
Yes, I know JEE7 sucks and Angular 2 breaks backward compatibility with Angular 1 (not the focus of this article). And yes, I was once a native JavaScript fanboy, so I know frameworks like Angular are just awful too. I don’t really think they are awful, but I do think the slowness on mobile is. However, that is a conspiracy for another time (I think it stems from the unnecessary lack of memory and storage space on mobile devices so Google and Apple can sell cloud storage for your 20 billion baby pictures).
Back to my point (I think I have one).
Inevitably, we realize reduced complexity in app development is traded for increased complexity in operations; all of these services and separate clients and databases need to be built, deployed and managed in a sane way.
I dive head first into the “DevOps” world. I learn straight Docker and use Docker Compose to orchestrate running containers for app services, clients and a single MongoDB node. Plus, containers for etcd (name value storage for ip/ports), registrator (used to keep etcd in synch), nginx (load balancing services) and confd to reconfigure nginx dynamically as containers start and stop. “Yes, I am amazing!” At least, that is what I think until I realize Docker Compose falls very short of making easy the management of those containers.
I do not attempt to scale the setup with Docker Swarm. There are too many moving parts in the makeshift PAAS to justify selling it to management. If the complexity is too much running containers on one node, throwing Docker Swarm into the mix I imagine to be a nightmare.
At this point, my notion is that stateful applications, such as MongoDB, should run in containers alongside stateless clients and services. I didn’t know that makes me foolish. Like a naive dreamer, I thought the value of containers was to make your entire app easy to scale. I was unaware the community had already given an official name to their surrender (Backing Services).
With some Docker experience under my belt, I take another look at the Docker PAAS landscape. I need something a bit more robust than Docker Compose and Docker Swarm. How do I choose? There is Kubernetes, Fleet and Mesos. But wait. I don’t use those directly? They are for the use of platform developers?
Eventually I understand; they are the technology behind a number of other platforms meant for use by me (the developer). I need to choose my platform after I choose from the lower level technology. Kubernetes, being from Google, has a huge following, is open source and plays well with the Docker knowledge I already have. I choose Kubernetes.
Now I need to choose a Kubernetes platform. I already use Red Hat Enterprise Linux to run Docker containers and some of the JBoss stack, so I choose OpenShift Origin. I spend weeks learning OpenShift. I spend hours communicating with OpenShift developers and finally get everything I had running with Docker and Docker Compose running on OpenShift Origin.
Except now I have a build and deployment piece, images automatically get pushed to a registry, I can dynamically scale up and down pods, easily, with a single line command and load balancing is baked into every service I create. Life is good. Except I am still stuck on scaling my stateful MongoDB service. Every PAAS has templates for running a MongoDB database. They all stop at the easy part and leave the hard part to us.
If anyone has used MongoDB, they know the hard part is getting a “production-ready” MongoDB instance. That means at least running a MongoDB replica set. For a truly production-level configuration, a sharded cluster is required. It gets complicated quickly. There should be a template for doing that in someone’s platform. There is not. Right now, OpenShift is blocked because Kubernetes needs to figure out how to attach separate persistent volumes to pods in the same service. They have made that a Priority 2. It should be a Priority 1.
This makes me seriously consider switching. Tutum is bought by Docker. OK, maybe I go back to the Docker camp and use Tutum. No, they also stop at the easy part and only have an example for a single development MongoDB node. They are still trying to integrate with Flocker, which means they are nowhere near solving the problem.
All right, I consider leaving the Docker container world entirely for stateful services because I find an Ansible template to do what I want. However, it is out of date and only works with Red Hat 6. That is the problem with Ansible; Ansible playbooks depend on their environment. Docker images are portable and I know they will run.
I want a tool that has a repository of templates for making formations of very hard things easy. If that existed I would not need Amazon to do every hard thing for me. I can mostly get what I want on Amazon. But look at the prices:
They get you hooked for free and the next level is $1,496 per month… wtf! MongoLabs is little better. I don’t understand why everything is becoming an exorbitantly priced service. Keep in mind, platform as a service providers should have templates for hard-to-set-up stateful services like MongoDB.
Rather than use templates from IAAS providers, I should be able to use someone else’s platform running on my chosen cloud/IAAS provider (call it what you like), like Amazon, to put pressure on them to lower prices. If they know it is easy to leave, their prices will go down. Let’s take back some of our ability to do hard things wherever we choose so that running a database does not cost … wait, what, $21,430 per month?
Please, tell me I’m crazy. Tell me why I’m wrong. I sincerely want to know. So many products are inches from being able to compete with Amazon and give choice back to developers whether they want to run on Amazon or retain a bit more freedom (perhaps even run in another cloud or local cloud environment). Soon, Kubernetes will allow complicated stateful services to run inside containers.
Meanwhile, Amazon announces competing components daily. Amazon has API Gateways, CloudFormations to spin up almost any stack or service, CodePipeline for continuous delivery, load balancing; you name, it they have it.
Now, you might be wondering, what is the problem? Just go with Amazon, everyone is doing it. “You aren’t cool unless you’re using Amazon.” I do work for a large organization that can afford to run everything on Amazon (maybe ; some would disagree). However, I also work intensely on the multi-way trading platform Abecorn.com. That has no VC funding; I have learned to “do more with less.”
Using Amazon for everything feels wrong to me — a bit like the rebels in Star Wars renting a Star Destroyer from Darth Vader rather than recruiting the Millennium Falcon to fight the Empire. Amazon spins up virtual machines (Amazon Machine Images) for many services. They should be using containers to increase density and lower costs. Besides that, there are features missing in the Amazon container platform.
Native Amazon container support comes in two flavors: Amazon EC2 Container Service and Docker on AWS Elastic Beanstalk. Docker on AWS Elastic Beanstalk is more equivalent to a PAAS and uses Amazon EC2 Container Service under the covers. That offers a way to run and automatically scale containers; Amazon does not offer a way to seamlessly build and store container images. You must pull pre-built Docker containers from a third-party registry because the Amazon Container Registry is not yet available.
Plus, there is no Amazon tool designed specifically to build containers. Combining layers of Docker files to create a complete container image is no trivial task. A number of platforms, including OpenShift, already have container build configurations to make the process easy. Another crucial component is a shared file system for Docker containers.
Right now, Amazon’s Elastic File System is in preview mode. You have to ask permission to use it (I am still waiting). OpenShift already allows me to use a number of different shared file systems, including NFS (Network File System), which Amazon Elastic File System is. It is useful to have the ability to share persistent data between containers (potentially running on separate nodes in a cluster) using NFS mounts. A Network File System would allow Lucene indexes stored on the file system to be shared between containers spanning multiple nodes.
I have run containers on bare metal and in Amazon. Performance can be noticeably slower in Amazon on machines with similar specifications. A detailed explanation why is in this informative video by Bryan Cantrill. It is a talk about how layering containers on top of virtual machines is a recipe for poor performance and wasted resource utilization. He equates it to developers “being given fire, then proceeding to put the fire out.”
All Amazon machine images are virtual machines. When you run containers in Amazon you are running them in virtual machines. New providers are available that let you run containers on bare metal to maximize their performance. They can provide that service economically due to the greatly improved density of applications running in containers.
I am all for “the cloud,” but the cloud does not necessarily mean All Amazon, All The Time. Amazon is actively blurring the line between IAAS (Infrastructure As a Service) and PAAS (Platform As a Service). It is only recently that they have introduced newer developer-focused DevOps tools like EC2 Container services and Docker on AWS Elastic Beanstalk. Just because they are accepted as the dominate provider of IAAS should not give them uncontested dominance in developer-centric platforms. It is OK to use Amazon’s infrastructure, but give yourself the flexibility to move by using a portable PAAS like Kubernetes.
If not, soon, we all may be waiting for Amazon to come out with the missing service we need. Let me give you a glimpse of the future.
Example conversation between Bob the architect and management.
Management: Bob, how do you suggest this application be written?
Bob: Amazon
Management: But what about…
Bob: Amazon
Management: Yeah but…
Bob: Amazon
Management: Genius Bob, you’ve done it again. I have what I need, let me run this up the flag pole.
Comment