Storage Field Day 15: Dropbox the high availability in a “pocket”
How many of you are storing personal and professional files in a cloud storage? How many of you are sharing the same files with mates, colleagues and business people. At Storage Field Day we had the pleasure to meet Dropbox, one of the most important cloud company which starting with the idea of “sync and share” your documents, images and files in general, becoming one of the leader in Gartner Magic Quadrant for vision and execution.
Dropbox is a big company situated in San Francisco. It counts about 500 mln registered users with more than 300k of business customers, more than 1 EB of stored file and a high resilient service distributed over the entire world.
The Dropbox’s idea was really disruptive for the ability of provide roaming backup and an easy way to share files without limitations. Now integrations and services around this is the challenge that
They were one of the biggest customer for Amazon but in 2016 something weird happened: they moved in their own data center leaving Amazon AWS. They realized the biggest data exodus of the last times story with a redefinition of their software.
Changing direction
In my previous posts, written after Cloud Field Day 2, I exposed the lift-and-shift trend as a consequence of cost/benefits re-calculation and the opportunity to break boundaries with cloud providers. Obviously this needs affects principally the mid-ent sized company, while small companies and startups are keeping themselves in the cloud to reduce TCOs and/or offload many activities that are in the edge of the principal activity, like infrastructure maintenance and datacenter operations.
Dropbox is not an Infrastructure cloud provider, they believe in the cloud paradigm since they are selling storage as a service. They were in AWS because the focus of the company was not keeping datacenter cost and infrastructure integrations. For this reason choosing in a high integrated platform, like AWS, was what they needed or believed since new cost vision redefinition (since 2015).
What happened? Why they are gone away? The answer stays in the performance/money ratio including some other facts:
- Watching the workload cost and the forecasts for next years (the growth is quite constant to 20%), it is possible that as-a-service formula, is not so affordable as expected. In this state of Dropbox life the TCO is not a problem, because the company is already running with stability and vision. for this reason the equation cost-performance-agility is moving to the infrastructure acquisition.
- From tech side, Dropbox wants to store data and make available data with services for their customers. AWS is selling workloads, S3 buckets with a high level integration. These two companies are doomed to take different directions, because the visions are different: Dropbox gain is given by the ability to store data and make them available with services that are not following in term of costs the amount of EC2 instances or get/put into S3 bucket.
- Focusing on the data, not all people knows the genesis and the mechanism adopted before: only the “heavy part” of the data were stored in S3; the metadata were housed in datacenter. This give a right sense to the operation: they previously were suffering for the AWS overhead; if you’re jailed in a platform with rules and routes, you’re less efficient than having owned infrastructure.
The Magic Pocket
Magic Pocket is the release name of the actual Dropbox version. Dropbox coders and integrators has built, moved and improved the whole platform without compromising the availability:
- Make own hardware solution to improve the performances.
- Code the new release testing a small copy of production data
- Move data and services out of AWS
They based the whole system infrastructure (about 50K systems) on Ubuntu without the use of virtualization or known software defined solution.
Quite all cloud providers use virtualization as fundamental block to realize a software defined solution enabling the multi-tiering and being more efficient than a physical infrastructure. But the amount of data and the different kind of business from infrastructure reseller brought Magic Pocket a huge software defined storage without consuming underline software defined entities. They simply place their stack upon physical infrastructures and organized in: zones, and cells.
Another good point is the extreme high availability system that runs in continuous verification and redistribution of the entire workload and data to avoid data and service loss. The only way to ensure unpredictable behaviors during the fault, is procure a fault: Dropbox, taking the advantage to be a software company, realizing a true continuous resuming system with also an elevated automation that gives back resiliency in more than common failure situations.
During the conversation with Dropbox I was really surpise how they dedicated a lot of resources in networking to guarantee transfer performances and ensure the accessibility in all countries of the world. The improved edge acceleration is ensured by 19 PoPs spread across the world.
This must be a lesson that every cloud provider must learn, because service without network availability means a downtime even if data and computing are still on.
A lot cloud architect are focusing every day in storage/computing stuff on top on SLA, forgetting that if access connectivity goes down, you are registering a downtime like being a computing loss!
Video and Links with further informations
Wired – The Exodus from cloud empire: https://www.wired.com/2016/03/epic-story-dropboxs-exodus-amazon-cloud-empire/
Dropbox blog about Magic Pocket: https://blogs.dropbox.com/tech/2016/05/inside-the-magic-pocket/
Disclaimer: I have been invited to Storage Field Day 15 by Gestalt IT who paid for travel, hotel, meals and transportation. I did not receive any compensation for this event and None obligate me to write any content for this blog or other publications. The contents of these blog posts represent my personal opinions about technologies presented during this event. Because I signed NDA I’ll not provide any information about the datacenter and the place I visited except the conversation available on web (youtube channel).