Customers who use the Amazon Web Services (AWS) Simple Storage Service (S3) experienced an outage that kept them from their data for four hours until AWS corrected the problem. The S3 outage took place several weeks ago in the AWS US-East-1 Region in Virginia, which is the oldest and largest AWS data center. Four hours may not seem like a long time except for AWS customers who depend on S3 to keep their data available.
S3 is a public storage cloud operated by AWS. You own the data, but AWS controls the infrastructure. You pay a fee to keep your data at AWS and to access your data. AWS is huge. Consider the scale at which AWS operates its storage service. S3 contains trillions of data objects stored in "bit barns" throughout the world. Customers use S3 for web content distribution, media file storage, backup data, shared data, log data, sensor data, and archived data. You can store any unstructured data you want in S3.
Amazon's S3 is one of the first services AWS offered when the company opened its cloud for business in October 2006. S3 is Object-Based Storage (OBS) technology accessed through the AWS S3 Application Programming Interface (API). The success of S3 in the marketplace validated the importance of Object-Based Storage. S3 put Object-Based Storage on the map, which created the market for S3-compliant storage. The S3 API is now the de facto standard for storing and accessing unstructured data in public and private clouds.
So when S3 sneezes, hundreds of thousands of AWS customers and millions of Internet users get colds. The S3 outage also broke the AWS status dashboard. The company resorted to Twitter to provide status information to customers. Give AWS a failing grade for debugging a problem with the S3 billing system that shutdown S3 and caused other AWS S3 dependent services to stop functioning in the US-East-1 Region.
The good news is customers did not lose data but is S3 good enough? If AWS fails to keep data accessible, is it wise to keep putting all of your data in S3? Your response might be to rely less on AWS by using a competing storage service from Google or Microsoft. Alternatively, you might use a private, S3-compatible storage cloud to keep all or part of your data on premises or in a third-party data center.
It is important to understand that AWS is not your storage architect or storage administrator. AWS operates a public cloud storage service on a pay-as-you-go basis. How to deploy S3 to store and protect your data is up to you. Using "the cloud" as a convenient place to dump data does not relieve you of the responsibility to manage your data, which includes how to keep your data available during an outage.
The AWS S3 outage is an opportunity to rethink your data storage strategy. If you already have data storage and management applications that support S3, you have the choice of using them with an S3-compliant private cloud on your premises. You can keep all of your data in a private storage cloud, or you can migrate data from a private storage cloud to a public storage cloud based on what works best for your organization.
Here are a few things to keep in mind. Public cloud storage is not better than private cloud storage. Public cloud storage is not cheaper than private cloud storage. Object-Based Storage is not rocket science, and you do not need a storage administrator for every 500TB of data. Start with the storage capacity you need today. You can expand your private cloud when you require more storage. There is no rip-and-replace to increase your private cloud storage capacity. Object-Based Storage is something any organization can use even if you do not have petabytes of data to protect.
What is the value of AWS S3 to your organization? Well, the network effects of AWS are huge. Hundreds of third-party vendors leverage S3 in their product or service including those who can provide you with S3-compliant storage on your premises. The more people who use AWS S3, the more valuable it becomes to everyone.
If you want to build or buy your private storage cloud, then choose an Object-Based Storage vendor to work with who can do what you need. More than a dozen companies are in the OBS market. With some research, you can whittle that number down to the three most likely to fit your environment. Every vendor will tell you they are an industry-leading provider of cloud storage. They will promote their strengths and downplay their weaknesses. Ignore vendor appeals based on marketing and make your selection by putting them through a Proof-of-Concept based on a data storage project you are considering.
Remember that having S3-compliant storage is important because the S3 partner ecosystem is a valuable resource. Don’t forget that data is "sticky," and this has implications for its availability, durability, and cost. Armed with your Proof-of-Concept results and cost data, decide what mixture of public cloud storage, private cloud storage or private, hybrid cloud storage will best meet your needs for the long term. Public cloud storage is good enough for many use cases. Just recognize under what circumstances it is an appropriate choice for your organization.