Now that you know the basics of Amazon S3 and are equipped with tools to manage files on S3, let’s focus on how data gets stored on Amazon S3 using Buckets.
Just like a bucket holds water, Amazon buckets are like a container for your files. You can name your buckets the way you like but it should be unique across the Amazon system.
The suggestion is to follow a domain naming convention, like downloads.xyz.com or media.xyz.com.
This is recommended because xyz.com is very likely to be unique and this also lets you easily map your existing web domains or sub-domains directly to Amazon S3 by adding/changing your DNS CNAME entries.
So in the example above - media.xyz.com bucket on Amazon S3 will correspond to a web address like http://media.xyz.com.s3.amazonaws.com/ while download.xyz.com will correspond to a URL like http://download.xyz.com.s3.amazonaws.com/
Now if you upload an image (say animals.jpg) and drop it inside the media bucket (media.xyz.com) - that image can be accessed on the Internet using the following address:
http://media.xyz.com.s3.amazonaws.com/animals.jpg
So you can map the media.xyz.com domain to http://media.xyz.com.s3.amazonaws.com/ and then your image can be accessed using the simple URLs like media.xyz.com/animals.jpg instead of using that complex Amazon URL - http://media.xyz.com.s3.amazonaws.com/animals.jpg
Later, if you move from Amazon S3 to another cloud hosting from say Microsoft or Google, just change the CNAME entries and none of the URL will break.
Another important part is that Amazon S3 defaults file access to READ/WRITE only by the owner and thus other people may not be able to access your files unless you set the right file attributes (something like chmod in Unix).
For publicly accessible files (like web images on your blog or public downloads), you should set them to “World:Read” and “Owner:Write”. You can set the File Attribute after you upload a batch to S3 or change the default setting.
Why You May Want to Use Amazon S3 Storage ?
There are two reasons why you may want to host files with Amazon S3 - it’s very reliable and you pay only for what you consume (see pricing & S3 calculator).
S3 pricing is something like this - you pay 15 cents per GB-Month of storage and 20 cents per GB of data transferred (applicable when you are uploading or downloading data from Amazon S3).
I use S3 primarily for hosting media files (images, audio and video) and here’s how Amazon bills me:
* I’ve about 1GB of media files stored on S3 = $0.15 /mo (storage)
* Suppose I upload about 0.2GB (200MB) each month = $0.02 /mo (data transfer in)
* Last month, my media files consumed bandwidth of about 25GB = $4.25 /mo ($0.170 /GB /mo for the first 10TB)
* My upload (PUT, POST, LIST) requests are less = $0.01 /mo (for the first 1,000 requests)
* My download (GET) requests are very less = $0.01 /mo (for the first 10,000 requests)
Thus, the total cost at the end of the month would be $4.44! Let’s assume that I follow this pattern (even increasingly on the upper bracket) for the next 10 years, I’m likely to pay just about (4.44 + 4.5) = $8.95, calculated that I increased the storage to 30GB at the end of 10 years (roughly about 200+ MB a month). Even if my bandwidth consumption multiplies to 1TB, I would still be paying just (8.95 + 17.0) = $25.95 in the year 2018.
So, storing about 30GB of files with a bandwidth consumption of 1TB a month on a highly scalable, durable hardware, maintained by experts at the cost of just about $25 a month; that’s just awesome.
The writer is an Adobe Community Expert and an ardent believer of KISS. Brajeshwar writes at Brajeshwar.com and recently launched oCricket.com - a social platform for people who love the game.
Find this article at: http://www.labnol.org/internet/tools/amazon-s3-buckets-tutorial/3890/
web: http://www.labnol.org/ email: amit@labnol.org
Reader Comments
Great post Amit - very useful information. You should stress to your readers that if you are doing to use DNS to access your buckets, then the bucket names HAVE TO BE lowercase, otherwise S3 will not recognise them in certain cases. The only workaround is to change your HTTP headers from:
GET /photos/myphoto.jpg HTTP/1.1
Host: MyBucket.s3.amazonaws.com
which won’t work, to:
GET /MyBucket/photos/myphoto.jpg HTTP/1.1
Host: s3.amazonaws.com
Written by Devan on 12.15.08
You say:
“Another important part is that Amazon S3 defaults file access to READ/WRITE only by the owner and thus other people may not be able to access your files unless you set the right file attributes (something like chmod in Unix).”
But you don’t say how to change these permissions!
Written by Chris Cohen on 02.05.09
Are you aware that you made a mistake in calculating the price of 1TB bandwidth? In the article you say 1TB is 8.95. Which is actually only 100GB of bandwidth. 1TB = 1024GB. 1TB of bandwidth actually costs you $174.08!
Written by Alan on 03.04.09
Amazons documentation left me scratching my head, but your post cleared it all up, thanks a lot.
Written by Stefan on 03.06.09
I’ve got my Amazon S3 account, uploaded my media, but it seems I can only access them with the pre-signed urls, which makes this approach very cumbersome, as my images are loaded dynamically from a database. I can change the prefix for the images, but adding the rest of the url string with an access key, signature, and expiration date means I’d have to do this manually for each image. Is this the only way?
Written by Kevin on 04.03.09
Very helpful posts about setting up s3 service thank you. So i’ve managed to get the s3 service running on my site no problems. But do you have any idea of to serve assets on to secure pages (https) without getting that pesky popup in IE and the line through the padlock in firefox?
It happen because the links to my s3 buckets are all http: - Have you managed to do this for https:
Written by Alex on 04.18.09