Last week, I moved all the common web images, CSS, JavaScript and other static files of this blog to Amazon S3 Storage service and that alone reduced the average CPU load / memory requirement of the web server by almost 90% – see graph.
Why Use Amazon S3 Storage for Hosting Files
There are multiple advantage of hosting images on Amazon S3 – your site’s downtime is reduced because there are lesser number of concurrent connections to your main web service (and hence lower memory requirements) and two, the site’s overall load-time is reduced because static images and other files are served via the more efficient content delivery network of Amazon
How to Host Images on Amazon S3 Storage
Let’s assume that you have an account at amazon.com (who doesn’t have one) and you want to use the sub-domain files.labnol.org for hosting images that in turn are stored on Amazon Simple Storage Service.
Step 1: Go to Amazon.com and sign-up for the S3 service. You may use the same account that you created for shopping on the main amazon.com portal.
Create CNAME Record for Amazon S3
Step 2: Login to the control panel of your web hosting service and create a new CNAME record - we’ll set the name as files (same as sub-domain) and assign it a value of files.labnol.org.s3.amazonaws.com (for details check this article on Amazon S3 Buckets).
Step 3: Install S3 Fox – this is my favorite Amazon S3 client though it works inside Firefox. Check this S3 Guide for a list of other popular S3 clients.
Step 4: Now we’ll associate S3 Fox with our Amazon S3 account. First go here to access your secret Access Key ID. Then click the S3 Fox button in the Firefox status bar and complete the associate via "Manage Accounts."
Create folders to host files
Step 5: In the "Remote View" tab of S3 Fox, create a new folder that has the same name as your sub-domain. Drag-n-drop all your images, static files and other folder from the desktop into this folder and they’ll automatically get uploaded to your Amazon S3 account.
Amazon S3 Upload Queue
Step 6: This is important. By default all files uploaded on Amazon S3 are accessible only to the owner but since your are hosting web pages for a public website, anyone should be able to read these files.
Set File Permissions on Amazon S3
To change the default permission, right-click the main folder files.labnol.org and choose "Edit ACL". Now select "Read" for "Everyone" and "Apply to All Folders".
Step 7: This is optional but if you don’t like your web pages to be crawled by Google and other spiders, you may create a robots.txt file and place it in root directory.
User-agent: * Disallow: /
This may be a good idea because Amazon S3 charges you for every byte of requested data so you may block web bots and thus reduce your overall bandwidth bills.
Is Amazon S3 More Expensive Than Your Web Host
Here’s a detailed report of my Amazon S3 usage for one week. I’ll have to shell out around $28 per week or a little over $100 per month.
Itemized Bill - Amazon S3 storage
Now DreamHost Private Server hosting used to cost me around $150-200 a month but after moving the images to Amazon S3, that charge has reduced by around 60% so the total monthly cost of hosting website + images still remains the same.
Find this article at: http://www.labnol.org/internet/host-images-files-on-amazon-s3-storage/4923/
web: http://www.labnol.org/ email: amit@labnol.org
Reader Comments
Nice tips. There was a major downtime with Amazon S3. However, is there any need of altering the codes in the Web site?
Written by Lighting Fast on 10.13.08
Hi Amit,
On Dreamhost which service are you using, unlimited space and bandwidth or a dedicated server?
I am also hosting domains with Dreamhost and just need to know about plans you have purchased.
-Heeren
Written by Heeren Pandya on 10.13.08
currently i am using my wordpress installation directory for hosting images, if i moved to amazon s3 storage so is i have to chage the path of my all old iamges to amazon s3 or it will map automatically to amazon s3 ?
do u have any idea when amazon rolling out it;s cloud services for windows operating system ?
Written by Ajay on 10.13.08
Brilliant!!!
Thanks for sharing Amit!
Written by Syed Abbas on 10.13.08
Thank you Amit.
I think Amazon’s price is unbeatable, but the bad side is the payment. As far as I rememeber they don’t allow PayPal? Isn’t a credit card required?
Also, is there a way to only allow requests from your web server ip? I mean, imagine someone is requesting a file / image that you are hosting with Amazon 100,000 times… this would increase your bill…. :|
Written by Marco on 10.13.08
Wow, interesting concept. Hopefully one day I’ll generate enough traffic in order to force me to start doing this.
Written by TheWindowsFix on 10.13.08
@LF You need to change the URL of your CSS and web images so that they point to Amazon S3 buckets or your new sub-domain.
@Heeren I am on a private server of dreamhost so don’t think there’s a limit on bandwidth or storage.
@Ajay You may consider reading about .htaccess based redirection so that your image folder base points to Amazon S3.
@Marco I think you require an International Credit Card to pay your Amazon S3 bills. Not sure if you can block requests based on IP but there may be workarounds.
Written by Amit on 10.14.08
Great post Amit !!!
It’ll be interesting to see a comparison of having an image server on say GoDaddy vs. having them on Amazon S3…
What was the performance comparison like?
Written by Sumedh on 10.14.08
thanks amit for your reply as of now i am only having 2000 unique visitors per days so is it a good idea to host all my iamges and static file on amazon s3. ?
Written by Ajay on 10.15.08
That’s a good guide on using S3 - the S3Fox extension is getting very popular (atleast on non-Mac platforms).
Amit, I’m curious to know why you chose Dreamhost, at a steep price, when other providers (like MT and my personal favorite, Slicehost) were available with enough resources and at a lower price. Some of the most trafficked blogs (especially from social media) - Zenhabits, TUTS group of sites etc. are on Slicehost and none of them have ever had a complaint.
Ok, maybe I sound like a marketer for Slicehost, but the point is, is there any specific reason to chose DH? Surely you didn’t think this blog wouldn’t grow as fast(or did you?)
Written by Sumesh on 10.19.08
I would like to share an additional tip. Amazon S3 also charges by number of GET requests ($0.01 for 1000 req) for files even when it returns 304 Not Modified headers.
I am developing apps on Social Networks (www.bitrhymes.com) and we currently have ~3M users with about ~400K daily returning visitors. Thats a lots of traffic and S3 was the obvious choice. But soon we found that the bill was growing. When I digging in HTTP headers, I found out that by default S3 doesn’t have any cache request headers set. So its wise to set request long headers for files that don’t change frequently, like the site logo or images inside the post (also js and css) so that people browsing from one page to another inside the website don’t contribute to multiple get requests.
We set cached headers (like Cache-Control max-age=864000) etc. and we found our S3 bill down by like 40%
Unfortunately S3 Fox doesn’t allow you to set custom headers while uploading and I am not aware of any tools in the market. I upload it using a modified S3 PHP REST library.
More tips about speeding:
link
Written by Lalit on 10.21.08
Talking about post Amit only, if you spend ~$200 a month on hosting then best choice will be a dedicated box, in that price you can even get a nice Managed Dedicated Server.. And then you will be the Boss of site..
Regarding post Lalit.. Amazon s3 is good but going with some clustered hosting and spreading among various network and world location won’t cost you much but yes.. Expertise is required.
Written by Ricky on 01.25.09
Hi Amit,
How can i host my website on amazon s3.
Please send me step by step guide, if possible.
I want to host my site to amazon
Written by Kiran on 02.26.09
Heloo Amit thanks for sharing this nice article
i have question for you
I’m trying to use S3 for hosting my images. I have created a bucket and pointed my sub domain to that bucket but when i directly enter bucket URL in browser it shows complete list of files in xml format. Is there any way to hide this listing from browser?
I have noticed your S3 bucket img.labnol.org/ shows ur index page and doesn’t show file listing how can i achieve this too for my bucket.
I will appreciate your early reply
best wishes
Written by Ghulam Rasool on 05.17.09
Ghulam - I think you have set the main S3 folder as “public” and hence the sub-domain is showing the entire list of files in XML format. Maybe you should provide public access only to individual files, not the folder.
Written by Amit on 05.18.09