Currently, I mainly submit picture and movie galleries to large TGP’s such as TheHun, Thumbzilla and the like.Although I have 1 TB (1024 GB) of bandwidth for my hosting account and quite a fast host, bandwidth costs and availability/speed always remains an issue for me. This is especially an issue when I get up to 200,000 visitors in a very short time from a listing at TheHun, for example.

Some weeks back, I heard about Amazon’s S3 (Simple Storage System) which promised to provide unlimited storage. Frankly, this made me a little curious.

What are the main features of this service? Let’s take a look at how Amazon describes S3:

…Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, fast, inexpensive data storage infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers…

Okay, let’s rephrase this so everyone understands it: You have literally unlimited webspace available for your projects.

… highly scalable, fast, inexpensive data storage infrastructure… The S3 data is stored in the same high-speed and reliable data centers as all the Amazon sites and is not free!
… global network of sites… Your data will be delivered from various data centers worldwide. So, no matter where your visitors are, they always get the data from the nearest data center, speeding up the overall speed of your site.

Okay, this concept sound interesting. But what does it cost then?

The pricing is quite interesting:
• $0.10 per GB per month of storage
• $0.18 per GB (up to 10 TB, then it’s $0.16 up to 40 TB, and $0.18 for over 50TB) per month for transfer in and transfer out.

Let’s give an example for a “common” site:

Every month I calculate with about 2 GB of storage (movies and images) and would burn about 100GB of traffic. Given the above examples, costs would be $20.71 per month! By the way, 1 TB (1024 GB) of traffic would cost about $205.50 (prior to June 1 2007).

If you want to calculate the figures yourself, go to the Amazon AWS S3 Calculator and fill out the form. For my example, you’d use the following figures: Storage: 2GB (that’s the files you stored over one month), Data Transfer-in: 2GB (that’s for uploading the files that you store), Data Transfer-out: 100GB.

Remember that uploading (Data Transfer-in) will also cost you money, though most Web hoster do not charge you for incoming FTP traffic!

To use it yourself, you’ll need some special software. Unfortunately, Amazon doesn’t provide a GUI or command-line application for easily transferring files to S3; it’s only a set of SOAP and REST APIs (that’s code snippets for uploading and downloading files through applications/Web services you’d have to program yourself). So, for the average webmaster who just knows how to use FTP/HTML and maybe a little PHP, this is too much to handle.

Fortunately, some guy programmed a great Firefox plugin; S3Fox Organizer for Firefox, which acts sort of like an FTP client within Firefox. Additionally, there are two great freeware applications; JungleDisk (available for Windows/Linux and Mac) and S3Drive (Windows only). Both applications will integrate your S3 drive into your Windows network drives so that you may upload/download files easily, even without having to use an FTP client. Unless you have a Mac or Linux computer, I’d go for S3Drive as it seems to be more reliable and generally faster.

Getting started is easy, by the way. Sign-up for the Amazon S3 service and get your AWS Access Key in order to login to your account. Then you’ll have to create a “bucket.” Think of a bucket as a partition on your computer. With http://inkongruent.com as my website for example, I named it “incongruent.” But you can name it anything you want, just start with a lowercase letter.

Now, if you upload files, you’ll find them under: http://s3.amazonaws.com/bucket/.
For my example it would be: http://s3.amazonaws.com/inkongruent/. I then created a CNAME record for my bucket (ask your hoster if you don’t know what this is):
http://media.inkongruent.com pointing to http://s3.amazonaws.com/inkongruent/.

Why would you do that? Imagine your site is hosted at inkongruent.com but all your images and movies point to http://s3.amazonaws.com/inkongruent/. This would make your content invisible to surfers using certain “banner ad removers” as those ad removers would “think” that the content must be an ad because it’s stored on another machine, as many banners actually are, in order for webmasters to save bandwidth or to provide some sort of visitor statistic.

Now, if you add a movie to your bucket, let’s say; “anal.avi” for example, it would be available under the urls: “http://media.inkongruent.com/anal.avi” and http://s3.amazonaws.com/inkongruent/anal.avi

Never forget, as you upload or get traffic, that you’ll pay for it! Nevertheless, I still think S3 is worth it as some sites that get a lot of traffic can be shut down from their hosting providers. S3 dramatically reduces overall server load and traffic to your site, taking much load from your Web host, even though you’ll pay for it this way.

Do the math and use it when you need it. For example, when you ate your bandwidth for this month, your Web host charged a dollar per additional gigabyte. In this case, S3 is definitely the cheapest solution. You could also mirror your entire site and use it if your site gets slash dotted or “thehuned.” Getting “Slash dotted” means receiving a ridiculously huge amount of traffic from a high authority site such as slashdot.org, which has a Google PR of nine, in a very short time. Many sites, even some that had dedicated servers, got shut down temporarily once a link to them was placed on slashdot.org. Most shared hosting accounts just can’t handle the traffic.

If you want to mirror your entire site, please take into consideration that you can only use static sites. Everything HTML and JavaScript is OK as this is being assembled on the client side (your browser). PHP, ASP, and SSI (Server Side Includes) is being processed server-side, delivering the result of the operations to the browser. As S3 can only store, but not parse, files this would not work. The surfer would get the raw (source code) files and could even for example take a look at your MySQL account if he manages to load the right file. So, dynamic content (PHP, ASP and the like) is a no-go.

Some webmasters also talked about long latency of up to one second. I personally never had this problem, although I can believe that this may happen. I advise you to periodically ping some files to see how fast they can be reached/the service responds. If you want to test the speed of the service, just download the SecondLife client, as this file is being distributed by S3. Unfortunately, I can not offer you a big file myself as every download would cost me money (even it it’s only very little).

I even use it to store my DivX movies and find it much cheaper compared to buying another external hard drive.

If you have any questions regarding this article, just mail me as inkongruent at gmail \./ com. I check my mail multiple times a day and will answer you… if you don’t insult me of course 😉