I just spent the last ten days trying to do an import on Amazon S3 using their Import/Export service. Basically Import/Export allows you to send a drive to Amazon via snail mail and they will hook the drive to their system and import the data locally. It’s much faster than trying to upload the first snapshot of the data via the good old internet.
The problem is that the service requires you to produce a manifest file (to let them know what bucket to load unto as well as some other information) and a signature file on the unit itself that verifies that you are the legitimate owner of the process.
The official documentation can be read here, but I will give you the “real” how to.
The first thing you do is to make a manifest file. Instead of writing the file, I opted to use the S3Fox Firefox extension, which writes the file for you. This is a problem, and I will explain to you why later on.
You will be better off writing the file yourself, and it’s not that complicated. Just write the following:
I added a couple of extra variables.
Because I wanted the files to go into a specific path, I added the “prefix: myownpath” so that data is sucked in there. The “deviceid” is the serial number of the unit you’re shipping.
I also listed the types of files (which is kind of weird, but Amazon requires it). Here’s the list I did:
contentTypes:
flv:application/x-shockwave-flash
mov:video/quicktime
wmv:video/x-msvideo
mp4:video/mp4
pdf:application/pdf
ai:application/postscript
zip:application/zip
mpg:video/mpeg
mp3:audio/mpeg
psd:image/psd
png:image/png
jpg:image/jpeg
gif:image/gif
Now here’s my nutshell of advice to y’all: Make sure you create this file on a Linux/Unix environment and not using Windows programs or S3Fox. Again you’ll see why later. Create this file using vim or emacs and then save it as “manifest.txt”.
Send an email to “awsimportexport@amazon.com” with the subject “CREATE JOB” (all caps) and attach the manifest.txt file to the email.
Amazon will take its time and then reply back with something like this:
SUCCESS
JOBID: J5N9M
deviceId:97333A9360CC3
You will now need to create a SIGNATURE file. Download the signature generating tool from Amazon and put it on /tmp or create a folder and unzip it.
Go to the directory and run the following command:
java -jar lib/AWSImportClientSignatureTool-1.0.jar manifest.txt J5N9M SIGNATURE
As you can see, you need the JOBID that Amazon sent on the email. The “SIGNATURE” is the name of the file. Leave it like that, all caps.
The script will ask you for your Amazon AWS Secret file. Copy and paste it and then it will generate the SIGNATURE file. Copy that to the root folder of the unit you will ship.
Now you just need to print out a shipping label to put inside the box with the storage unit, and UPS/Fedex to Amazon. Here’s the documentation for that.
You should get an email when Amazon gets your unit and starts the importing.
So for the HOW-NOT-TO-DO-IT guide: I created the manifest file using the S3Fox tool and saved it on my Windows Desktop. I opened it using Notepad to double check everything was fine.
I sent the CREATE JOB email and received the JOBID. I then used S3Fox again to create the SIGNATURE and shipped the unit.
When I got the email from Amazon that they had received the device, it had a failure: The SIGNATURE file was invalid.
I had a back and forth with AWS for 5 days. They were really helpful (Thanks Chesley and Eric!) but I wasn’t clear what I was doing wrong. It turns out that when the manifest file is generated and saved on Windows (or opened in Notepad) Windows will insert character returns at the end of each line, and the SIGNATURE generated in S3Fox takes this into consideration. But when Amazon processes the manifest, it strips out these character returns and generates a SIGNATURE that, when compared to the one you send, is not the same, and thus marks as invalid.
So make sure to just do the whole process on your Linux terminal and not on your Windows GUI and you’ll have your backup done in no time.
{ 2 comments }
