For the past couple of days I've been working on setting up a GoGrid MyGSI image (the GoGrid equivalent of an Amazon Machine Image - AMI) for our app. The benefits are obvious - instead of manually preparing each new instance with the right apps, roles, configuration etc..., create a bootable image. This would speed up deployment time and would be invaluable whenever scaling up our grid horizontally or vertically.
I've been doing this in Amazon EC2 for the past 18 months. For those not familiar with that service, any EC2 instance can be bundled up and saved to S3, then registered as a private or public image for future use. On Linux, this requires a couple shell commands. For windows, you can use the EC2 console, a nice little web GUI. A couple clicks, that's it. Very convenient. Based on my experience, AMI bundling and saving usually takes about 10 minutes, less for a thin Linux image, more for a heavy Windows setup. So far, I haven’t noticed a difference between the original instance and the one started from the bundled image, other than public & private IP addresses. Admin passwords were restored, and everything seemed to be working properly.
GoGrid does things slightly differently. In order to create an image, you are required to start a GoGrid SandBox from an existing image. The sandbox is not intended to be used in production. It uses 2 GB RAM minimum, so leaving it on for an extended period will cost you. You’re supposed to start the sandbox, install whatever you need, windows updates, etc… then run a prep script, wait a few seconds and save the image using the GoGrid admin panel.
Once the image is saved, it shows up in the list of available images when starting a new GoGrid instance.
At first glance, this is not a big deal. But keep in mind that starting a sandbox takes about 10 minutes. Running prep and saving the image, another 15 to 20 minutes (I was using a 4 GB Win 2008 image). Then starting a new instance takes another 10 minutes (the virtual server looks ready after about 7 minutes, but it really takes another 2 to 5 minutes before you can connect to it). So updating your image and deploying it will take you roughly 30 to 45 minutes. You better be sure your image has everything you want on it, because before you know it, your whole day’s gone by the time your instances are really ready.
Now here’s my other gripe, and the reason why I had to update my image about 8 times already. Instances started from the saved MyGSI image are slightly different than the sandbox you were working on. The prep script does something to your image to make it bootable as a GoGrid virtual server. Here’s what I’ve noticed so far:
- The administrator password gets reset every time a new instance gets created. Retrieving the password is easy, it shows up in the GoGrid passwords panel. But if you’re relying on identical username / password combinations between your instances for network communication & security purposes, you’ll need to rely on a different account. The good news is you can create a new user account on your sandbox, and that one won’t get modified by the prep script. It gets properly restored, and the password doesn't change.
- Much more troublesome, GoGrid made the decision that by default, users are not interested in having their instances share a common private network. I don’t know why; it seems completely baffling to me. You get a private cloud storage space with your GoGrid account, but by default any new instance won’t be able to connect to it. To get this to work, you need to add a route, enable the second adapter, assign a static IP address, then mount the cloud storage account using the provided username & password, which can’t be modified. That’s about 3 manual steps required for every new virtual server that might need to connect to other virtual servers in your grid or to your cloud storage space. And you need to keep track of your static IP addresses in a separate spreadsheet, GoGrid won’t help you manage that. When you need to deploy a new instance quickly, that’s a real pain in the neck. You can create the route on your MyGSI sandbox, and it will be restored. However, configuring the private network adapter won’t help, the prep script will always disable it. So I strongly recommend the following:
- Start a cheap 512 MB instance and set it up as a DHCP server on your private network.
- Add the cloud storage route to your custom sandbox.
- Configure the MyGSI sandbox private network adapter to use DHCP
- Create a batch file on your MyGSI instance that will do the following:
- enable the network adapter:
netsh interface set interface name="Local Area Connection 2" admin=ENABLED - wait for a 30 seconds to let the adapter find your DHCP server (I found this trick to add a “wait” batch command on windows)
- optional – you might wan to force an IP address refresh by doing
ipconfig /release “Local Area Connection 2”
ipconfig /renew “Local Area Connection 2” - mount your cloud storage space
net use Z: \\[accountId].cloud.storage.gogrid.com\[accountId] [accountPassword] /USER:[accountId] /Persistent:Yes - Put that file on your MyGSI image, and set up windows scheduler or a RunOnce registry key to run the file on computer startup.
- After booting up my MyGSI image, SSL wasn’t working on my instance. Turns out that creating the instance broke my certificate. I don’t know if it’s a machine key problem or what, but the cert private key was gone. So I put a .pfx export of the certificate on the MyGSI image and added one line to my startup batch file:
- certutil –importpfx myFile.pfx
note: you don’t have to use a PFX file, look at the certutil help for more options
So now every new instance I create is automatically setup for private network and cloud storage access whenever started up or rebooted. Much, much better. Why GoGrid instances are not always set up this way is beyond me. I would expect GoGrid to provide private IP address assignment the same way Amazon does, and pre-configure the network adapter and cloud storage mapping. What’s the point of creating a grid if each node can’t communicate with the others? Cloud storage access, distributed caching & DB access are all basic requirements for any cloud app, and all require a private network. The GoGrid approach is way too cumbersome.
This is all well and good if your image doesn’t change very often. But we live in a beta world, and app deployments can be almost as frequent as nightly builds. Rebundling an image for each deployment and launching new instances from it every time you make a change to a web page is not exactly practical.
I’ve been using wowza media server for EC2 for a year, and I love their approach. You just add xml configuration data at launch time (looks like a build file), and the wowza instance goes through it the first time it’s instantiated. Installing a custom app just requires zipping up the jar & config files using the wowza folder structure, then defining the http location of that zip file in the startup script. No need to create custom image for your application. Upgrading to the latest version of the wowza AMI is a painless process.
Without going into as much depth & flexibility as wowza, we decided to follow a similar approach. Zip up our custom code & apps in a known folder structure, post it to our cloud storage app, and load it at startup using a startup batch file or command-line app. This way, updating a cloud with a bunch of nodes or scaling horizontally only requires uploading the archive, a few clicks in the GoGrid UI, and that’s it. Using the GoGrid API, we should even be able to put a web front end on our DHCP server (the one that’s always on) to upload the archive, put it on the cloud storage space, and remotely reboot all related instances.
We finally reached our holy grail: one-click, 10 minute deployment.
