Launching Asia Pacific EC2 Instances using Mr.Awsome

Since the AWS announcment of Asia Pacific Region, I’m testing launching some instances and using Fabric for deployment. I found another tool by Florian Schulze which integrates Fabric in his tool to launch EC2 instance: mr.awsome.

I just added some lines to be able to get to use a different user when executing the Fabric commands, my fork is found here.

My aim here is to be able to launch and test EC2 instances at Asia-Pac region plus experimenting with mr.awsome tool.
Download the latest EC2-API Tools and setup EC2 environment.

$ wget http://ec2-downloads.s3.amazonaws.com/ec2-api-tools.zip
$ unzip ec2-api-tools.zip

On the terminal execute the following. To make it permanent add these lines on the ~/.bashrc

export EC2_HOME=$HOME/<where your ec2-api-tools>
export EC2_PRIVATE_KEY=$HOME/<where your private key is>/pk-XXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export EC2_CERT=$HOME/<where your certificate is>/cert-XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.pem
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/
export AWS_ACCESS_KEY_ID=<your-accesskey>
export AWS_SECRET_ACCESS_KEY=<your-secretkey>
export PATH=$PATH:$EC2_HOME/bin


Create Keypair on Asia-Pacific Region.

$ ec2-add-keypair apac_keypair --region ap-southeast-1  > id-apac-keypair
$ chmod 600 id-apac-keypair

I’ve added the ssh keypair using the command ssh-add below:

$ ssh-add id-apac-keypair

Test if we can describe an instance located at ap-southeast-1

$  ec2-describe-images ami-a5f38cf7 --region ap-southeast-1

Now we can get into running the mr.awsome:
1. Clone the repo.

$ git clone git://github.com/cocoy/mr.awsome.git
$ cd mr.awsome
$ sudo python setup.py develop

2. To create a project:

$ mkdir -p ~/myproj/etc
$ cd ~/myproj
$ vi etc/aws.conf , (or use your favorite editor, and copy the lines below)
[securitygroup:default]
description = Our Demo-Server
connections =
tcp 22 22 0.0.0.0/0
tcp 80 80 0.0.0.0/0

[instance:demo-server]
keypair = gsg-keypair
securitygroups = default
region = us-east-1
placement = us-east-1a
# we use images from `http://alestic.com/`
# Ubuntu 9.10 Karmic server 32-bit Europe
image = ami-bb709dd2
#startup_script = startup-demo-server
fabfile = fabfile.py
server_user = ubuntu

[instance:apac-lucid-server]
keypair = apac_keypair
securitygroups = default
region = ap-southeast-1
placement = ap-southeast-1a
# we use images from `http://alestic.com/`
# Ubuntu Lucid server 32-bit Asia-Pac
image = ami-a5f38cf7
fabfile = fabfile.py
server_user = ubuntu

$ vi etc/fabfile.py ( or use your favorite editor and copy these lines below)

from fabric.api import env, run
env.reject_unknown_hosts = True
env.disable_known_hosts = True

def get_syslog():
    run("tail /var/log/syslog")

Now start EC2 instance, (just make sure we’re still inside our myproj directory) :

$ cd ~/myproj
$ aws start apac-lucid-server
INFO: Instance 'apac-lucid-server' unavailable
INFO: Creating instance 'apac-lucid-server'
INFO: Instance created, waiting until it's available
......
INFO: Instance 'apac-lucid-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-175-41-168-128.ap-southeast-1.compute.amazonaws.com
INFO: Instances public DNS name ec2-175-41-168-128.ap-southeast-1.compute.amazonaws.com
WARNING: Console output not (yet) available. SSH fingerprint verification not possible.

Check status until SSH verification is possible, I have to issue the command several times here.

$ aws status apac-lucid-server
INFO: Instance 'apac-lucid-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-175-41-168-128.ap-southeast-1.compute.amazonaws.com
INFO: Instances public DNS name ec2-175-41-168-128.ap-southeast-1.compute.amazonaws.com
INFO: Console output available. SSH fingerprint verification possible.

Ok, testing the Fabric defined command:

$ aws do apac-lucid-server -l
Available commands:

get_syslog

$ aws do apac-lucid-server get_syslog

INFO: Instance 'apac-lucid-server' available
run: tail /var/log/syslog
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 59 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 60 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 46 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 49 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 58 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 62 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 53 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 57 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 2 activation: Invalid argument
out: May  4 04:48:17 ip-10-128-57-240 console-kit-daemon[520]: WARNING: Error waiting for native console 1 activation: Invalid argument

Done.
Disconnecting from apac-lucid-server... done.

The command get_syslog was able to execute tail /var/log/message file inside the EC2 instance. So the next step is to copy the same Fabric recipe for deployment for these set of servers and get ready to roll.

We make sure to terminate this when we are finish of the instance, since this is testing only.

$ aws terminate apac-lucid-server
INFO: Instance 'apac-lucid-server' available
INFO: Instance terminated

Update: 05/06/2010
After more testing with this tool, it seems there are problems with it. 🙂
After adding another instace definition with the same security group, the new lucid-server is being started, it consider this already running where in fact it’s the previous demo-server was launch.

It might be on the zone, but after changing the zone (placement) of lucid-server, it was not able to launch still.

Update: 05/18/2010
Well this is not a bug so far, it is because I was using the same security group for the server instance. Thanks for Florian and Nate for pointing this out.

I’ts undocumented on the mr.awsome README that the tagging of servers is through security groups. We should have each unique group to identify each of the server instance. 🙂 Note that if there are existing one or more instance for a specific group, mr.awsome thinks that you have already a running server. Also undocumented is to use Python2.6. I’ve used Python2.5.2 and the startup_script options don’t work.

My new server config are shown below using webserver and dbserver as groups:

 
[securitygroup:dbserver]
description = Our Demo-Server
connections =
    tcp 22 22 0.0.0.0/0
    tcp 80 80 0.0.0.0/0

[securitygroup:webserver]
description = Web-Server
connections =
    tcp 22 22 0.0.0.0/0
    tcp 80 80 0.0.0.0/0

[instance:demo-server]
keypair = gsg-keypair
securitygroups = dbserver 
region = us-east-1
placement = us-east-1a
# we use images from `http://alestic.com/`
# Ubuntu 9.10 Karmic server 32-bit Europe
image = ami-bb709dd2
#startup_script = startup.sh
fabfile = fabfile.py
server_user = ubuntu


[instance:demo-server2]
keypair = gsg-keypair
securitygroups = webserver 
region = us-east-1
placement = us-east-1a
# we use images from `http://alestic.com/`
# Ubuntu 9.10 Karmic server 32-bit Europe
image = ami-bb709dd2
#startup_script = startup.sh
fabfile = fabfile.py
server_user = ubuntu

This is the session of starting each server.

(pysandbox)tim@tim-desktop:~$ aws start demo-server
INFO: Instance 'demo-server' unavailable
INFO: Creating instance 'demo-server'
INFO: Instance created, waiting until it's available
...........
INFO: Instance 'demo-server' available
INFO: Instance running.
INFO: Instances DNS name ec2-75-101-207-244.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-75-101-207-244.compute-1.amazonaws.com
WARNING: Console output not (yet) available. SSH fingerprint verification not possible.

(pysandbox)tim@tim-desktop:~$ aws start demo-server2
INFO: Instance 'demo-server2' unavailable
INFO: Creating instance 'demo-server2'
INFO: Instance created, waiting until it's available
.......
INFO: Instance 'demo-server2' available
INFO: Instance running.
INFO: Instances DNS name ec2-184-73-40-253.compute-1.amazonaws.com
INFO: Instances public DNS name ec2-184-73-40-253.compute-1.amazonaws.com
WARNING: Console output not (yet) available. SSH fingerprint verification not possible.

(pysandbox)tim@tim-desktop:~$ list_instances 
ID                   Groups                         Hostname
i-c3486ca8           dbserver                       ec2-75-101-207-244.compute-1.amazonaws.com
i-f14a6e9a           webserver                      ec2-184-73-40-253.compute-1.amazonaws.com

I am looking for similar tool to be able to provision and at the same time configure servers on the fly.
But mainly, I would be able group the instances such as I can launch 10-100 instances(w/ different EBS,EIP,etc) in one go.

You can’t do it using Elasticfox because to launch a AMI, you have to launch it one at a time, then attach the EBS volumes individually to your running instance, and then assign Elastic IP individually also.

Thoughts?

9 thoughts on “Launching Asia Pacific EC2 Instances using Mr.Awsome

  1. Nate Aune

    Hi Rodney,

    Thanks for your post about mr.awsome. I’m also using it for a project with Rice University, and they’re launching Rhaptos instances in the asia-pacific region.

    mr.awsome utilizes the security group as a unique identifier for the EC2 instances. Since you used the same security group “default” for each of your instances, mr.awsome thought they were the same one.

    The best practice way is to give each new EC2 instance you launch its own security group. Then you’ll be able to start/stop them independently.

    I also extended mr.awsome to be able to attach EBS volumes created from snapshots when you launch the EC2 instance. http://github.com/natea/mr.awsome/commit/6266363597d2a8f5b0929c6e41446f7897d5267c

    Nate

  2. rodney Post author

    Hi Nate,

    Thanks, Florian pointed about the security group also. And that should be included I guess on the updated README. 🙂

  3. sebastien requiem

    Hi rodney,

    I discovered mr.awsome through your blog and I also updated the snapshot with the ability to add the instances in a load balancer (this should be useful also when having a template of server).

    I guess, we are getting there 🙂

  4. fenris

    hi,
    can u guys help me .. im getting this:

    $ /opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/aws start demo-server
    Traceback (most recent call last):
    File “/opt/local/Library/Frameworks/Python.framework/Versions/2.6/bin/aws”, line 9, in
    load_entry_point(‘mr.awsome==0.8’, ‘console_scripts’, ‘aws’)()
    File “/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pkg_resources.py”, line 305, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
    File “/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pkg_resources.py”, line 2244, in load_entry_point
    return ep.load()
    File “/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/pkg_resources.py”, line 1954, in load
    entry = __import__(self.module_name, globals(),globals(), [‘__name__’])
    ImportError: No module named awsome

  5. fenris

    sorry to bother u guys again .. here is the latest error i got :

    $ aws start demo-server
    /Library/Python/2.6/site-packages/pycrypto-2.0.1-py2.6-macosx-10.6-universal.egg/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
    import warnings
    /Library/Python/2.6/site-packages/pycrypto-2.0.1-py2.6-macosx-10.6-universal.egg/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
    import warnings
    Traceback (most recent call last):
    File “/usr/local/bin/aws”, line 8, in
    load_entry_point(‘mr.awsome==0.8’, ‘console_scripts’, ‘aws’)()
    File “/Library/Python/2.6/site-packages/mr.awsome-0.8-py2.6.egg/mr/awsome/__init__.py”, line 687, in aws
    return aws(argv)
    File “/Library/Python/2.6/site-packages/mr.awsome-0.8-py2.6.egg/mr/awsome/__init__.py”, line 681, in __call__
    args.func(sub_argv, args.func.__doc__)
    File “/Library/Python/2.6/site-packages/mr.awsome-0.8-py2.6.egg/mr/awsome/__init__.py”, line 474, in cmd_start
    choices=list(self.ec2.instances))
    File “/Library/Python/2.6/site-packages/mr.awsome-0.8-py2.6.egg/mr/awsome/__init__.py”, line 363, in ec2
    self._ec2 = EC2(self.configfile)
    File “/Library/Python/2.6/site-packages/mr.awsome-0.8-py2.6.egg/mr/awsome/__init__.py”, line 327, in __init__
    self.config = Config(configpath)
    File “/Library/Python/2.6/site-packages/mr.awsome-0.8-py2.6.egg/mr/awsome/config.py”, line 78, in __init__
    _config.read(config)
    File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ConfigParser.py”, line 286, in read
    self._read(fp, filename)
    File “/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/ConfigParser.py”, line 510, in _read
    raise e
    ConfigParser.ParsingError: File contains parsing errors: /Users/fenris/Documents/BiS/EC2/IntOffice/etc/aws.conf
    [line 4]: ‘tcp 22 22 0.0.0.0/0\n’
    [line 5]: ‘tcp 80 80 0.0.0.0/0\n’
    [line 10]: ‘tcp 22 22 0.0.0.0/0\n’
    [line 11]: ‘tcp 80 80 0.0.0.0/0\n’

    while checking the port .. there is no haslib for python 2.6:

    $ sudo port search hashlib
    py25-hashlib @2.5.4 (python)
    This is a stub. hashlib is now built with python25

  6. fenris

    $ aws start demo-server
    /Library/Python/2.6/site-packages/pycrypto-2.0.1-py2.6-macosx-10.6-universal.egg/Crypto/Hash/SHA.py:6: DeprecationWarning: the sha module is deprecated; use the hashlib module instead
    import warnings
    /Library/Python/2.6/site-packages/pycrypto-2.0.1-py2.6-macosx-10.6-universal.egg/Crypto/Hash/MD5.py:6: DeprecationWarning: the md5 module is deprecated; use hashlib instead
    import warnings
    usage: aws start [-h] [-o [OVERRIDE [OVERRIDE …]]] instance
    aws start: error: argument instance: invalid choice: ‘demo-server’ (choose from ‘demo-dbserver’, ‘demo-webserver’)

    how do i remove @ ignore / hide the import warnings?

Leave a Reply

Your email address will not be published. Required fields are marked *