Create EBS Boot AMI

57 Replies

After the release of New Amazon EC2 Feature: Boot from Elastic Block Store , there are folks asking on AWS forum on how to create an EBS Boot AMI and so I’m posting another way to do it. I hope this will be plain and simple to follow. 😉

Note that I’m using Create EBS Boot AMI scripts courtesy of fairchild.

To be able to register EBS Boot AMI, make sure you install the latest EC2 API Tools, you can check that using:

prompt> ec2-version
1.3-45772 2009-10-31

And be sure you set the the EC2 private and cert:

prompt> export EC2_PRIVATE_KEY=/home/tim/ec2-keys/pk.pem
prompt> export EC2_CERT=/home/tim/ec2-keys/cert.pem

I will be using the Canonical Hardy AMI: ami-7e28ca17
The first thing is to get the AKI and ARI of the AMI using ec2-describe-images command or using ElasticFox.

prompt> ec2-describe-images ami-7e28ca17
IMAGE	ami-7e28ca17	ubuntu-images-us/ubuntu-hardy-8.04-i386-server-20091130.manifest.xml	099720109477	available	public		i386	machine	aki-92ba58fb	ari-94ba58fd		instance-store

Here are the AKI: aki-92ba58fb and ARI: ari-94ba58fd information.

Launch the AMI and set the keypair and availability zone:

prompt> ec2-run-instances ami-7e28ca17 -k yourkeypair -z us-east-1a
RESERVATION    r-237fed4a    853279305000    default
INSTANCE  i-d9add0b0  ami-7e28ca17  pending   0  m1.small  2009-11-14T12:38:24+0000  us-east-1a  aki-92ba58fb  ari-94ba58fd  monitoring-disabled

Create the volume and attached it to the instance. Make sure the volume has the same availability zone with the current running instance.

prompt> ec2-create-volume  --size 20 --availability-zone us-east-1a
VOLUME vol-4d826724 20 us-east-1a creating 2009-11-14T12:39:24+0000

prompt> ec2-attach-volume vol-4d826724 -i i-d9add0b0 -d /dev/sdh
ATTACHMENT vol-4d826724 i-6058a509 /dev/sdh attaching 2009-11-14T12:43:24+0000

Make sure the volume has status “attached” before doing the next step.

Now we need to get a copy of instance-to-ebs-ami.sh script and execute it inside the running instance.

prompt> ssh -k yourkeypair ubuntu@public-dns-of-my-instance

prompt> wget https://raw.github.com/gist/1018285/cee984da6acda43c3063cf6d9e8c564d37812428/instance-to-ebs-ami.sh

prompt> chmod +x /tmp/instance-to-ebs-ami.sh
prompt> sudo /tmp/instance-to-ebs-ami.sh

(...output truncated...assuming the script was executed successfuly)
prompt> exit

Now we are back to our machine’s xterm, let’s detach the volume, make snapshot and register as new EBS Boot AMI:

prompt> ec2-detach-volume vol-4d826724
ATTACHMENT vol-4d826724 i-6058a509 /dev/sdh detaching 2009-11-15T00:55:24+0000

prompt> ec2-create-snapshot vol-4d826724 --description "EBS Ubuntu Hardy"
SNAPSHOT  snap-c070c5a9  vol-4d826724  pending 2009-11-15T1:31:29+0000  111111111111  1  EBS Ubuntu Hardy

prompt> ec2-register --snapshot snap-c070c5a9 --kernel aki-92ba58fb \
--ramdisk ari-94ba58fd --description "EBS Ubuntu Hardy" \ 
--name "ebs-ubuntu-hardy-8.04-i386-server-20091130" \
-–block-device-mapping /dev/sda2=ephemeral0 \
--architecture i386 --root-device-name /dev/sda1

Yes of course you can make use the scripts provided on the github link above. But I’m sure now you get the idea. Now start to convert the remaining non-EBS AMI you may have. 😉

By default EBS Boot AMI does not attached the ephemeral storage. See these links for reference:
Attached Ephemeral Storage on EBS Boot AMI
Ephemeral Storage Capacity

There’s also a question of “Can you resize the EBS boot AMI?”. I believe, yes. Will give it a try and post the results here.

Update:
I have posted before how to increase or enlarge the resize EBS volumes.
And about my question of resizing the EBS AMI, I found Eric Hammond’s post which is very helpful:Increasing Root Disk Size of an “EBS Boot” AMI on EC2

Update 01-28-2010:
If your looking for Public EBS Boot AMIs for Ubuntu on Amazon EC2, Eric Hammond have provided them in his site. Thanks Eric.

Update 06-10-2011:
Add label to mkfs.ext3
Add -–block-device-mapping /dev/sda2=ephemeral0 for i386 or –block-device-mapping /dev/sdb=ephemeral0 for x86_64 arch.

References:
http://coderslike.us/2009/12/07/amazon-ec2-boot-from-ebs-and-ami-conversion/
AWS EC2 Thread: Booting from EBS Image Creation

57 thoughts on “Create EBS Boot AMI”

  5. mklatsky

    Thanks for a clear and concise writeup. My question is – what does one do when there is no identified ARI associated with a particular AMI that we are using?

  6. Radek

    Rodney, is this correct? It seems you are downloading bundle-to-ebs.sh but saving as /tmp/instance-to-ebs-ami.sh ? Should you instead download instance-to-ebs-ami.sh?
    Cheers

  8. Radek

    Hey Rodney, thanks a lot! I managed to create the EBS bootable image thanks to your post! I based it on offical canonical ubuntu-hardy-9.10-i386. One thing which is kind of weird is that whenever I terminate the instance the EBS volume attached to that instance is deleted as well… Do you have any idea why this may be happening?

  10. rodney Post author

    Radek,

    On EBS AMI, if you terminate the instance it will delete the volume. If you want to make a point in time copy of your EBS AMI, you can stop it using AWS Management console and use Create Image(EBS AMI). Or just create a new snapshot of the volume then you can register it using the command line tools.

    Rodney

  11. rodney Post author

    As posted here by Ton:
    http://groups.google.com/group/ec2ubuntu/browse_thread/thread/11b9c0ff3e477a53

    Hi Rodney,

    After a small change Michael’s scripts worked fine for me in eu-west
    zone. The ec2- commands in the script did not have the –region
    directive set. In that case they default to ‘us-east-1’, so the script
    works out-of-the-box for all people in that region, but in other
    regions like ‘eu-west-1’ and ‘us-west-1’ the –region directive must
    be explicitly set to that region.

    Best regards, Ton

  12. rodney Post author

    Radek,

    Please check the deleteOnTerminate attribute when you run the instance. This will not delete the volumes as describe on the BootFromEBSGuide manual.

  13. Youri

    Michael,

    I’ve built multiple EBS based AMI’s all working great!
    But i think I’ve found some minor security issues.
    The script doesn’t exclude the authorization_keys found in .ssh which get created on instance initialization.
    Also, the file ec2-init-user-data gets chmodded.
    But the same file could be named ec2-init like i found on other distributions, so you should check it manually after mounting the volume.

    Hope this helps a bit 🙂

  14. Ron

    When I run the following command (filling in my info where appropriate) I just get the help menu for the command. Does anyone see anything wrong? I’m not the most advanced command line person but everything else seemed to go fine.

    ec2-register –snapshot snap-c070c5a9 –kernel aki-92ba58fb –ramdisk ari-94ba58fd –description=”EBS Ubuntu Hardy” –name=”–name=”ebs-ubuntu-hardy-8.04-i386-server-20091130″” –architecture i386 –root-device-name /dev/sda1

  16. Mark

    Ron – also it could be that you’re running the old version of the ec2 tools without support for boot from EBS.

  17. Rares

    I’ve tried this over and over again, I’ve got my volume and snapshot dialed, i can’t register the ami from it, whenever I run

    ec2-register –snapshot snap-98bb4cf1 –kernel aki-7e0d250a –ramdisk ari-7d0d2509 –description=”my image” –name=”gentoo-stage3″ –architecture i386 –root-device-name /dev/sda1 –region eu-west-1

    I get ec2reg/ec2-register usage information, no debugging info, no sign of success or reason of failure… it simply won’t work…

    I’ve tried everything, including:

    ec2-cmd RegisterImage -n my_image -s snap-98bb4cf1

  18. Ron

    Thanks for your response. So I took out the double name and ran this. I’ve looked at a lot of examples and tried some of them but…same result. This makes sense anyway. All of the other commands went well so I think my environment is good. Do I need an s3 bucket for this? Do I need to include a manifest? I’ve included the response output. Thanks for your help.

    ec2-register –snapshot snap-6db3f004 –kernel aki-5f15f636 –ramdisk ari-0915f660 –description=”ebs based ami” –name=”abc-ami-ebs1″ –architecture i386 –root-device-name /dev/sda1 –region us-east-1d
    SYNOPSIS
    ec2reg (ec2-register)
    ec2reg [GENERAL OPTIONS] MANIFEST
    GENERAL NOTES
    Any command option/parameter may be passed a value of ‘-‘ to indicate
    that values for that option should be read from stdin.
    DESCRIPTION
    Register an AMI manifest for use with EC2.
    The MANIFEST parameter is the manifest (in S3) to register.
    e.g. mybucketname/image.manifest

    GENERAL OPTIONS

    -K, –private-key KEY
    Specify KEY as the private key to use. Defaults to the value of the
    EC2_PRIVATE_KEY environment variable (if set). Overrides the default.

    -C, –cert CERT
    Specify CERT as the X509 certificate to use. Defaults to the value
    of the EC2_CERT environment variable (if set). Overrides the default.

    -U, –url URL
    Specify URL as the web service URL to use. Defaults to the value of
    ‘https://ec2.amazonaws.com’ or to that of the EC2_URL environment
    variable (if set). Overrides the default.

    –region REGION
    Specify REGION as the web service region to use.
    This option will override the URL specified by the “-U URL” option and EC2_URL environment variable.

    -v, –verbose
    Verbose output.

    -?, –help
    Display this help.

    -H, –headers
    Display column headers.

    –debug
    Display additional debugging information.

    –show-empty-fields
    Indicate empty fields.

    –connection-timeout TIMEOUT
    Specify a connection timeout TIMEOUT (in seconds).

    –request-timeout TIMEOUT
    Specify a request timeout TIMEOUT (in seconds).

  20. Ron

    Hey you may be right.
    When I run: ec2-version
    I get: 1.3-36506 2009-04-04

    I downloaded the latest (46266) and put it in /usr/local/ec2-api-tools-1.3-46266. I made that its home.If I run:
    $EC2_HOME/bin/ec2-version

    I get: 1.3-46266 2009-11-30

    As I am just moving to Linux and trying to remember Unix commands from yesteryear I may be doing this wrong. Did I not install the latest ec2 api tools properly? Thanks…

  22. brett cave

    Great article, helped plenty in moving images to EBS.

    Some gotcha’s I faced: firstly the links to the script seemed to miss the instance-to-ebs (only grabbed the bundle-to-ebs). The rsync should also have –exclude=”exclude” with the inverts, otherwise get a nice mess in the target. also exclude “/sys/*”

    Before the ec2-register phase, you need to wait for the snapshot to complete. You can hang around and watch, or stick it in:
    while ! ec2-describe-snapshots $SNAP | grep -q completed; do
    echo “Snapshot is `ec2-describe-snapshots $SNAP | awk ‘{print $6}’` complete”
    done

  24. Jonah

    Thanks for the quick rundown! Worked like a charm. Just wanted to clarify a point made above, which is that there are two typos in this line:

    ec2-register –snapshot snap-c070c5a9 –kernel aki-92ba58fb –ramdisk ari-94ba58fd –description=”EBS Ubuntu Hardy” –name=”–name=”ebs-ubuntu-hardy-8.04-i386-server-20091130″” –architecture i386 –root-device-name /dev/sda1

    The first is the ‘–name=”–name’ thing, and the second is that for –name and –description, you shouldn’t use equal signs, I don’t think. Anyway, thank you!

  26. Pingback: How to create an EBS image from an existing AMI « technikh

  29. Pingback: Migrating a Linux S3 Based AMI to an EBS Based AMI « すでにそこにある雲

  30. Munts

    well i tried cloudyscripts, plus several other sites then came back to this one. Eventually got my head around linux and managed to get it working. thanks!

    here are my notes:

    # Setup new repo, update and install latest e2c tools:

    sudo apt-get update && sudo apt-get install python-software-properties

    sudo add-apt-repository ppa:ubuntu-on-ec2/ec2-tools

    sudo apt-get update && sudo apt-get install ec2-api-tools ec2-ami-tools

    # Next up, we’ll set some paths in bash profile so the OS knows where the EC2 tools are located.

    sudo vim ~/.bash_profile

    # Add these lines, replacing “YOURKEYNAME” with the actual file name of your private and public keys, then save.

    export EC2_HOME=~/.ec2
    export PATH=$PATH:$EC2_HOME/bin
    export EC2_PRIVATE_KEY=/home/ubuntu/pk-“YOURKEYNAME”.pem
    export EC2_CERT=/home/ubuntu/cert-“YOURKEYNAME”.pem
    export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home/

    # To get the changes noticed by the OS immediately, run source.

    source ~/.bash_profile

    # To test the tools, run the following from a new terminal:

    ec2-describe-images -o self -o amazon

    # You should receive a list of all the publicly available EC2 images published by Amazon.
    # Now to register as new EBS Boot AMI:

    ec2-register –snapshot “YOURSNAPSHOT” –description “EBS TEST Ubuntu” –name “ebs-ubuntu-i386-server” –architecture i386 –root-device-name /dev/sda1

    # It should return your new AMI number, go into AWS web console to take a look and launch as new instance!

  31. Munts

    Oh and it took me ages to work out how to get the right certs (download the 2 X.509 certs cert-xxx…pem and pk-xxx…pem) from amazon accounts page and then even longer to figure out how to upload them to the server.

    Eventually found some software called winscp which runs over ssh and worked like a treat for uploading the certs to the right place ready for the ec2-register command.

    Sure would be a bit easier if the amazon folk integrated register function into the AWS mgmt web interface sigh.

    For all the linux old hands out there this stuff must be a piece of cake – for us noobs its like pulling teeth.

    😉

  33. cloudsuser

    Nice tutorial. Its strange for me unable to do ssh. The instance created using EBS image is up, but unable to login.

  35. narender

    hey,
    Do you have steps for converting existing instance based AMI to EBS based for Windows platform.

    TIA,

  36. FreeMind

    Hi,

    This was really useful.

    I am trying to use a bootable windows ebs volume but I am not able to figure out how to point the boot ami to the windows ebs volume.

    Any help is appreciated.

    Thanks

  37. rodney Post author

    FreeMind and narender:

    This script dont work with Windows AMI.
    You need to use the existing EBS boot Windows AMI and bundle it from the AWS management console or Elasticfox.

  38. Artur

    Nice one. You should add a note that after cooking up AMI you can remove volume but you need to keep a snapshot you used.

  39. Ratheeshekm

    Hai I have created one EBS boot instance from existing instance with reference to your article.But I cannot do ssh to my newly created instance.I don’t no what is the problem.Can you tell me what will be the problem.

  41. andy

    one needs to add the option –block-device-mapping /dev/sda2=ephemeral0 to the ec2-register command as of sept 2010 otherwise it hangs on boot and fails to setup the ssh keys

  42. Michael

    Hi Andy (and all others of course)!

    You said: “…otherwise it […] fails to setup the ssh keys”. What is the exact result of this issue?

    My freshly created EBS-backed AMI (Debian 5.0 lenny) starts and keeps running but the SSH server is not started. The system log (accessed via AWS Console) shows the following problem: “Starting OpenBSD Secure Shell server: sshd failed – PRNG is not seeded”.

    I know, this also happens when there’s a problem with /dev/urandom or missing host keys but all looks fine. Any thoughts?

  43. Michael

    (me again)…

    I figured out that this issue is associated with the kernel/ramdisk you chose. I could not find a kernel/ramdisk that works with my Debian image but *without* parameters –kernel and –ramdisk in command “ec2-register”, /dev/urandom can be addressed and the SSH server starts.

    Downside: the instance uses Amazon’s default kernel version 2.6.16-xenU – pretty old.

    I would be more than happy if someone has a suggestion how to address the kernel/ramdisk+SSH issue.

  45. Michael

    rodney,
    Yes, great alternative… but in this case our client had very specific requirements. Their policies dictated to set up an instance from scratch (install a fresh Debian system via bootstrap), not using an existing AMI for security reasons (“don’t trust anybody”) 🙂 and the instance needed to be Debian lenny 🙁

    So we bundled our own image, uploaded it to Amazon and run it as an AMI-instance. Afterwards my plan was to convert it to an EBS instance but I struggle at the kernel/ramdisk. Apart of this, the AMI-to-EBS conversation works pretty good (thanks to the infos and links on this page!).

  46. Bob

    Great tutorial. After going through all the steps, I get an error when I launch a new instance using the ami produced and try to ssh into it:

    ssh: connect to host ec2-XXXXXXX.us-west-1.compute.amazonaws.com port 22: Connection refused

    All status messages appear fine for the instance, but no-go on shh. Any ideas? Either way, thanks a lot!

  49. rodney Post author

    Hmm.. A possible for Lucid AMI.

    Assuming you already have the rsync’ed EBS volume, just mount the volume to any instance, edit the /etc/fstab and snapshot.

    Then register it as a new AMI. 🙂

  50. Michael

    Hey Bob!
    As rodney suggested: check security groups and the output of the boot process (system log at AWS console). Maybe you have the same issue I had (unable to start SSH server), see my post above (Michael on 20 September 2010, 9:17 pm).

Leave a Reply

Your email address will not be published. Required fields are marked *