The release of AWS share EBS snapshots
How to Boot EC2 Instances from EBS
Cool Things You can do with Shared EBS
Hidden Dangers in Creating Public EBS Snapshots on EC2
Making an AMI use EBS to load the system files
Using the references above I tried to test Bootable EBS volumes, I’ve made some experiments (duh) to test bootable EBS volumes base on Alestic AMIs:
Ubuntu Hardy(ami-e546a78c)
Debian Lenny AMI(ami-ff46a796)
Note that both of these AMI’s are using the same AKI (aki-a71cf9ce) and ARI (ari-a51cf9cc). Therefore the two boot EBS volumes created has this properties. To create the Lenny and Hardy bootable EBS volumes, I simply follow Shlomo’s How to Boot EC2 Instances from EBS tutorial.
The first boot AMI is based on Alestic Ubuntu Hardy(ami-e546a78c) where the device used was set to /dev/sdj instead of /dev/sdp in Shlomo’s tutorial.
It was able to boot and I can successfully connect using SSH to these two instances without problems.
Here are the output of using two different EBS volumes:
Debian Lenny bootable EBS Output using boot AMI based on Alestic Hardy(ami-e546a78c)
EXT3 FS on sda1, internal journal Making pivot_root device node /new-root-dev Attempting to mount pivot_root volume on /new-root-dev 1 scsi_mod: no version for "struct_module" found: kernel tainted. SCSI subsystem initialized register_blkdev: cannot get major 8 for sd mount: /new-root-dev is not a valid block device 2 mount: /new-root-dev is not a valid block device 3 mount: /new-root-dev is not a valid block device 4 mount: /new-root-dev is not a valid block device 5 mount: /new-root-dev is not a valid block device 6 mount: /new-root-dev is not a valid block device 7 mount: /new-root-dev is not a valid block device 8 mount: /new-root-dev is not a valid block device 9 mount: /new-root-dev is not a valid block device 10 mount: /new-root-dev is not a valid block device sdj: unknown partition table 11 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Mounted pivot_root volume. Mount failed for selinuxfs on /selinux: No such file or directory INIT: version 2.86 booting /proc is empty (not mounted ?) hostname: the specified hostname is invalid Starting the hotplug events dispatcher: udevd. Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...input: PC Speaker as /class/input/input1 done.
Ubuntu Hardy bootable EBS Output using boot AMI based on Alestic Hardy(ami-e546a78c)
EXT3 FS on sda1, internal journal Making pivot_root device node /new-root-dev Attempting to mount pivot_root volume on /new-root-dev 1 scsi_mod: no version for "struct_module" found: kernel tainted. SCSI subsystem initialized register_blkdev: cannot get major 8 for sd mount: /new-root-dev is not a valid block device 2 mount: /new-root-dev is not a valid block device 3 mount: /new-root-dev is not a valid block device 4 mount: /new-root-dev is not a valid block device 5 mount: /new-root-dev is not a valid block device 6 mount: /new-root-dev is not a valid block device 7 mount: /new-root-dev is not a valid block device 8 mount: /new-root-dev is not a valid block device 9 mount: /new-root-dev is not a valid block device sdj: unknown partition table 10 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Mounted pivot_root volume. /proc is empty (not mounted ?) hostname: the specified hostname is invalid * Setting preliminary keymap... [80G [74G[ [31mfail [39;49m] * Starting basic networking... [80G [74G[ OK ] * Starting kernel event manager... [80G
Next test is using the public boot AMI nimlabs/pivot-sdj-20080824.manifest.xml
(note this AMI is using also AKI aki-a71cf9ce and ARI ari-a51cf9cc ):
These bootable EBS boots successfully and SSH are successful to the instances also.
Debian Lennyy bootable EBS Output using nimlabs/pivot-sdj-20080824.manifest.xml:
EXT3-fs warning: checktime reached, running e2fsck is recommended EXT3 FS on sda1, internal journal mknod Attempting to mount /new-root-dev 1 scsi_mod: no version for "struct_module" found: kernel tainted. SCSI subsystem initialized register_blkdev: cannot get major 8 for sd mount: /new-root-dev is not a valid block device 2 mount: /new-root-dev is not a valid block device 3 mount: /new-root-dev is not a valid block device 4 mount: /new-root-dev is not a valid block device 5 mount: /new-root-dev is not a valid block device 6 mount: /new-root-dev is not a valid block device 7 mount: /new-root-dev is not a valid block device 8 mount: /new-root-dev is not a valid block device 9 mount: /new-root-dev is not a valid block device 10 mount: /new-root-dev is not a valid block device 11 mount: /new-root-dev is not a valid block device 12 mount: /new-root-dev is not a valid block device sdj: unknown partition table 13 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Mounted. Mount failed for selinuxfs on /selinux: No such file or directory INIT: version 2.86 booting /proc is empty (not mounted ?) hostname: the specified hostname is invalid Starting the hotplug events dispatcher: udevd. Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...input: PC Speaker as /class/input/input1 done. Activating swap...Adding 917496k swap on /dev/sda3. Priority:-1 extents:1 across:917496k done. Checking root file system...fsck 1.41.3 (12-Oct-2008)
Ubuntu Hardy bootable EBS Output using nimlabs/pivot-sdj-20080824.manifest.xml:
EXT3-fs warning: checktime reached, running e2fsck is recommended EXT3 FS on sda1, internal journal mknod Attempting to mount /new-root-dev 1 scsi_mod: no version for "struct_module" found: kernel tainted. SCSI subsystem initialized register_blkdev: cannot get major 8 for sd mount: /new-root-dev is not a valid block device 2 mount: /new-root-dev is not a valid block device 3 mount: /new-root-dev is not a valid block device 4 mount: /new-root-dev is not a valid block device sdj: unknown partition table 5 kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Mounted. /proc is empty (not mounted ?) hostname: the specified hostname is invalid * Setting preliminary keymap... [80G [74G[ [31mfail [39;49m] * Starting basic networking... [80G [74G[ OK ] * Starting kernel event manager... [80G [74G[ OK ] * Loading hardware drivers... [80G scsi_id[394]: scsi_id: unable to access parent device of '/block/sdj' input: PC Speaker as /class/input/input1 [74G[ OK ] * Loading kernel modules... [80G * Loading manual drivers... [80G fuse init (API version 7.8) fuse distribution version: 2.7.3 [74G[ OK ] * Setting kernel variables... [80G error: "kernel.maps_protect" is an unknown key error: "vm.mmap_min_addr" is an unknown key [74G[ [31mfail [39;49m] * Activating swap... [80G [74G[ OK ] * Checking root file system... [80G fsck 1.40.8 (13-Mar-2008) /dev/sdj: clean, 19126/327680 files, 157399/1310720 blocks [74G[ OK ] * Checking file systems... [80G fsck 1.40.8 (13-Mar-2008) [74G[ OK ] * Mounting local filesystems... [80G mount: none already mounted or /sys busy mount: according to mtab, /sys is already mounted on /sys [74G[ [31mfail [39;49m] * Activating swapfile swap... [80G [74G[ OK ] * Checking minimum space in /tmp... [80G [74G[ OK ] * Skipping firewall: ufw (not enabled)... [80G [74G[ OK ] * Configuring network interfaces... [80G [74G[ OK ] /etc/init.d/rc: 317: /etc/rcS.d/S60ec2-ssh-host-key-gen: Permission denied * Starting system log daemon... [80G
Now for a change, I built a new boot AMI based on Canonical Hardy(ami-5d59be34).Note that this has a different AKI and ARI with previous boot AMI’s describe above.
I was able to boot but the console output says it was not able to load kernel modules. SSH to these instances were not available.
Debian Lennyy bootable EBS Output resulting to MISMATCH kernel modules with boot AMI based on Canonical Hardy(ami-5d59be34) :
[ 7.399096] EXT3 FS on sda1, internal journal Making pivot_root device node /new-root-dev Attempting to mount pivot_root volume on /new-root-dev 1 [ 22.425756] SCSI subsystem initialized [ 22.427103] register_blkdev: cannot get major 8 for sd [ 22.427113] Driver 'sd' needs updating - please use bus_type methods mount: /new-root-dev is not a valid block device [ 32.601893] sdj: unknown partition table 2 [ 37.456636] kjournald starting. Commit interval 5 seconds [ 37.456656] EXT3-fs: mounted filesystem with ordered data mode. Mounted pivot_root volume. Mount failed for selinuxfs on /selinux: No such file or directory INIT: version 2.86 booting [ 37.808875] [ 37.808881] *************************************************************** [ 37.808884] *************************************************************** [ 37.808887] ** WARNING: Currently emulating unsupported memory accesses ** [ 37.808889] ** in /lib/tls glibc libraries. The emulation is ** [ 37.808892] ** slow. To ensure full performance you should ** [ 37.808894] ** install a 'xen-friendly' (nosegneg) version of ** [ 37.808896] ** the library, or disable tls support by executing ** [ 37.808899] ** the following as root: ** [ 37.808905] ** mv /lib/tls /lib/tls.disabled ** [ 37.808907] ** Offending process: init (pid=2159) ** [ 37.808910] *************************************************************** [ 37.808913] *************************************************************** [ 37.808917] [ 37.808918] Pausing... 5Pausing... 4Pausing... 3Pausing... 2Pausing... 1Continuing... [ 43.868166] /proc is empty (not mounted ?) hostname: the specified hostname is invalid Starting the hotplug events dispatcher: udevd. Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...done. Activating swap...[ 46.875974] Adding 917496k swap on /dev/sda3. Priority:-1 extents:1 across:917496k done. Checking root file system...fsck 1.41.3 (12-Oct-2008) /dev/sdj: clean, 21027/327680 files, 165956/1310720 blocks done. [ 48.045344] EXT3 FS on sdj, internal journal Cleaning up ifupdown.... Loading kernel modules...FATAL: Could not load /lib/modules/2.6.24-6-xen/modules.dep: No such file or directory Checking file systems...fsck 1.41.3 (12-Oct-2008) done. Setting kernel variables (/etc/sysctl.conf)...done. Mounting local filesystems...[ 48.579219] kjournald starting. Commit interval 5 seconds [ 48.579241] EXT3-fs warning: checktime reached, running e2fsck is recommended [ 48.601282] EXT3 FS on sda2, internal journal [ 48.601290] EXT3-fs: mounted filesystem with ordered data mode. mount: none already mounted or /sys busy mount: according to mtab, sysfs is already mounted on /sys [31mfailed. [39;49m Activating swapfile swap...done. Setting up networking.... Configuring network interfaces...Internet Systems Consortium DHCP Client V3.1.1 Copyright 2004-2008 Internet Systems Consortium. All rights reserved. For info, please visit http://www.isc.org/sw/dhcp/ modprobe: FATAL: Could not load /lib/modules/2.6.24-6-xen/modules.dep: No such file or directory socket: Address family not supported by protocol - make sure CONFIG_PACKET (Packet socket) and CONFIG_FILTER (Socket Filtering) are enabled in your kernel configuration! Failed to bring up eth0. done. /etc/init.d/rc: line 221: /etc/rcS.d/S60ec2-ssh-host-key-gen: Permission denied INIT: Entering runlevel: 4 Starting enhanced syslogd: rsyslogd. Starting OpenBSD Secure Shell server: sshd.
Ubuntu Hardy bootable EBS Output resulting to MISMATCH kernel modules with boot AMI based on Canonical Hardy(ami-5d59be34):
[ 7.321856] EXT3 FS on sda1, internal journal Making pivot_root device node /new-root-dev Attempting to mount pivot_root volume on /new-root-dev 1 [ 22.342885] SCSI subsystem initialized [ 22.344216] register_blkdev: cannot get major 8 for sd [ 22.344227] Driver 'sd' needs updating - please use bus_type methods mount: /new-root-dev is not a valid block device 2 mount: /new-root-dev is not a valid block device 3 mount: /new-root-dev is not a valid block device 4 mount: /new-root-dev is not a valid block device [ 72.231476] sdj: unknown partition table 5 [ 82.386877] kjournald starting. Commit interval 5 seconds [ 82.386894] EXT3-fs: mounted filesystem with ordered data mode. Mounted pivot_root volume. /proc is empty (not mounted ?) hostname: the specified hostname is invalid * Setting preliminary keymap... [80G [74G[ [31mfail [39;49m] * Starting basic networking... [80G [74G[ OK ] * Starting kernel event manager... [80G [74G[ OK ] * Loading hardware drivers... [80G scsi_id[2346]: scsi_id: unable to access parent device of '/block/sdj' [74G[ OK ] * Loading kernel modules... [80G * Loading manual drivers... [80G [74G[ OK ] * Setting kernel variables... [80G [74G[ OK ] * Activating swap... [80G [74G[ OK ] * Checking root file system... [80G fsck 1.40.8 (13-Mar-2008) /dev/sdj: clean, 19125/327680 files, 157410/1310720 blocks [74G[ OK ] * Checking file systems... [80G fsck 1.40.8 (13-Mar-2008) [74G[ OK ] * Mounting local filesystems... [80G modprobe: FATAL: Could not load /lib/modules/2.6.24-6-xen/modules.dep: No such file or directory mount: none already mounted or /sys busy mount: according to mtab, /sys is already mounted on /sys [74G[ [31mfail [39;49m] * Activating swapfile swap... [80G [74G[ OK ] * Checking minimum space in /tmp... [80G [74G[ OK ] * Skipping firewall: ufw (not enabled)... [80G [74G[ OK ] * Configuring network interfaces... [80G modprobe: FATAL: Could not load /lib/modules/2.6.24-6-xen/modules.dep: No such file or directory [74G[ OK ] /etc/init.d/rc: 317: /etc/rcS.d/S60ec2-ssh-host-key-gen: Permission denied * Starting system log daemon... [80G [74G[ OK ] * Starting kernel log daemon... [80G [74G[ OK ] * Starting OpenBSD Secure Shell server sshd [80G [74G[ OK ]
Now, testing adding modules to match kernel of boot AMI Canonical Hardy(/lib/modules/2.6.24-6-xen)
Trying to copy modules ONLY.. to Alestic Hardy and Debian bootable EBS volumes.
#Copy modules from Canonical Hardy..to test.:) # Attache EBS volume to /dev/sdg sudo mount /dev/sdg /ebs # copy kernel modules.. sudo rsync -a /lib/modules/2.6.24-6-xen /ebs/lib/modules/ # unmount /ebs sudo umount /ebs
Booting again, the Debian Lennyy bootable EBS Output:
[ 67.400454] EXT3-fs: mounted filesystem with ordered data mode. Mounted pivot_root volume. INIT: version 2.86 booting [ 67.661318] [ 67.661324] *************************************************************** [ 67.661327] *************************************************************** [ 67.661330] ** WARNING: Currently emulating unsupported memory accesses ** [ 67.661332] ** in /lib/tls glibc libraries. The emulation is ** [ 67.661335] ** slow. To ensure full performance you should ** [ 67.661337] ** install a 'xen-friendly' (nosegneg) version of ** [ 67.661343] ** the library, or disable tls support by executing ** [ 67.661345] ** the following as root: ** [ 67.661348] ** mv /lib/tls /lib/tls.disabled ** [ 67.661351] ** Offending process: init (pid=2167) ** [ 67.661355] *************************************************************** [ 67.661358] *************************************************************** [ 67.661361] [ 67.661362] Pausing... 5Pausing... 4Pausing... 3Pausing... 2Pausing... 1Continuing... [ 73.719951] /proc is empty (not mounted ?) hostname: the specified hostname is invalid Starting the hotplug events dispatcher: udevd. Synthesizing the initial hotplug events...done. Waiting for /dev to be fully populated...done. Activating swap...[ 76.115492] Adding 917496k swap on /dev/sda3. Priority:-1 extents:1 across:917496k done. Checking root file system...fsck 1.41.3 (12-Oct-2008) /dev/sdj: clean, 23335/327680 files, 180869/1310720 blocks done. [ 76.669409] EXT3 FS on sdj, internal journal Cleaning up ifupdown.... Loading kernel modules...[ 76.964791] SGI XFS with ACLs, security attributes, realtime, large block numbers, no debug enabled [ 76.965291] SGI XFS Quota Management subsystem done. Checking file systems...fsck 1.41.3 (12-Oct-2008) done. Setting kernel variables (/etc/sysctl.conf)...done. Mounting local filesystems...[ 77.219069] kjournald starting. Commit interval 5 seconds [ 77.219083] EXT3-fs warning: checktime reached, running e2fsck is recommended [ 77.228390] EXT3 FS on sda2, internal journal [ 77.228397] EXT3-fs: mounted filesystem with ordered data mode. mount: none already mounted or /sys busy mount: according to mtab, sysfs is already mounted on /sys [31mfailed. [39;49m Activating swapfile swap...done. Setting up networking.... Configuring network interfaces...Internet Systems Consortium DHCP Client V3.1.1 Copyright 2004-2008 Internet Systems Consortium. All rights reserved. For info, please visit http://www.isc.org/sw/dhcp/ [ 78.802833] NET: Registered protocol family 17 Listening on LPF/eth0/12:31:38:00:45:e6 Sending on LPF/eth0/12:31:38:00:45:e6 Sending on Socket/fallback DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8 DHCPOFFER from 169.254.1.0 DHCPREQUEST on eth0 to 255.255.255.255 port 67 DHCPACK from 169.254.1.0 bound to 10.252.74.20 -- renewal in 36549 seconds. done. /etc/init.d/rc: line 221: /etc/rcS.d/S60ec2-ssh-host-key-gen: Permission denied INIT: Entering runlevel: 4 Starting enhanced syslogd: rsyslogd. Starting OpenBSD Secure Shell server: sshd[ 82.888492] NET: Registered protocol family 10 [ 82.888887] lo: Disabled Privacy Extensions . Starting periodic command scheduler: crond.
Booting again using the updated EBS volume, Ubuntu Hardy bootable EBS Output:
[ 7.474054] EXT3 FS on sda1, internal journal Making pivot_root device node /new-root-dev Attempting to mount pivot_root volume on /new-root-dev 1 [ 22.514913] SCSI subsystem initialized [ 22.516316] register_blkdev: cannot get major 8 for sd [ 22.516327] Driver 'sd' needs updating - please use bus_type methods mount: /new-root-dev is not a valid block device 2 mount: /new-root-dev is not a valid block device 3 mount: /new-root-dev is not a valid block device 4 mount: /new-root-dev is not a valid block device 5 mount: /new-root-dev is not a valid block device 6 mount: /new-root-dev is not a valid block device 7 mount: /new-root-dev is not a valid block device 8 mount: /new-root-dev is not a valid block device 9 mount: /new-root-dev is not a valid block device 10 mount: /new-root-dev is not a valid block device [ 158.981003] sdj: unknown partition table 11 [ 172.565283] kjournald starting. Commit interval 5 seconds [ 172.565299] EXT3-fs: mounted filesystem with ordered data mode. Mounted pivot_root volume. /proc is empty (not mounted ?) hostname: the specified hostname is invalid * Setting preliminary keymap... [80G [74G[ [31mfail [39;49m] * Starting basic networking... [80G [74G[ OK ] * Starting kernel event manager... [80G [74G[ OK ] * Loading hardware drivers... [80G scsi_id[2372]: scsi_id: unable to access parent device of '/block/sdj' [74G[ OK ] * Loading kernel modules... [80G * Loading manual drivers... [80G [74G[ OK ] * Setting kernel variables... [80G [74G[ OK ] * Activating swap... [80G [74G[ OK ] * Checking root file system... [80G fsck 1.40.8 (13-Mar-2008) /dev/sdj: clean, 21435/327680 files, 172306/1310720 blocks [74G[ OK ] * Checking file systems... [80G fsck 1.40.8 (13-Mar-2008) [74G[ OK ] * Mounting local filesystems... [80G mount: none already mounted or /sys busy mount: according to mtab, /sys is already mounted on /sys [74G[ [31mfail [39;49m] * Activating swapfile swap... [80G [74G[ OK ] * Checking minimum space in /tmp... [80G [74G[ OK ] * Skipping firewall: ufw (not enabled)... [80G [74G[ OK ] * Configuring network interfaces... [80G [74G[ OK ] /etc/init.d/rc: 317: /etc/rcS.d/S60ec2-ssh-host-key-gen: Permission denied * Starting system log daemon... [80G [74G[ OK ] * Starting kernel log daemon... [80G [74G[ OK ] * Starting OpenBSD Secure Shell server sshd [80G [74G[ OK ] * Reloading OpenBSD Secure Shell server's configuration sshd * Starting deferred execution scheduler atd [80G [74G[ OK ] * Starting periodic command scheduler crond [80G [74G[ OK ] * Running local boot scripts (/etc/rc.local) [80G [74G[ OK ]
I was able to boot and login using SSH on these updated bootable EBS volumes(containing kernel modules).
These are my observation after the testing for the Alestic (Lenny and Hardy) based bootable EBS:
1. Bootable EBS can easily be booted if it has have the same AKI and ARI with the boot AMI.
2. If different AKI and ARI is used by boot AMI, we can copy the kernel modules to these bootable EBS volumes and boot it again.
(but I’m not sure if this is a good idea because reading from EC2 forums this might pose other unexpected kernel/AMI problems)
3. Different distro like Ubuntu and Debian are using different /sbin/init programs (upstart/sysvinit). And because the boot AMI is Ubuntu distro, Debian Lenny prints a log below:
Mount failed for selinuxfs on /selinux: No such file or directory
This test may help somehow in having tranportable/sharable bootable EBS snapshots soon.
(Btw, thank you to all of those people who provided the references, ideas and links .. cheers!)
Pingback: New Amazon EC2 Feature: Boot from Elastic Block Store