Today, I was bundling my AMI capable of loading from S3 during bootup and upload files before shutdown. Using the the S3Sync and simple shell scripts I made.
At first I was having trouble getting the init script run during shutdown, until I recheck the chkconfig command manual.A lock file solves the problem, which I absolutely missed at first. š
I have created the scripts below:
#!/bin/bash
#
# Run-level Startup script for the Oracle s3 backup
#
# chkconfig: 345 81 36
# Filename: s3oracle_init
# description: Startup/Shutdown download S3 Oracle datafiles.
if [ -f /var/lock/subsys/oracle ]
then
echo "S3 Datafile Download cancelled. "
echo "Oracle is running. Shutdown Oracle first."
exit 1
fi
case "$1" in
start)
# Oracle is not running.. ok to download files.
/usr/ora_scripts/s3download.sh
touch /var/lock/subsys/s3oracle_init
echo "OK"
;;
stop)
/usr/ora_scripts/s3upload.sh
rm -f /var/lock/subsys/s3oracle_init
echo "OK"
;;
reload|restart)
# $0 stop
# $0 start
;;
*)
echo "Usage: $0 start|stop|restart|reload"
exit 1
esac
exit 0
#!/bin/sh
#
#
# Filename: s3download.sh
# Description: download S3 datafiles.
# S3 access keys are defined at s3config.yml or can be
# done by exporting the variables below:
#
export AWS_ACCESS_KEY_ID=valueGoesHere
export AWS_SECRET_ACCESS_KEY=valuesGoesHere
export SSL_CERT_DIR=valuesGoesHere
S3SYNC_DIR=/root/s3sync
DEST_DIR=/mnt/samdata
# s3 bucket, s3sync format ex. bucketname:key
BUCKET=samdata:
# check dest dir
if [ ! -d $DEST_DIR ]; then
if [ -f $DEST_DIR ]; then
echo "$DEST_DIR is a file. Deleting $DEST_DIR.."
rm -rf $DEST_DIR
echo "OK"
fi
# create a new directory
echo "Creating $DEST_DIR"
mkdir $DEST_DIR
echo "OK"
else
echo "Opps! $DEST_DIR found."
fi
### download s3 datafiles
###--code adapted from:
### http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/
echo -n "Downloading s3 $BUCKET files to $DEST_DIR"
cd $S3SYNC_DIR
ruby s3sync.rb -r --delete $BUCKET $DEST_DIR
echo ""
echo "DONE"
exit 0
#!/bin/sh # # # Filename: s3upload.sh # Description: upload S3 datafiles. # S3 access keys and cert dir are defined inside the file $S3SYNC_DIR/s3config.yml S3SYNC_DIR=/root/s3sync SRC_DIR=/mnt/samdata export AWS_ACCESS_KEY_ID=valueGoesHere export AWS_SECRET_ACCESS_KEY=valuesGoesHere export SSL_CERT_DIR=valuesGoesHere # s3 bucket, s3sync format ex. bucketname:key # tagging upload by date BUCKET=samdata:/ ### download s3 datafiles ### adapated from ### http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/ echo -n "Uploading $SRC_DIR files to $BUCKET" cd $S3SYNC_DIR ruby s3sync.rb -r --delete $SRC_DIR $BUCKET echo "" echo "DONE" exit 0
After setting up the AWS variables , I test first the s3upload.sh and s3download.sh and Iām getting lucky to have it running well.
I start bundling the image, by excluding directories I want to be hidden on my new image. Upload it to S3, and register it as new AMI. After a while I can check the console output of the instance using the ec2-get-console-output command. Shown below is my session showing the my s3oracle_init service working very fine.
~$ ec2-run-instances ami-1020c579 -k gsg-keypair
RESERVATION r-d928c0b0 608316554735 default
INSTANCE i-0ba65762 ami-1020c579 pending gsg-keypair 0
pending gsg-keypair 0
(After seeing the instance is running, it takes some time though depending
on your datafiles to download from S3, I make sure the file is download by checking the console output.)
~$ ec2-terminate-instances i-0ba65762 INSTANCE i-0ba65762 running shutting-down ~$ ec2-get-console-output i-0ba65762 i-0ba65762 2007-12-20T09:45:37+0000 Linux version 2.6.16-xenU (builder@xenbat.amazonsa) (gcc version 4.0.1 20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007 BIOS-provided physical RAM map: Xen: 0000000000000000 - 000000006a400000 (usable) 980MB HIGHMEM available. 727MB LOWMEM available. NX (Execute Disable) protection: active IRQ lockup detection disabled Built 1 zonelists Kernel command line: root=/dev/sda1 ro 4 Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 PID hash table entries: 4096 (order: 12, 65536 bytes) Xen reported: 2600.000 MHz processor. Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) Software IO TLB disabled vmalloc area: ee000000-f53fe000, maxmem 2d7fe000 Memory: 1718700k/1748992k available (1958k kernel code, 20948k reserved, 620k data, 144k init, 1003528k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay using timer specific routine.. 5202.10 BogoMIPS (lpj=26010507) Mount-cache hash table entries: 512 CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line) CPU: L2 Cache: 1024K (64 bytes/line) Checking 'hlt' instruction... OK. Brought up 1 CPUs migration_cost=0 Grant table initialized NET: Registered protocol family 16 Brought up 1 CPUs xen_mem: Initialising balloon driver. highmem bounce pool size: 64 pages VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Initializing Cryptographic API io scheduler noop registered io scheduler anticipatory registered (default) io scheduler deadline registered io scheduler cfq registered i8042.c: No controller found. RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize Xen virtual console successfully installed as tty1 Event-channel device installed. netfront: Initialising virtual ethernet driver. mice: PS/2 mouse device common for all mice md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 md: bitmap version 4.39 NET: Registered protocol family 2 Registering block device major 8 IP route cache hash table entries: 65536 (order: 6, 262144 bytes) TCP established hash table entries: 262144 (order: 9, 2097152 bytes) TCP bind hash table entries: 65536 (order: 7, 524288 bytes) TCP: Hash tables configured (established 262144 bind 65536) TCP reno registered TCP bic registered NET: Registered protocol family 1 NET: Registered protocol family 17 NET: Registered protocol family 15 Using IPI No-Shortcut mode md: Autodetecting RAID arrays. md: autorun ... md: ... autorun DONE. kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. Freeing unused kernel memory: 144k freed INIT: version 2.85 booting /etc/rc.d/rc.sysinit: line 80: /dev/tty1: Read-only file system /etc/rc.d/rc.sysinit: line 80: /dev/tty2: Read-only file system /etc/rc.d/rc.sysinit: line 80: /dev/tty3: Read-only file system /etc/rc.d/rc.sysinit: line 80: /dev/tty4: Read-only file system /etc/rc.d/rc.sysinit: line 80: /dev/tty5: Read-only file system /etc/rc.d/rc.sysinit: line 80: /dev/tty6: Read-only file system /etc/rc.d/rc.sysinit: line 84: /dev/tty7: Read-only file system /etc/rc.d/rc.sysinit: line 85: /dev/tty8: Read-only file system Starting udev: [ OK ] Initializing hardware... mmap /dev/mem: Bad address mmap /dev/mem: Bad address storage network audio done[ OK ] mount: can't find /dev/pts in /etc/fstab or /etc/mtab Configuring kernel parameters: [ OK ] Setting clock (localtime): Thu Dec 20 17:40:54 PHT 2007 [ OK ] Setting hostname localhost.localdomain: [ OK ] Checking root filesystem [/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1 /dev/sda1: clean, 89995/1313280 files, 779997/2621440 blocks [ OK ] Remounting root filesystem in read-write mode: [ OK ] Setting up Logical Volume Management: [ OK ] mount: can't find /dev/pts in /etc/fstab or /etc/mtab Checking filesystems Checking all file systems. [/sbin/fsck.ext3 (1) -- /mnt] fsck.ext3 -a /dev/sda2 /dev/sda2: clean, 12/19546112 files, 661386/39088128 blocks [ OK ] Mounting local filesystems: [ OK ] Enabling local filesystem quotas: [ OK ] Enabling swap space: [ OK ] INIT: Entering runlevel: 4 Entering non-interactive startup Starting sysstat: [ OK ] Setting network parameters: [ OK ] Bringing up loopback interface: [ OK ] Bringing up interface eth0: [ OK ] Starting system logger: [ OK ] Starting kernel logger: [ OK ] Starting irqbalance: [ OK ] Mounting other filesystems: [ OK ] Starting lm_sensors: [ OK ] Starting automount:No Mountpoints Defined[ OK ] Starting sshd:[ OK ] ntpd: Synchronizing with time server: [ OK ] Starting ntpd: [ OK ] Starting vsftpd for vsftpd: [ OK ] Starting ddclient: [ OK ] Starting s3oracle_init: [ OK ] Starting crond: [ OK ] Starting xfs: [ OK ] Starting oracle: [ OK ] Starting anacron: [ OK ] Starting atd: [ OK ] Starting system message bus: [ OK ] Starting HAL daemon: [ OK ] Linux Kernel 2.6.16-xenU on an i686 INIT: Switching to runlevel: 0 INIT: Sending processes the TERM signal Stopping HAL daemon: [ OK ] Stopping system message bus: [ OK ] Stopping atd: [ OK ] Shutting down xfs: [ OK ] Stopping oracle: [ OK ] Stopping sshd:[ OK ] Shutting down ddclient: [ OK ] Stopping s3oracle_init: [ OK ] Shutting down vsftpd: [ OK ] Stopping crond: [ OK ] Shutting down ntpd: [ OK ] Shutting down kernel logger: [ OK ] Shutting down system logger: [ OK ] Shutting down interface eth0: [ OK ] Shutting down loopback interface: [ OK ] Stopping sysstat: [ OK ] Starting killall: [ OK ] Sending all processes the TERM signal... Sending all processes the KILL signal... Saving random seed: Syncing hardware clock to system time Turning off swap: Turning off quotas: Unmounting file systems: Halting system... md: stopping all md devices. md: md0 switched to read-only mode. System halted.
Notice above that the script above download first the datafiles before Oracle starts, and upload the files when Oracle is stopped.
So far, this serves as a simple backup procedure to S3 from EC2. Next challenge would probably get the DB backup every 20mins.
Some of my references are(I thank them for sharing their ideas also):
http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync
http://khalid.blogspot.com.pk/2007/11/14/how-i-got-my-oracle-10g-xe-installation-running-on-amazon-ec2-and-s3