Yearly Archives: 2007

Oracle10g and Amazon EC2 simple backup to S3

Today, I was bundling my AMI capable of loading from S3 during bootup and upload files before shutdown. Using the the S3Sync and simple shell scripts I made.

At first I was having trouble getting the init script run during shutdown, until I recheck the chkconfig command manual.A lock file solves the problem, which I absolutely missed at first. šŸ™‚

I have created the scripts below:

#!/bin/bash
#
# Run-level Startup script for the Oracle s3 backup
#
# chkconfig: 345 81 36
# Filename: s3oracle_init
# description: Startup/Shutdown download S3 Oracle datafiles.

if [ -f /var/lock/subsys/oracle ]
 then
       echo "S3 Datafile Download cancelled. "
       echo "Oracle is running. Shutdown Oracle first."
       exit 1
fi

case "$1" in
     start)

         # Oracle is not running.. ok to download files.
         /usr/ora_scripts/s3download.sh
         touch /var/lock/subsys/s3oracle_init
         echo "OK"
         ;;
       stop)
         /usr/ora_scripts/s3upload.sh
         rm -f /var/lock/subsys/s3oracle_init
         echo "OK"
         ;;
       reload|restart)
        # $0 stop
        # $0 start
         ;;
     *)
         echo "Usage: $0 start|stop|restart|reload"
         exit 1
 esac
exit 0


#!/bin/sh
#
#
# Filename: s3download.sh
# Description: download S3 datafiles.
# S3 access keys are defined at s3config.yml or can be
# done by exporting the variables below:
#
export AWS_ACCESS_KEY_ID=valueGoesHere
export AWS_SECRET_ACCESS_KEY=valuesGoesHere
export SSL_CERT_DIR=valuesGoesHere

S3SYNC_DIR=/root/s3sync
DEST_DIR=/mnt/samdata

# s3 bucket, s3sync format ex. bucketname:key
BUCKET=samdata:

# check dest dir
if [ ! -d $DEST_DIR ]; then
       if [ -f $DEST_DIR ]; then
         echo "$DEST_DIR is a file. Deleting $DEST_DIR.."
         rm -rf $DEST_DIR
         echo "OK"
       fi
       # create a new directory
       echo "Creating $DEST_DIR"
       mkdir $DEST_DIR
       echo "OK"
else
       echo "Opps! $DEST_DIR found."
fi

### download s3 datafiles
###--code adapted from:
### http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/

echo -n "Downloading s3 $BUCKET files to $DEST_DIR"
cd $S3SYNC_DIR
ruby s3sync.rb -r  --delete $BUCKET $DEST_DIR
echo ""
echo "DONE"
exit 0


#!/bin/sh
#
#
# Filename: s3upload.sh
# Description: upload S3 datafiles.
# S3 access keys and cert dir are defined inside the file $S3SYNC_DIR/s3config.yml

S3SYNC_DIR=/root/s3sync
SRC_DIR=/mnt/samdata

export AWS_ACCESS_KEY_ID=valueGoesHere
export AWS_SECRET_ACCESS_KEY=valuesGoesHere
export SSL_CERT_DIR=valuesGoesHere

# s3 bucket, s3sync format ex. bucketname:key
# tagging upload by date
BUCKET=samdata:/

### download s3 datafiles
### adapated from
### http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/

echo -n "Uploading $SRC_DIR files to $BUCKET"
cd $S3SYNC_DIR
ruby s3sync.rb -r  --delete $SRC_DIR $BUCKET
echo ""
echo "DONE"
exit 0

After setting up the AWS variables , I test first the s3upload.sh and s3download.sh and Iā€™m getting lucky to have it running well.

I start bundling the image, by excluding directories I want to be hidden on my new image. Upload it to S3, and register it as new AMI. After a while I can check the console output of the instance using the ec2-get-console-output command. Shown below is my session showing the my s3oracle_init service working very fine.

~$ ec2-run-instances ami-1020c579 -k gsg-keypair
RESERVATION     r-d928c0b0      608316554735    default
INSTANCE        i-0ba65762      ami-1020c579                    pending gsg-keypair     0
                pending gsg-keypair     0

(After seeing the instance is running, it takes some time though depending
on your datafiles to download from S3, I make sure the file is download by checking the console output.)


~$ ec2-terminate-instances  i-0ba65762
INSTANCE        i-0ba65762      running shutting-down

~$ ec2-get-console-output i-0ba65762
i-0ba65762
2007-12-20T09:45:37+0000
Linux version 2.6.16-xenU (builder@xenbat.amazonsa) (gcc version 4.0.1 20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 000000006a400000 (usable)
980MB HIGHMEM available.
727MB LOWMEM available.
NX (Execute Disable) protection: active
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro 4
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 65536 bytes)
Xen reported: 2600.000 MHz processor.
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Software IO TLB disabled
vmalloc area: ee000000-f53fe000, maxmem 2d7fe000
Memory: 1718700k/1748992k available (1958k kernel code, 20948k reserved, 620k data, 144k init, 1003528k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5202.10 BogoMIPS (lpj=26010507)
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
Checking 'hlt' instruction... OK.
Brought up 1 CPUs
migration_cost=0
Grant table initialized
NET: Registered protocol family 16
Brought up 1 CPUs
xen_mem: Initialising balloon driver.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
i8042.c: No controller found.
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
netfront: Initialising virtual ethernet driver.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
Registering block device major 8
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI No-Shortcut mode
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 144k freed
INIT: version 2.85 booting
/etc/rc.d/rc.sysinit: line 80: /dev/tty1: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty2: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty3: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty4: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty5: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty6: Read-only file system
/etc/rc.d/rc.sysinit: line 84: /dev/tty7: Read-only file system
/etc/rc.d/rc.sysinit: line 85: /dev/tty8: Read-only file system

Starting udev:  [  OK  ]
Initializing hardware... mmap /dev/mem: Bad address
mmap /dev/mem: Bad address
storage network audio done[  OK  ]
mount: can't find /dev/pts in /etc/fstab or /etc/mtab
Configuring kernel parameters:  [  OK  ]
Setting clock  (localtime): Thu Dec 20 17:40:54 PHT 2007 [  OK  ]
Setting hostname localhost.localdomain:  [  OK  ]
Checking root filesystem
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
/dev/sda1: clean, 89995/1313280 files, 779997/2621440 blocks
[  OK  ]
Remounting root filesystem in read-write mode:  [  OK  ]
Setting up Logical Volume Management: [  OK  ]
mount: can't find /dev/pts in /etc/fstab or /etc/mtab
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /mnt] fsck.ext3 -a /dev/sda2
/dev/sda2: clean, 12/19546112 files, 661386/39088128 blocks
[  OK  ]
Mounting local filesystems:  [  OK  ]
Enabling local filesystem quotas:  [  OK  ]
Enabling swap space:  [  OK  ]
INIT: Entering runlevel: 4
Entering non-interactive startup
Starting sysstat:  [  OK  ]
Setting network parameters:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [  OK  ]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting irqbalance: [  OK  ]
Mounting other filesystems:  [  OK  ]
Starting lm_sensors:  [  OK  ]
Starting automount:No Mountpoints Defined[  OK  ]
Starting sshd:[  OK  ]
ntpd: Synchronizing with time server: [  OK  ]
Starting ntpd: [  OK  ]
Starting vsftpd for vsftpd: [  OK  ]
Starting ddclient: [  OK  ]
Starting s3oracle_init:  [  OK  ]
Starting crond: [  OK  ]
Starting xfs: [  OK  ]
Starting oracle:  [  OK  ]
Starting anacron: [  OK  ]
Starting atd: [  OK  ]
Starting system message bus: [  OK  ]
Starting HAL daemon: [  OK  ]

Linux
Kernel 2.6.16-xenU on an i686

INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
Stopping HAL daemon: [  OK  ]
Stopping system message bus: [  OK  ]
Stopping atd: [  OK  ]
Shutting down xfs: [  OK  ]
Stopping oracle:  [  OK  ]
Stopping sshd:[  OK  ]
Shutting down ddclient: [  OK  ]
Stopping s3oracle_init:  [  OK  ]
Shutting down vsftpd: [  OK  ]
Stopping crond: [  OK  ]
Shutting down ntpd: [  OK  ]
Shutting down kernel logger: [  OK  ]
Shutting down system logger: [  OK  ]
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Stopping sysstat:  [  OK  ]
Starting killall:  [  OK  ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Turning off swap:
Turning off quotas:
Unmounting file systems:
Halting system...
md: stopping all md devices.
md: md0 switched to read-only mode.
System halted.

Notice above that the script above download first the datafiles before Oracle starts, and upload the files when Oracle is stopped.
So far, this serves as a simple backup procedure to S3 from EC2. Next challenge would probably get the DB backup every 20mins.

Some of my references are(I thank them for sharing their ideas also):
http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync
http://khalid.blogspot.com.pk/2007/11/14/how-i-got-my-oracle-10g-xe-installation-running-on-amazon-ec2-and-s3

Amazon EC2 and S3 my Invasion

For the past 3 months, I was playing with sample AMI’s for testing purposes. I tried to build an RHEL image from my VMWare and try to bundle it for AWS. So far so good. But after loading and registering it to AWS, and run an instance of it.

My problems encountered:

1. On the first case, I tried to upgrade the ruby version to1.8.5 in my host machine(getting rid of ruby errors)
2. Time not sync with Amazon.. need to run NTP on the host machine before uploading the image.
3. EXT2-fs: unable to read superblock and Kernel Panic
a. Workaround for this one is to make sure your /etc/fstab (ex. /mnt is mounted /dev/sda2)
b. And to copy the 2.6.16-xenU /lib/modules

After this, I started to customize the running AMI at the amazon at and install software as needed. Rebundle and upload it back to Amazon S3.

Rather than having the ready made AMI to work with, I built our own AMI, the learning process is what makes you feel confident, but experience ec2 guys would say “it best to use the existing AMI”. I wont argue, but naturally now,I can build more stuff from using existing AMIs the faster way (confident ec2 user’s way).

Soon, I discoverted the Rightscale and the Firefox addon to manage my instance. But still I can’t do away the command line utilities.

Data Persistent

To make our data persistent is s3sync which is open source. I like this better. I’m currently writing a simple script that will download the data from s3 to ec2 instance upon boot, and upload it back before shutdown. On special cases, if the instance would not crash this is solution will definitely great, but in some cases where my instance crashes, I will be losing some data. The next step, is to use back using s3sync using cron to update the s3 maybe hourly or daily, a script would do this for me.. šŸ™‚

It came to a point where I need a lots of dose of caffiene to get the persistency problem from EC2 to S3. I have tested the ElasticDrive, though there’s a 40Gb limit for free use, thinking in the future if your data gets bigger than that, there extra charges. Reading the forum for ElasticDrive support gets you into S3 as fuse base filesystem. Mounting the S3 buckets as virtual drive. I need to test this further though.

Until next time.

Rodney