Oracle10g and Amazon EC2 simple backup to S3

Today, I was bundling my AMI capable of loading from S3 during bootup and upload files before shutdown. Using the the S3Sync and simple shell scripts I made.

At first I was having trouble getting the init script run during shutdown, until I recheck the chkconfig command manual.A lock file solves the problem, which I absolutely missed at first. šŸ™‚

I have created the scripts below:

# Run-level Startup script for the Oracle s3 backup
# chkconfig: 345 81 36
# Filename: s3oracle_init
# description: Startup/Shutdown download S3 Oracle datafiles.

if [ -f /var/lock/subsys/oracle ]
       echo "S3 Datafile Download cancelled. "
       echo "Oracle is running. Shutdown Oracle first."
       exit 1

case "$1" in

         # Oracle is not running.. ok to download files.
         touch /var/lock/subsys/s3oracle_init
         echo "OK"
         rm -f /var/lock/subsys/s3oracle_init
         echo "OK"
        # $0 stop
        # $0 start
         echo "Usage: $0 start|stop|restart|reload"
         exit 1
exit 0

# Filename:
# Description: download S3 datafiles.
# S3 access keys are defined at s3config.yml or can be
# done by exporting the variables below:
export AWS_ACCESS_KEY_ID=valueGoesHere
export AWS_SECRET_ACCESS_KEY=valuesGoesHere
export SSL_CERT_DIR=valuesGoesHere


# s3 bucket, s3sync format ex. bucketname:key

# check dest dir
if [ ! -d $DEST_DIR ]; then
       if [ -f $DEST_DIR ]; then
         echo "$DEST_DIR is a file. Deleting $DEST_DIR.."
         rm -rf $DEST_DIR
         echo "OK"
       # create a new directory
       echo "Creating $DEST_DIR"
       mkdir $DEST_DIR
       echo "OK"
       echo "Opps! $DEST_DIR found."

### download s3 datafiles
###--code adapted from:

echo -n "Downloading s3 $BUCKET files to $DEST_DIR"
ruby s3sync.rb -r  --delete $BUCKET $DEST_DIR
echo ""
echo "DONE"
exit 0

# Filename:
# Description: upload S3 datafiles.
# S3 access keys and cert dir are defined inside the file $S3SYNC_DIR/s3config.yml


export AWS_ACCESS_KEY_ID=valueGoesHere
export AWS_SECRET_ACCESS_KEY=valuesGoesHere
export SSL_CERT_DIR=valuesGoesHere

# s3 bucket, s3sync format ex. bucketname:key
# tagging upload by date

### download s3 datafiles
### adapated from

echo -n "Uploading $SRC_DIR files to $BUCKET"
ruby s3sync.rb -r  --delete $SRC_DIR $BUCKET
echo ""
echo "DONE"
exit 0

After setting up the AWS variables , I test first the and and Iā€™m getting lucky to have it running well.

I start bundling the image, by excluding directories I want to be hidden on my new image. Upload it to S3, and register it as new AMI. After a while I can check the console output of the instance using the ec2-get-console-output command. Shown below is my session showing the my s3oracle_init service working very fine.

~$ ec2-run-instances ami-1020c579 -k gsg-keypair
RESERVATION     r-d928c0b0      608316554735    default
INSTANCE        i-0ba65762      ami-1020c579                    pending gsg-keypair     0
                pending gsg-keypair     0

(After seeing the instance is running, it takes some time though depending
on your datafiles to download from S3, I make sure the file is download by checking the console output.)

~$ ec2-terminate-instances  i-0ba65762
INSTANCE        i-0ba65762      running shutting-down

~$ ec2-get-console-output i-0ba65762
Linux version 2.6.16-xenU (builder@xenbat.amazonsa) (gcc version 4.0.1 20050727 (Red Hat 4.0.1-5)) #1 SMP Mon May 28 03:41:49 SAST 2007
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 000000006a400000 (usable)
980MB HIGHMEM available.
727MB LOWMEM available.
NX (Execute Disable) protection: active
IRQ lockup detection disabled
Built 1 zonelists
Kernel command line:  root=/dev/sda1 ro 4
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
PID hash table entries: 4096 (order: 12, 65536 bytes)
Xen reported: 2600.000 MHz processor.
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Software IO TLB disabled
vmalloc area: ee000000-f53fe000, maxmem 2d7fe000
Memory: 1718700k/1748992k available (1958k kernel code, 20948k reserved, 620k data, 144k init, 1003528k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 5202.10 BogoMIPS (lpj=26010507)
Mount-cache hash table entries: 512
CPU: L1 I Cache: 64K (64 bytes/line), D cache 64K (64 bytes/line)
CPU: L2 Cache: 1024K (64 bytes/line)
Checking 'hlt' instruction... OK.
Brought up 1 CPUs
Grant table initialized
NET: Registered protocol family 16
Brought up 1 CPUs
xen_mem: Initialising balloon driver.
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
io scheduler noop registered
io scheduler anticipatory registered (default)
io scheduler deadline registered
io scheduler cfq registered
i8042.c: No controller found.
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
Xen virtual console successfully installed as tty1
Event-channel device installed.
netfront: Initialising virtual ethernet driver.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
NET: Registered protocol family 2
Registering block device major 8
IP route cache hash table entries: 65536 (order: 6, 262144 bytes)
TCP established hash table entries: 262144 (order: 9, 2097152 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 262144 bind 65536)
TCP reno registered
TCP bic registered
NET: Registered protocol family 1
NET: Registered protocol family 17
NET: Registered protocol family 15
Using IPI No-Shortcut mode
md: Autodetecting RAID arrays.
md: autorun ...
md: ... autorun DONE.
kjournald starting.  Commit interval 5 seconds
EXT3-fs: mounted filesystem with ordered data mode.
VFS: Mounted root (ext3 filesystem) readonly.
Freeing unused kernel memory: 144k freed
INIT: version 2.85 booting
/etc/rc.d/rc.sysinit: line 80: /dev/tty1: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty2: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty3: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty4: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty5: Read-only file system
/etc/rc.d/rc.sysinit: line 80: /dev/tty6: Read-only file system
/etc/rc.d/rc.sysinit: line 84: /dev/tty7: Read-only file system
/etc/rc.d/rc.sysinit: line 85: /dev/tty8: Read-only file system

Starting udev:  [  OK  ]
Initializing hardware... mmap /dev/mem: Bad address
mmap /dev/mem: Bad address
storage network audio done[  OK  ]
mount: can't find /dev/pts in /etc/fstab or /etc/mtab
Configuring kernel parameters:  [  OK  ]
Setting clock  (localtime): Thu Dec 20 17:40:54 PHT 2007 [  OK  ]
Setting hostname localhost.localdomain:  [  OK  ]
Checking root filesystem
[/sbin/fsck.ext3 (1) -- /] fsck.ext3 -a /dev/sda1
/dev/sda1: clean, 89995/1313280 files, 779997/2621440 blocks
[  OK  ]
Remounting root filesystem in read-write mode:  [  OK  ]
Setting up Logical Volume Management: [  OK  ]
mount: can't find /dev/pts in /etc/fstab or /etc/mtab
Checking filesystems
Checking all file systems.
[/sbin/fsck.ext3 (1) -- /mnt] fsck.ext3 -a /dev/sda2
/dev/sda2: clean, 12/19546112 files, 661386/39088128 blocks
[  OK  ]
Mounting local filesystems:  [  OK  ]
Enabling local filesystem quotas:  [  OK  ]
Enabling swap space:  [  OK  ]
INIT: Entering runlevel: 4
Entering non-interactive startup
Starting sysstat:  [  OK  ]
Setting network parameters:  [  OK  ]
Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [  OK  ]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting irqbalance: [  OK  ]
Mounting other filesystems:  [  OK  ]
Starting lm_sensors:  [  OK  ]
Starting automount:No Mountpoints Defined[  OK  ]
Starting sshd:[  OK  ]
ntpd: Synchronizing with time server: [  OK  ]
Starting ntpd: [  OK  ]
Starting vsftpd for vsftpd: [  OK  ]
Starting ddclient: [  OK  ]
Starting s3oracle_init:  [  OK  ]
Starting crond: [  OK  ]
Starting xfs: [  OK  ]
Starting oracle:  [  OK  ]
Starting anacron: [  OK  ]
Starting atd: [  OK  ]
Starting system message bus: [  OK  ]
Starting HAL daemon: [  OK  ]

Kernel 2.6.16-xenU on an i686

INIT: Switching to runlevel: 0
INIT: Sending processes the TERM signal
Stopping HAL daemon: [  OK  ]
Stopping system message bus: [  OK  ]
Stopping atd: [  OK  ]
Shutting down xfs: [  OK  ]
Stopping oracle:  [  OK  ]
Stopping sshd:[  OK  ]
Shutting down ddclient: [  OK  ]
Stopping s3oracle_init:  [  OK  ]
Shutting down vsftpd: [  OK  ]
Stopping crond: [  OK  ]
Shutting down ntpd: [  OK  ]
Shutting down kernel logger: [  OK  ]
Shutting down system logger: [  OK  ]
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Stopping sysstat:  [  OK  ]
Starting killall:  [  OK  ]
Sending all processes the TERM signal...
Sending all processes the KILL signal...
Saving random seed:
Syncing hardware clock to system time
Turning off swap:
Turning off quotas:
Unmounting file systems:
Halting system...
md: stopping all md devices.
md: md0 switched to read-only mode.
System halted.

Notice above that the script above download first the datafiles before Oracle starts, and upload the files when Oracle is stopped.
So far, this serves as a simple backup procedure to S3 from EC2. Next challenge would probably get the DB backup every 20mins.

Some of my references are(I thank them for sharing their ideas also):

2 thoughts on “Oracle10g and Amazon EC2 simple backup to S3

  1. laurence

    Hi Rodney,

    Do you offer any Amazon EC2 consultation or consulting services?
    I want to work with this new technology using Oracle or Mysql and Ruby.


  2. cocoy

    Hi laurence,

    I been using Amazon aws for little longer now.You can reach me via my email posted in my profile section for details of your project.


Leave a Reply

Your email address will not be published. Required fields are marked *