MongoDB using ec2-consistent-snapshot

I got this idea to test Eric Lubow’s ec2-consistent-snapshot With Mongo.
Using MongoDB 1.8.1 and Ubuntu Lucid AMI from www.alestic.com

On a running Ubuntu Lucid EC2 instance, connect using SSH and do the following:

1. Setup MongoDB. We stop it so we can move it’s files to the RAID volume.

echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' |   \
sudo tee /etc/apt/sources.list.d/10gen.list 
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 

sudo apt-get update 
sudo apt-get -y install mongodb-10gen 
sudo /etc/init.d/mongodb stop 

2. Create 4 EBS volumes on the same zone of the instance and attached the volumes using the devices from /dev/sdh to /dev/sdk

3. Setup RAID0 volumes for MongoDb.

sudo apt-get install -y mdadm xfsprogs
sudo mdadm --create /dev/md0 --level 0 --chunk=256  \
  --metadata=1.1 --raid-devices=2 /dev/sdh /dev/sdi /dev/sdj /dev/sdk 

echo DEVICE /dev/sdh /dev/sdi /dev/sdj /dev/sdk  | sudo tee /etc/mdadm/mdadm.conf
sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

sudo mkfs.xfs /dev/md0
echo "/dev/md0 /raiddrive xfs noatime 0 0" | sudo tee -a /etc/fstab

sudo mkdir /raiddrive
sudo mount /raiddrive

sudo mv /var/lib/mongodb /raiddrive/
sudo mv /var/log/mongodb /raiddrive/mongodb-log

sudo mkdir /var/lib/mongodb 
sudo mkdir /var/log/mongodb 

echo "/raiddrive/mongodb /var/lib/mongodb none bind" | sudo tee -a /etc/fstab 
echo "/raiddrive/mongodb-log /var/log/mongodb none bind"  |   \
sudo tee -a /etc/fstab 

sudo mount /var/lib/mongodb 
sudo mount /var/lib/mongodb

sudo /etc/init.d/mongodb start 

4. Install Alestic ec2-consistent-snapshot

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv BE09C571
echo 'deb http://ppa.launchpad.net/alestic/ppa/ubuntu lucid main' |   \
sudo tee -a  /etc/apt/sources.list.d/alestic-ppa-lucid.list
sudo apt-get update 
sudo apt-get install -y ec2-consistent-snapshot

5. Use ec2-consistent-snapshot with MongoDB support which I modified a bit for Ubuntu. It’s on https://github.com/cocoy/ec2-consistent-snapshot.

sudo apt-get -y install build-essential libio-socket-ssl-perl libdatetime-perl 
sudo PERL_MM_USE_DEFAULT=1 cpan -fi MongoDB MongoDB::Admin

6. Copy this ec2-consistent-snapshot to /usr/bin/ replacing old file. And make it executable.

 sudo wget -q https://github.com/cocoy/ec2-consistent-snapshot/raw/master/ec2-consistent-snapshot \
   -O /usr/bin/ec2-consistent-snapshot
sudo chmod +x /usr/bin/ec2-consistent-snapshot

7. Run the snapshot.

export AMAZON_ACCESS_KEY_ID=youraccesskeyid
export AWS_SECRET_ACCESS_KEY=yoursecretkey
sudo ec2-consistent-snapshot --debug \
--mongo \
--xfs-filesystem=/raiddrive/ \ 
--region us-east-1 \
--description "MongoDB RAID snapshot $(date +'%Y-%m-%d %H:%M:%S')"  \
vol-xxxx vol-xxxxx vol-xxxxx vol-xxxxx

It should be resulting to something like:

ec2-consistent-snapshot: Using AWS access key: xxxxxxx
ec2-consistent-snapshot: Wed Jun  8 09:11:56 2011: mongo connect on localhost:27017
ec2-consistent-snapshot: Wed Jun  8 09:11:56 2011: locking mongo
ec2-consistent-snapshot: Wed Jun  8 09:11:56 2011: mongo locked
ec2-consistent-snapshot: Wed Jun  8 09:11:56 2011: sync
ec2-consistent-snapshot: Wed Jun  8 09:11:56 2011: xfs_freeze -f /raiddrive/
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: create EC2 object
ec2-consistent-snapshot: Endpoint: https://ec2.us-east-1.amazonaws.com
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: ec2-create-snapshot vol-xxxx
snap-xxxxx
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: ec2-create-snapshot vol-xxxx
snap-xxxxx
...
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: xfs_freeze -u /raiddrive/
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: unlocking mongo
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: mongo unlocked
ec2-consistent-snapshot: Wed Jun  8 09:11:58 2011: done

References:
ec2-consistent-snapshot With Mongo
EBS RAID volumes
https://launchpad.net/~alestic/+archive/ppa
MongoDB EC2 Backup and Restore

7 thoughts on “MongoDB using ec2-consistent-snapshot

  1. Brett

    This is a great solution, and you’ve packaged it in an easy to digest and executable way.

    The big question left on my end is how to restore from the snapshot in a similarly simple one step way?

  2. rodney Post author

    Hi Brett,

    To restore you have to:
    1. Create the new volumes from the snapshots.
    2. Create a new instance with the same installed packages mdadm, xfsprogs, and mongodb
    3. Stop mongodb.
    4. Attached the new volumes similar to the old volumes devices. Should be on /dev/sdh /dev/sdi /dev/sdj /dev/sdk
    5. Copy the /etc/mdadm/mdadm.cnf and restart mdadm.
    6. Mount the directories.
    7. Start MongoDB.

    I like your idea. It would be nice to have a restore tutorial. 🙂

  3. David Nesbitt

    Does the mongod.lock file exist when the snapshot is taken? I am concerned about having to a repair process as part of the restore. Do you have any experience with this?

  4. rodney Post author

    Hi David,

    The mongod.lock file exists when snapshot is taken.
    I suggest to work first with a mongo with smaller size EBS volumes, perform the snapshots and restore procedure and do the same process for the production.

  5. David Nesbitt

    Thanks, Rodney. I will check with 10gen to see if it is safe to remove the mongod.lock file without doing a repair. I am assuming it is, because of the fsync_lock, but I would like to get their confirmation.

  6. Dara Dan

    Hey,
    thanks for the great post. this is working fine when you have one shard, but what would you suggest when there are more shards. shouldnt we snapshot all the shards in the same time for consistency. imagine shards with global and sharded collections. would be great to hear your ideas 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *