I got this idea to test Eric Lubow’s ec2-consistent-snapshot With Mongo.
Using MongoDB 1.8.1 and Ubuntu Lucid AMI from www.alestic.com
On a running Ubuntu Lucid EC2 instance, connect using SSH and do the following:
1. Setup MongoDB. We stop it so we can move it’s files to the RAID volume.
echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | \ sudo tee /etc/apt/sources.list.d/10gen.list sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10 sudo apt-get update sudo apt-get -y install mongodb-10gen sudo /etc/init.d/mongodb stop
2. Create 4 EBS volumes on the same zone of the instance and attached the volumes using the devices from /dev/sdh to /dev/sdk
3. Setup RAID0 volumes for MongoDb.
sudo apt-get install -y mdadm xfsprogs sudo mdadm --create /dev/md0 --level 0 --chunk=256 \ --metadata=1.1 --raid-devices=2 /dev/sdh /dev/sdi /dev/sdj /dev/sdk echo DEVICE /dev/sdh /dev/sdi /dev/sdj /dev/sdk | sudo tee /etc/mdadm/mdadm.conf sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf sudo mkfs.xfs /dev/md0 echo "/dev/md0 /raiddrive xfs noatime 0 0" | sudo tee -a /etc/fstab sudo mkdir /raiddrive sudo mount /raiddrive sudo mv /var/lib/mongodb /raiddrive/ sudo mv /var/log/mongodb /raiddrive/mongodb-log sudo mkdir /var/lib/mongodb sudo mkdir /var/log/mongodb echo "/raiddrive/mongodb /var/lib/mongodb none bind" | sudo tee -a /etc/fstab echo "/raiddrive/mongodb-log /var/log/mongodb none bind" | \ sudo tee -a /etc/fstab sudo mount /var/lib/mongodb sudo mount /var/lib/mongodb sudo /etc/init.d/mongodb start
4. Install Alestic ec2-consistent-snapshot
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv BE09C571 echo 'deb http://ppa.launchpad.net/alestic/ppa/ubuntu lucid main' | \ sudo tee -a /etc/apt/sources.list.d/alestic-ppa-lucid.list sudo apt-get update sudo apt-get install -y ec2-consistent-snapshot
5. Use ec2-consistent-snapshot with MongoDB support which I modified a bit for Ubuntu. It’s on https://github.com/cocoy/ec2-consistent-snapshot.
sudo apt-get -y install build-essential libio-socket-ssl-perl libdatetime-perl sudo PERL_MM_USE_DEFAULT=1 cpan -fi MongoDB MongoDB::Admin
6. Copy this ec2-consistent-snapshot to /usr/bin/ replacing old file. And make it executable.
sudo wget -q https://github.com/cocoy/ec2-consistent-snapshot/raw/master/ec2-consistent-snapshot \ -O /usr/bin/ec2-consistent-snapshot sudo chmod +x /usr/bin/ec2-consistent-snapshot
7. Run the snapshot.
export AMAZON_ACCESS_KEY_ID=youraccesskeyid export AWS_SECRET_ACCESS_KEY=yoursecretkey sudo ec2-consistent-snapshot --debug \ --mongo \ --xfs-filesystem=/raiddrive/ \ --region us-east-1 \ --description "MongoDB RAID snapshot $(date +'%Y-%m-%d %H:%M:%S')" \ vol-xxxx vol-xxxxx vol-xxxxx vol-xxxxx
It should be resulting to something like:
ec2-consistent-snapshot: Using AWS access key: xxxxxxx ec2-consistent-snapshot: Wed Jun 8 09:11:56 2011: mongo connect on localhost:27017 ec2-consistent-snapshot: Wed Jun 8 09:11:56 2011: locking mongo ec2-consistent-snapshot: Wed Jun 8 09:11:56 2011: mongo locked ec2-consistent-snapshot: Wed Jun 8 09:11:56 2011: sync ec2-consistent-snapshot: Wed Jun 8 09:11:56 2011: xfs_freeze -f /raiddrive/ ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: create EC2 object ec2-consistent-snapshot: Endpoint: https://ec2.us-east-1.amazonaws.com ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: ec2-create-snapshot vol-xxxx snap-xxxxx ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: ec2-create-snapshot vol-xxxx snap-xxxxx ... ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: xfs_freeze -u /raiddrive/ ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: unlocking mongo ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: mongo unlocked ec2-consistent-snapshot: Wed Jun 8 09:11:58 2011: done
References:
ec2-consistent-snapshot With Mongo
EBS RAID volumes
https://launchpad.net/~alestic/+archive/ppa
MongoDB EC2 Backup and Restore
Good Stuff!
This is a great solution, and you’ve packaged it in an easy to digest and executable way.
The big question left on my end is how to restore from the snapshot in a similarly simple one step way?
Hi Brett,
To restore you have to:
1. Create the new volumes from the snapshots.
2. Create a new instance with the same installed packages mdadm, xfsprogs, and mongodb
3. Stop mongodb.
4. Attached the new volumes similar to the old volumes devices. Should be on /dev/sdh /dev/sdi /dev/sdj /dev/sdk
5. Copy the /etc/mdadm/mdadm.cnf and restart mdadm.
6. Mount the directories.
7. Start MongoDB.
I like your idea. It would be nice to have a restore tutorial. 🙂
Does the mongod.lock file exist when the snapshot is taken? I am concerned about having to a repair process as part of the restore. Do you have any experience with this?
Hi David,
The mongod.lock file exists when snapshot is taken.
I suggest to work first with a mongo with smaller size EBS volumes, perform the snapshots and restore procedure and do the same process for the production.
Thanks, Rodney. I will check with 10gen to see if it is safe to remove the mongod.lock file without doing a repair. I am assuming it is, because of the fsync_lock, but I would like to get their confirmation.
Hey,
thanks for the great post. this is working fine when you have one shard, but what would you suggest when there are more shards. shouldnt we snapshot all the shards in the same time for consistency. imagine shards with global and sharded collections. would be great to hear your ideas 🙂