Moving a BackupPC Pool

I have been using BackupPC to automatically back up the systems on my LAN for years now. It started out with a 3x250GB RAID5 as the storage pool and when I ran out of space on that I added another disk to bring it up to about 700GB. BackupPC does an excellent job of pooling common files together so that they don’t take up extra space. This is especially useful if you are backing up system files on multiple systems running the same OS release.

The way it achieves this is through the use of hardlinks. A hardlink references the inode of the original file instead of making a copy of its data. This only works on a single filesystem since inodes are not unique between filesystems. When you have 700GB of data backed up this results in alot of hardlinks. So many in fact that none of the file copy tools available can deal with copying them to a new filesystem.

BackupPC also keeps track of how full the pool is, and when it hit about 95% I started getting emails about it not backing up systems. As it expires old backups it would drop back down to 90% but I realized it was finally time to tackle the problem of moving the pool to a new RAID1 array. I decided to go with RAID1 of 2T drives instead of a RAID5 because as drives get bigger the chances of hitting an unrecoverable error increase. This means that when a 4 disk RAID5 loses a disk and you insert a new one it becomes more likely that you will hit another error before the recovery is complete. RAID10 helps mitigate this somewhat, using a RAID5 of mirrors, but that pretty much defeats the ‘Inexpensive’ part of RAID. The article “RAID’s Days May Be Numbered” covers these issues pretty well.

I am using Linux’s mdraid to manage the array. This array was originally created while running Debian on an old single CPU 2.8GHz Celeron. Over the years I have upgraded the hardware and even switched to using Fedora without any problems recognizing the RAID array. I don’t remember exactly why, but it was setup as a single LVM physical volume with a single VG/LV on top of that using ext3. These days I typically skip LVM and setup ext4 right on the disk to eliminate any unneeded complexity. But in this case I’m glad I had LVM setup, and will continue to use it for large blocks of extra disk.

The most common solution to moving a BackupPC pool is to use tar to pipe to another tar process that uncompresses it on the target filesystem. tar knows how to handle hardlinks, devices, etc. So it is a god choice for copying things around when you can’t use rsync. But when I gave this a try it ran fairly quickly up to about 90% and then slowed way down. So slow that I could count the number of blocks being transferred using df. According to atop and iotop it was spending all of its time reading and very little time writing. I’m not sure why, and didn’t feel like stracing it to figure it out.

The next most common method is using dd to copy the filesystem over and then use resize tools to grow it. This would probably have worked, but I was concerned about it also copying UUIDs and ending up with a cloned fs and a confused fstab. About this time I realized it was actually on LVM and that I could use its tools to handle the move.

The LVM Howto provides some pretty good examples of how to do things with LVM, including how to remove a disk. Section 13.5.2 covers moving things to a new disk. Here’s what I did (note, this worked for me, may not for you, YMMV). I’d remind you to backup your data, but in this case that’s exactly what we’re trying to accomplish, so…

I used parted to create a GPT disk label and partition the whole drive into a single partition. I also set the raid flag on the partition.

I setup the new array as /dev/md3 like this:

mdadm --create /dev/md3 --level=raid1 --raid-devices=2 /dev/sdb1 /dev/sdc1

In order to prevent mdadm from syncing the disks immediately I froze the sync:

echo frozen > /sys/blocl/md3/md/sync_action

This may not have been needed, but seems like a good idea to get the actual data moved over first, then let it sync.

Don’t forget to shut down backuppc and unmount the existing pool. All of these steps should be done on an unmounted system, just to be safe.

The existing RAID5 is called raid5-vol1 and it is on /dev/md0, I moved the data over to the new array using LVM tools:

pvcreate /dev/md3
vgextend raid5-vol1 /dev/md3
pvmove /dev/md0 /dev/md3

The move took about 4 hours and has regular updates. I suggest running all of this inside screen, just in case your ssh session/teminal/whatever goes away.

After that is done I removed the old array from the VG:

vgreduce raid5-vol1 /dev/md0

Now all the data is on the new array, but it is not able to take advantage of the extra space of the larger drives. So we need to resize the lv. Run vgdisplay and look at the ‘Total PE’, this is the maximum number of PE’s available. You will notice that the Alloc PE matches the size of the old array. We are going to resize it to use all of them with:

lvresize -l xxxxxx /dev/raid5-vol1/raid5

Where xxxxxx is the Total PE number. This operation is very quick. Now you are ready to resize the filesystem itself. I’m using ext3 so I use the e2fs tools to operate on it, first running a filesystem check to make sure there are no errors on the fs and then resizing it to use all the available space on the underlying device.

e2fsck -f /dev/radi5-vol1/raid5
resize2fs -p /dev/raid5-vol1/raid5

When that finishes (took about an hour and a half for me) you can then re-enable the array sync using:

echo check > /sys/blocl/md3/md/sync_action

And then mount your new pool and restart backuppc.

Pay careful attention to the PE number you use when resizing. It will ask you if you make a mistake and try to shrink it (yes, I found this out when using the -L option instead of -l). It is also easy to cut and paste the wrong number and not grow it as much as you thought. Again, I did this, only noticing when reviewing things for this article. I must have picked the free space number because I still had about 700GB of PE’s available. I followed the steps to lvresize and resize2fs above and am now using the full amount of space available.