Cleaning up after disk crashing season hasn’t been fun, but I am pleased with what I’ve managed to recover from the worst crash.
I wanted to get the latest data from the dead webserver. It was in MySQL, and stored in /var/lib/mysql. Unfortunately, the /var/lib directory no longer existed.
I didn’t want to try to recover it in place — with so many bad blocks, things can only get worse — so I copied the entire partition to a file on my laptop (the one with the shiny new disk):
ssh deadserver dd if=/dev/hda1 conv=noerror > hda1.img
(You need the
conv=noerror or else
dd will stop when it hits the first bad block.)
So, then I had most of a corrupt filesystem image. To make it useful I used the loop driver:
losetup /dev/loop0 hda1.img
Now I could try
reiserfsck to see what I could recover. I started with
reiserfsck --rebuild-sb /dev/loop0
to rebuild the superblock: even it if hadn’t been affected by the physical disk corruption, it would certainly be confused by it new home in a looped image that probably wasn’t the same size as the original partition. Next step was
reiserfsck --rebuild-tree /dev/loop0
to try to find the contents of the missing directories. I finished it off with
reiserfsck --check /dev/loop0
to make sure it was happy.
Now I can just
mount /dev/loop0 /mnt
and have a look in
/mnt/lost+found. The data is there!