Entries from September 2003 ↓

Cleaning up after disk crashing season hasn’t been fun, but I am pleased with what I’ve managed to recover from the worst crash.

I wanted to get the latest data from the dead webserver. It was in MySQL, and stored in /var/lib/mysql. Unfortunately, the /var/lib directory no longer existed.

I didn’t want to try to recover it in place — with so many bad blocks, things can only get worse — so I copied the entire partition to a file on my laptop (the one with the shiny new disk):

ssh deadserver dd if=/dev/hda1 conv=noerror > hda1.img

(You need the conv=noerror or else dd will stop when it hits the first bad block.)

So, then I had most of a corrupt filesystem image. To make it useful I used the loop driver:

losetup /dev/loop0 hda1.img

Now I could try reiserfsck to see what I could recover. I started with

reiserfsck –rebuild-sb /dev/loop0

to rebuild the superblock: even it if hadn’t been affected by the physical disk corruption, it would certainly be confused by it new home in a looped image that probably wasn’t the same size as the original partition. Next step was

reiserfsck –rebuild-tree /dev/loop0

to try to find the contents of the missing directories. I finished it off with

reiserfsck –check /dev/loop0

to make sure it was happy.

Now I can just

mount /dev/loop0 /mnt

and have a look in /mnt/lost+found. The data is there!

It must be something in the air, or maybe it is the heat.

Until my laptop disk died a few weeks ago, I hadn’t seen a major disk failure.

SWM had a similar problem a few days ago. And this morning, after an unexpected power shutdown at a datacentre in the middle of the night, I observed a catastrophic disk failure on a webserver.

I hope it doesn’t continue.

Whitehead reached its ADSL trigger level yesterday, with 250 registrations. We now have to wait for BT to do have a look and enable the exchange. BT can take a long time to do this, so I was pleased when I read that BT in Northen Ireland are faster, so I should be wired before Christmas.

So, I’ve started selecting my broadband ISP by playing with the service provider comparison service at http://www.adslguide.org.uk/.