My server is still missing!

I tried to contact Parcelforce yesterday morning, phoning every 20 minutes for over 2 hours. Each time I was told, by a recorded message, to call again later. I also tried email, to no effect. Their “phone me” link on their website seemed a bit more promising: I received a call less than 5 seconds after submitting the form. But this call was from another machine that kept me on hold for 22 minutes before letting me talk to a person.

The person (Dave) took my details and then gave me the same details I had already seen on the tracking website. He then told me he would contact the “search team” who would call me within the next two hours.

Six hours later, Steve from the search team called to say he would contact the local depot, but he wouldn’t get a response until Wednesday.

Steve called back today. The local depot had told him that they had attempted delivery on Saturday. I told him that they were either lying or had attempted to deliver to the wrong address: Redbus have 24-hour security who would have been there to accept the parcel.

My server is still missing!

The nice people at Black Cat had planned to install one of our new servers into their racks tonight, but the Parcelforce “guaranteed next day” delivery didn’t happen. That would be bad enough, but we didn’t find out until we arrived at Redbus.

I’m sure Simon from Black Cat had much better things to do, and I would have prefered to spend my £33 on something much tastier than a train ticket.

When I checked the parcel tracking details on the Parcelforce website, it showed that my server had arrived in London and had been sent out for delivery on Saturday; then it just disappeared.

The trip wasn’t a total disappointment: I remembered that London.pm had been invited to meet with London Java, so I went along hoping to either have a good chat with a Perl Monger or spend some time taunting the primitive Java users.

When I arrived I was surprised at how easy it was to categorise the programmers just by appearance. I was also happily surprised that the Java programmers were much more interesting than their language’s intended users. By the time I left, I had mostly forgotten about my server disappointment.

It seems there was a large power cut last night, sometime before 18:00, affecting a wide area of Northern Ireland. As I’m in England I probably wouldn’t have noticed; but the internet service provider in Belfast who hosts my servers went out with the electricity and killed my machines.

I heard that their UPSes worked to keep things alive for a while, and I also heard that their generator started quickly, but it seems that their distribution panel couldn’t take the strain: we can forgive them for that, as long as it doesn’t happen again.

My main complaint is that I didn’t hear any of these things from my them. We discovered the problem when one of our customers phoned us to complain.

So, I tried to contact our provider, but their phones were dead too.

Their network returned a while later. I could connect to two of my unhappy servers, but my third was still dead.

Their phones rang, but nobody answered.

Then I lost my servers again. This time it seemed to be a routing loop, that twisted all through the night until finally, more than 15 hours after the incident, someone answered the phone. The routing problem was quickly solved, and my third dead server was revived.

They still haven’t told me anything about what happened.

A service provider needs to maintain their customers’ confidence, especially when things go wrong. They should be able to identify routing problems in their network before I do. They should make sure everything is working after they think they’ve fixed it. And if they can’t do that, they should at least communicate: they should phone me to explain the problem, and not make it sound like it’s my fault for not having their mobile phone numbers.

This is the third time (that we know of) that we’ve had a service outage. The first time was when they unplugged our servers because they “didn’t know they were there”. The second time…we don’t know; we noticed one morning that our servers had been disconnected for a while; they were reconnected before we complained, so we waited to hear what the problem was, and we’re still waiting.

I haven’t had a reliable net connection while staying in Swindon. (Yes, that’s my excuse.) My landlord has a cable modem and a Netgear MR814v2 wireless router installed in one of the neighbouring houses, and I’ve been trying to use it. The signal strength has always been good enough, but the connection has always been really bad: on average, packet loss was over 40%; the connection would also frequently disappear for days at a time. This was happening on the LAN, before reaching the cable modem, so it had to be wireless router.

When the connection was lost I used kismet to try to try to find it. The router was still there, but transmitting on zero channel with a zero rate! (Using kismet I also found another one of my landlord’s routers, across the street.) I used tcpdump to watch the LAN while connected. The router was continuously arping for all the possible machines in its subnet. The router across the street wasn’t. A further comparison revealed that the dodgy router had a lower firmware version, so I figured that a firmware upgrade might solve the problem.

I downloaded the new firmware from Netgear, followed their instructions, and nothing happened. So I tried again, 6 times. That was yesterday.

Today I decided to try once more. The upgrade is performed by using a web browser to upload the firmware via the router’s web interface. I had tried this using the Mozilla Firebird browser, and it had failed. So this time I decided to try it with curl: curl -F mtenFWUpload=@mr814v2_5021.img -F Upgrade=” Upload ” http://192.168.0.1/upgrade.cgi

It worked first time! And it seems to have solved all the problems. But the whole experience has made me dislike Netgear products, despite their shiny appearance.

I’ve been trying to work out why the trains in England are so bad. The trains at home aren’t good either, but they also aren’t really that important. In England they are an essential part of the economic infrastructure, and they are frightenly bad.

The first problem I noticed was the price. I’ve been working in Swindon for the last few weeks (without Net access, but that’s another story), and I’ll be here until Christmas. So, I thought, Swindon is not too far from London; I’ll go and visit the London Perl Mongers. But the return ticket was going to cost 60 quid! At first I thought this was a mistake on my part: maybe I had selected a season ticket. No, that was a standard return. It’s cheaper to hire a car for a day.

I did discover later that there are /saver/ return tickets available if you book in advance, so I might do that.

The second problem I noticed was the timetable: it isn’t related to the actual times of the trains. I suspect that the rail companies have been brainwashing their passengers, as some people seemed to think I was weird for expecting the timetable to be relevant.

Then there are the displays on the platforms. At 09:40 the display showed: “current time is 09:40; the 09:10 train is expected at 09:36”. It might as well have said “we have no idea what we’re doing”. Surely a better system would at make sure that the expected arrival time was in the future.

I’m glad I only need to use the trains twice a week.