I was playing with Jim Breen’s Japanese dictionary and I wanted to make a list of the first kanji component in each entry. I wanted one result for each entry, so I used “(none)” if the entry has no kanji part. This is not a difficult problem, although XML makes it as slow and memory intensive as many difficult problems.

use XML::Twig;
my @keb = (); # for the results

sub entry {
    my ($t, $e) = @_;
    my $kt = "(none)";
    if (my $k = $e->first_child("k_ele")) {
        if(my $keb = $k->first_child("keb")) {
            $kt = $keb->text();
        }
    }
    $e->purge;
    push @keb, $kt;
}

my $twig = XML::Twig->new(
    twig_handlers => { entry => \&entry }
);
$twig->parsefile($ARGV[0]);
$twig->purge;

# now the results are in @keb

Using XML::Twig is quite simple. When I create the parser I tell it how to handle the elements I care about, and in this case I only care about “entry” elements. When the parser finds an entry, it calls my entry subroutine, passing the entry’s object as the second parameter, $e. Inside the entry routine I can use DOM-style methods on $e to extract the data I want. Notice that I call $e->purge when I’ve got the data out. This tells the parser that I won’t need that element again, so it can free the memory. This is how XML::Twig manages to parse a file that most other modules can’t.

For Perl, as I had thought, the XML::Twig module gave me a pleasant interface and was able to easily handle the document.

For Haskell it was a little bit trickier. I used the SAX parser in HaXml, but it is not like a regular SAX parser, since Haskell is so unlike any regular language. The parser returns a lazy list of SAX events, so I had to make sure I processed the list without evaluating the whole thing into memory.

Now that I’ve dealt with the memory issue it appears that I have a speed issue to deal with next.

My first attempt was a simple Haskell program, but I had to kill it after it ate over 1.3 GB (yes, 1.3 GB) of ram!

Haskell’s strings are known to be memory hogs, and the HaXml module I was using was making them even worse by not sensible decoding the UTF-8 text correctly. I decided to write a leaner Haskell program later, and switch to Perl to get the job done.

At this point I also decided to set a limit to the amount of memory the programs could consume. For a 39 MB file I hoped that 10 times that would be enough, so I rounded up and set the limit at 512 MB.

But Perl, using the XML::LibXML module, couldn’t process the file with that memory limit. I also ran a quick one-liner in Erlang, just to watch it crash out of memory too. I’m going to try some other languages to see if I can find one that can work in 512 MB.

My next useful step is to try the XML::Twig module in Perl. I’ve had good experiences with it before. It won’t be as fast as LibXML, but it probably has the best chance of surviving within my 512 MB limit. For Haskell, I think I’ll have to resort to a SAX style parser.

In one of these talks JRockway asked Shawn Moore how to turn a bad API into a good API, so I’m going to try that: adding antlers to Class::Accessor!

So now instead of writing:

package Foo;
use base qw(Class::Accessor);
Foo->mk_accessors(qw(alpha beta gamma));

If you prefer Moose-style you can write:

package Foo;
use Class::Accessor "antlers";
has alpha => ( is => "rw" );
has beta  => ( is => "rw" );
has gamma => ( is => "rw" );

The original API is still available, and everything is the same underneath.

I don’t post very often, but I’m going to try to change that. Is this my fifth attempt?

This time, to give myself a goal, I joined the Perl Ironman Challenge and I will try to blog at least once a week about Perl. So…

Perl: it’s alive!

There have been lots of reports over the last few years about Perl being dead. Those reports upset a lot of Perl mongers, and I didn’t fully understand that. Perl was not a family member, friend, or pet; so why the strong emotion? It was never really “alive”, so how did it “die”? And all these upset people were still using Perl, so they kept it breathing. And there were many more Perl users who weren’t upset, maybe because they never heard about the death.

It seems to me that Perl never died: it just became unfashionable for a while. And during the unfashionable period Perl did have some self-image issues, and maybe a lot of misdirected energy. But being unfashionable isn’t life-threatening.

Google said: “You are our base, all you CATS”. I contributed a better translation.

I thought the original phrase would be entertaining because it contains a Japanese verb that Google mistranslates brilliantly: いただく。

「いただく」 means “receive” or “accept”. But it’s a humble verb, so the receiver has a lower status than the giver and so receives with gratitude. I suspect that Oliver Twist would use a form of this verb when asking for more gruel.

If you were the receiver and used this verb you would most likely use it in the form of 「いただきます」(the same verb with the normal polite ending). And this is where Google translate gets really confused. Because, unless you frequently get gifts from important people or work in a shop, the most common situation where you would use this word is at the start of a meal: you’ve just been given some food, and you’re thankful, so you say “I (gratefully) accept (this food)”. It’s slightly idiomatic, although the meaning is clear. It’s a bit like “saying grace”. But a popular idiomatic expression used in that situation by many English speakers is (the French phrase) “Bon appétit”. Therefore Google always (as far as I can tell) translates 「いただきます」 as “Bon appétit”.

That really doesn’t work well if the main verb in your sentence is “receive”. One example, appropriate at this time of day, would be 「お休みをいただきます」, for which I would say “thankfully I’ll get some rest”. Google says “Rest for a bon appétit”.

My experience tonight supports that claim. When I first called them a guy answered, but I didn’t hang up. I asked for flights on dates that I knew had available seats for points (since I had checked on the ANA site; another bit of advice from Tony). But, with barely a pause to enter the information, the guy said there was absolutely no availability on those dates. He didn’t suggest any other options. So I hung up.

I immediately hit redial, and this time spoke to a girl who happily helped me get the flights I wanted.

I’m a white male, so the naïve observer may think “not black, not female, therefore not minority” (a fallacy, of course). Many of these observers claim to oppose racism and sexism, but still use my skin colour and gender to classify me. They justify that discrimination by saying that it has no negative impact on me because I’m part of the majority group. Are they really saying “you white guys all look the same”? Or worse, “all you white guys are the same”?

I’m actually a member of one minority group that is 0.00026 of the world’s population. I look like a short hairy version of the standard white guy, and I speak a dialect of the standard language. But the dialect reflects a cultural difference in thinking. And since “technology” as an industry is founded on thinking (although sometimes it doesn’t seem so) it is the difference in thinking that should be considered.

The biggest problem was his tag cloud generator from zoomclouds.com. It appears that their service is currently dead (it doesn’t ack my syn), so I set my local packet filter to reject connection attempt to their site, and that made it much better.

But since Alan’s blog takes 3 minutes and 20 seconds to finish loading, you’ll have to be patient.

Before that my favourite footwear was my Teva sandals, but they had some limitations: most importantly, I could not wear them to work as the dress code is a bit more formal than my usual “shorts ‘n’ sandals”. They also had the problems mentioned in the article.

I was keen to try two of the barefoot-style shoes mentioned: the Vibram Five Fingers and the Vivo Barefoot; so I did.

The Five Fingers were interesting. Some friends had already tried and spoke highly of them. Having now tried them quite a lot myself, I have mixed feelings. When I first put them on I thought they were great, although my toes felt slightly odd being each enclosed in their own little toe-sock. But on that first hot summer day that was fine. Unfortunately by the end of that day I had developed blisters on both feet. I was certain this was caused by a defect in the shoe since I could have managed the same travel completely barefoot without any problems. Schwern said the same thing happened to him: it was caused by lumps in the seam between the sole and the upper; and he fixed it by adding some moleskin. Simple solution, but why should I have to fix my new shoes?! Vibram, please keep the seam smooth, or move it to the side of the shoe where it can’t cause these sort of blisters.

A few months later I discovered another problem with the Five Fingers: they don’t dry well. They are good when permanently wet and great in the sun, but if you get them wet just once in the morning of a non-sunny day, they’ll still be wet the next morning. Compared to my Tevas this was a downgrade. The Tevas, not having any upper, would dry out in a few hours of use, and allow my feet to dry too. The Five Fingers stayed wet and kept my feet wet. Then both feet and shoes started to stink. At least the five fingers were easy to wash.

The Five Fingers are also not acceptable office wear (except at weekends), but they are a conversation starter. I think they’re great for outside the city, but not for urban use.

The Vivo Barefoot, however, are the greatest urban footwear I’ve ever seen. Quoting the article: The Vivos are a totally different experience, since they’re as close to going barefoot in the city as you can get. I bought the “Dharma” version. The sole is thin enough for me to feel the texture of the ground, but tough enough to protect my feet. Although enclosed, my feet feel free. The Vivo Barefeet do take a while to dry out if they get wet, but they don’t get as wet as easily as the Five Fingers. But for me the best feature is that nobody in the office notices that they aren’t normal shoes!