I was playing with Jim Breen’s Japanese dictionary and I wanted to make a list of the first kanji component in each entry. I wanted one result for each entry, so I used “(none)” if the entry has no kanji part. This is not a difficult problem, although XML makes it as slow and memory intensive as many difficult problems.

use XML::Twig;
my @keb = (); # for the results

sub entry {
    my ($t, $e) = @_;
    my $kt = "(none)";
    if (my $k = $e->first_child("k_ele")) {
        if(my $keb = $k->first_child("keb")) {
            $kt = $keb->text();
        }
    }
    $e->purge;
    push @keb, $kt;
}

my $twig = XML::Twig->new(
    twig_handlers => { entry => \&entry }
);
$twig->parsefile($ARGV[0]);
$twig->purge;

# now the results are in @keb

Using XML::Twig is quite simple. When I create the parser I tell it how to handle the elements I care about, and in this case I only care about “entry” elements. When the parser finds an entry, it calls my entry subroutine, passing the entry’s object as the second parameter, $e. Inside the entry routine I can use DOM-style methods on $e to extract the data I want. Notice that I call $e->purge when I’ve got the data out. This tells the parser that I won’t need that element again, so it can free the memory. This is how XML::Twig manages to parse a file that most other modules can’t.

For Perl, as I had thought, the XML::Twig module gave me a pleasant interface and was able to easily handle the document.

For Haskell it was a little bit trickier. I used the SAX parser in HaXml, but it is not like a regular SAX parser, since Haskell is so unlike any regular language. The parser returns a lazy list of SAX events, so I had to make sure I processed the list without evaluating the whole thing into memory.

Now that I’ve dealt with the memory issue it appears that I have a speed issue to deal with next.

My first attempt was a simple Haskell program, but I had to kill it after it ate over 1.3 GB (yes, 1.3 GB) of ram!

Haskell’s strings are known to be memory hogs, and the HaXml module I was using was making them even worse by not sensible decoding the UTF-8 text correctly. I decided to write a leaner Haskell program later, and switch to Perl to get the job done.

At this point I also decided to set a limit to the amount of memory the programs could consume. For a 39 MB file I hoped that 10 times that would be enough, so I rounded up and set the limit at 512 MB.

But Perl, using the XML::LibXML module, couldn’t process the file with that memory limit. I also ran a quick one-liner in Erlang, just to watch it crash out of memory too. I’m going to try some other languages to see if I can find one that can work in 512 MB.

My next useful step is to try the XML::Twig module in Perl. I’ve had good experiences with it before. It won’t be as fast as LibXML, but it probably has the best chance of surviving within my 512 MB limit. For Haskell, I think I’ll have to resort to a SAX style parser.

In one of these talks JRockway asked Shawn Moore how to turn a bad API into a good API, so I’m going to try that: adding antlers to Class::Accessor!

So now instead of writing:

package Foo;
use base qw(Class::Accessor);
Foo->mk_accessors(qw(alpha beta gamma));

If you prefer Moose-style you can write:

package Foo;
use Class::Accessor "antlers";
has alpha => ( is => "rw" );
has beta  => ( is => "rw" );
has gamma => ( is => "rw" );

The original API is still available, and everything is the same underneath.

I don’t post very often, but I’m going to try to change that. Is this my fifth attempt?

This time, to give myself a goal, I joined the Perl Ironman Challenge and I will try to blog at least once a week about Perl. So…

Perl: it’s alive!

There have been lots of reports over the last few years about Perl being dead. Those reports upset a lot of Perl mongers, and I didn’t fully understand that. Perl was not a family member, friend, or pet; so why the strong emotion? It was never really “alive”, so how did it “die”? And all these upset people were still using Perl, so they kept it breathing. And there were many more Perl users who weren’t upset, maybe because they never heard about the death.

It seems to me that Perl never died: it just became unfashionable for a while. And during the unfashionable period Perl did have some self-image issues, and maybe a lot of misdirected energy. But being unfashionable isn’t life-threatening.

$n=4;
undef $/;
@txt = split /\W+/, lc <>;
for($i = 0; @txt-$i > $n;  ++$i) {
    print "@txt[$i..$i+$n]\n";
}

(N-grams of 5 elements seem to be a good size for collocates, so we set $n=4.)

To look for interesting collocates we simply piped the output of that script through sort | uniq -c | sort -n | tail . As test data I ran the script against version 2 and 3 of the GPL. In version 2 the most common n-gram was “work based on the program”; but for version 3 it was “the gnu general public license”. That isn’t a particularly interesting result, but I’m sure we will find some when we look at more than 2 source documents.

On the way home we were comparing the Tokyo tech talks to ones we have seen in Europe. There are a lot of similarities, but we noticed one trend: in Europe the focus is on how you can do something, but here it is on what you can do.

We decided to do this talk together because we have different viewpoints. I’ve uploaded modules to CPAN so it makes sense to me. Karen was able to investigate the upload process from a first time perspective and so could spot the confusing parts. We hope that the combined perspective will make the talk useful.

Well, Karen wouldn’t have thought about the kanji characters, but she knew that “hashi” (箸) meant “chopsticks”, so she thought a “spork and chopsticks” name was a smart idea from Ingy.

I don’t think anyone else spotted that.  The “hashi” (橋) in Takahashi (高橋) means “bridge”; 高橋 is a surname that means ”high bridge”.