The Self-Organizing Nature of Open Source Projects

Thursday, December 18th, 2008

I’ve just finished reading an interesting paper called “Latent Social Structure in Open Source Projects“[pdf].

The authors looked at open source projects to discover if the project members self-organize and how successful the self-organization is. They also tried to determine if the ways that open source projects self-organize could provide useful lessons to aid in the building of commercial software teams.  A nice change from open source projects trying to learn from more traditional software projects.

I was particularly interested as one of the projects they looked at was Perl; the other four were Apache HTTPD, Arache ANT, PostgresSQL, and Python.

The authors detected community structure by data-mining the mailing lists used by the projects’ developers.  They are aware that developers use other methods of communication, such as private email and irc channels, but they considered mailing lists to be a good place to start.

There are a some things in the paper that I am not sure about.  They describe Perl and Python as being examples of projects that are monoarchist with a project leader.  I can’t comment on the Python project but is Larry Wall really “at the helm making informed important decisions”?  The paper contains a chart showing the development community structure in Perl from April to June 2007.  They have taken out the managers from this chart so I can’t tell if Larry was involved in any of the development being shown.  (Larry would fall under their definition of a manager as a person with intimate knowledge of large parts of the project who would link various sub-communities together.)  I am aware that Larry is still involved in many aspects of Perl development but I do feel that the project is much too big to think of any one person being at its helm.

I did find the chart showing Perl development fascinating though I have no idea what aspect of Perl development the sub-communities they show are involved in.  There is a sub-community of Paul Marquis and Xiao Liang Liu.  Another with Jonathan Stowe and Pelle Svensson.  One of the communities showing active development contains Arthur Bergman, Leon Brocard, H. Merijn Brand and Jarkko Hietaniemi.

They also gave some information on the data gathered to work out the sub-communities.  They looked at Perl mailing lists from 1st March 1999 to the 20th June 2007 (it doesn’t state which list or lists.)  They counted 112,514 messages with 3,261 participants.  They also extracted information from code taking the author, time of commit and the filename.  They then matched the author to the email addresses.  For this period of Perl they say that there were 92,502 commits.  But the figure that really shocked me was that for all these commits there were only 25 developers.  So we have a mailing list with 3,621 people on it but only 25 people actually making any changes that were agreed!

I assume that they were looking at the core Perl language and I am aware that Perl modules are being developed by thousands of developers but I’m still fascinated by the concept that so few people are committers on what appears to be a vast project. I would love to know more about their data.  Which list did they use, how many messages are there per month, is this number declining over time?  Maybe I’ll write and ask.

A lot of the things in the paper didn’t surprise me but one thing intrigues me. Do developers work better if they get to choose who to work with? – November Technical Meeting

Thursday, November 27th, 2008

I am always amazed at the number of speakers they manage to find for a technical meeting.  Granted they don’t have these very often but 17 speakers and 18 talks is still a lot for one meeting. This does mean that many of the talks are very short but this is something I really like.  It’s easy to listen to a speaker for a short time – no matter what they decide to talk about.  It’s one of the things that made lightning talks so popular at the YAPC conferences.

There are about 100 people here and the meeting also has remote venues (there is a screen at the front of the room showing a video feed of the people at the other three locations.)

The meeting, like every other Perl event I’ve attended in Japan, is very well organised.  The meeting room is large enough for the attendees, there is wireless, and everyone has table space for their computers.  Sometimes I don’t like wireless at meetings as the attendees appear to be spending more time playing with computers than listening the speaker but here it’s evident that they are paying attention to the speakers.  The people around me, who are using computers, are writing up the meeting (like I am) or commenting on it on various social networking sites.

The meeting is running very smoothly.  With so many speakers it would be easy to have problems with the projector but so far so good.  Marcel described this meeting as a conference and although that’s not quite the case there is no doubt that a lot of time has gone into organising this event.

YAPC::Europe – Lightning Talks On The First Day?

Wednesday, August 13th, 2008

When I first saw the schedule I wasn’t sure that lightning talks would work well on the first day.  A lot of them rely on humour and I had thought that they would work better later on in the conference when the attendees are more relaxed and have more of an idea what to expect from the conference speakers.

But they are about to start so I will soon find out if this works or not.

YAPC::Europe – Lunch Time

Wednesday, August 13th, 2008

I have noticed that YAPCs are starting to provide more food for the attendees.  I am surprised by how much they are able to do considering the low cost of attending the conference.  The lunch bag contained a bottle of water, salad, two rolls and a small yogurt based dessert.  It was very nice and it was good to not have to go outside to try to find somewhere to eat around the venue.

YAPC::Europe – Opening Day 1

Wednesday, August 13th, 2008

At the start of every conference I think it would be a good idea to blog more.  What usually happens is that I end up with lots of unfinished posts in my draft folder.  I still haven’t finished writing up YAPC::Asia and I think that took place in May.  I have an idea that I might use my YAPC::Asia material and write a comparison between the three YAPCs that I have attended this year – but who knows if I will actually get this finished.

YAPC::Europe started this morning in Copenhagen.  I didn’t have any problems getting to the venue as I travelled with JJ and Barbie and Barbie had taken the time to work out how to get there.  I am really glad that I didn’t need to find it by myself as it wasn’t that clear which building the conference was taking place in.

I had already registered last night at the pre-conference event so didn’t need to do that this morning.  Not that it would have been a problem as the organisers looked very well prepared and the opening of the conference has been very smooth.  The wifi has been much easier to connect to than the one at YAPC::Asia or YAPC::NA.  In Asia there was quite a complex registration process to go through.  In America the login details were not obviously available and I had to ask Yaakov what I was supposed to do to connect.

The conference is about to begin as Jonas is calling for the first speaker.  So far so good. Social Meeting

Friday, August 8th, 2008

Last night I managed to drop in to the 10th Anniversary social meeting.  When I planned my trip to YAPC::Europe I hadn’t realised that I would be arriving in London on the same day as that meeting.

It worked out surprisingly well as I was concerned that I would be much too tired to go and see anyone.  But the flight was good and I had arranged to meet up with Dean before going to the meeting as I did think I would be too tired to cope with a whole crowd of people for too long.  It took me longer to find Dean than he expected.  He had probably expected me to have more of an idea of where exactly I was in London.  He also didn’t know that asking me things like “are you on the south side or the north side of the river” may as well have been in a foreign language as I am never going to know the answer to that.  He was lucky I had even noticed that there was a river at all!

I was pleased that I got to catch up with some people who won’t be going to YAPC::Europe this year.

Miyagawa Wins White Camel Award

Wednesday, July 23rd, 2008

I’ve just seen Miyagawa announce on Twitter that he has received a White Camel award at this year’s OSCON.  I have not seen the official announcement yet but I imagine that he has been awarded this for his work in being a key organiser of YAPC::Asia and for his work with  Well done!

YAPC::NA – Morning 1

Monday, June 16th, 2008

Having to speak at YAPC::NA first thing in the morning meant that I was completely stressed out for the start of the conference. I don’t remember much about the registration process but it must have been really straight forward or I would remember. (Actually I’ve just remembered something about being given a glass which I no longer seem to have…).

I really like the venue so far – though I have only seen the main auditorium. It’s an easy room to speak in and the mikes and projectors work really well.

The only good thing about giving the first talk is that I can now enjoy the conference. It did mean that I didn’t attend the pre-conference dinner last night because I didn’t think I would be able to cope with meeting people. The people I did meet up with for breakfast and before the conference were all great and did help to calm me down.

YAPC::Asia – Registration

Saturday, May 31st, 2008

The registration process at YAPC::Asia made use of QR codes that Japanese mobile phones can scan and read. The attendees were supposed to print their unique QR code from the website and bring it with them in place of tickets. Last year tickets for the conference could be bought at a Lawson convenience store. I have never needed a ticket for the European or American YAPCs. I usually give my name to the person at the registration desk and they check this off a list or find a badge that contains this name. But YAPC::Asia makes good use of technologies that are common in Japan. I have bought tickets at Lawson for everything from flights to concerts. And QR codes are found even on the wrappers that McDonalds use for their burgers.

Although the registration process was very well organised I do think it should be possible to improve it as I thought it was a bit slow. Not because of the QR codes but because the codes related to a name badge that was filed by number in a box. You would get to the desk and someone would scan the code and then find your badge. This was done in a small area limiting the number of people who could help with registration. And even if the area had been bigger only so many people can look through a couple of boxes of badges at any one time.

One way to speed up the registration process would be to limit that number of things that need to be given to a specific person. If attendees printed their own badges with QR codes on them then these could have been scanned but no badge search would have been required. The conference bags were generic so handing these out was easy. The badges, however, were not the only things tied to a specific attendee. Each attendee was given details of how to access the wifi system and because of security reasons the wifi access accounts and passwords were connected to the userid of the person registering. They also were the property of the university and this information could not have been given to attendees via the web-site.

There was enough space, however, to allow more people to help with registration. And there were certainly enough people on the conference crew to do this. It should be possible to find a way to split up registration. For example speaker registration could have been done separately. Sometimes registration of a large number of people is set up alphabetically. This would have been hard to do though because the attendees queuing at the door wouldn’t have been able to see these signs and it could have caused quite a bit of confusion in the entrance hall – to say nothing about the fact that the attendees names wouldn’t all use the same alphabet…

As well as registration on the first morning a pre-conference registration took place the night before. I suppose I’m just worried about what would have happened if 400 people had all turned up 15 minutes before the conference started. Mind you, there is probably no fast way to deal with that.

YAPC::Asia – The Venue

Saturday, May 31st, 2008

One of the most important things about running a successful conference is the suitability of the venue. This year YAPC::Asia was held at the Tokyo Institute of Technology. I believe that this was the best venue that has been used for a Japanese YAPC::Asia – and not just because it’s within walking distance of my apartment!

The venue was easy to get to as it was beside a train station. I always prefer conference venues that can be reached by public transport. I don’t drive and I have had to use taxis in America to get to conferences that were held in hotels that weren’t close to anything. Last year at YAPC::NA I didn’t get to see any of Houston as the venue was outside the city and it wasn’t obvious how to get to anywhere without calling for a taxi. I understand that at the American YAPCs it’s important to have access to University Halls for cheap accommodation and that this restricts where a conference can be held. This doesn’t seem to be an option in Tokyo nor does it seem to be a requirement. As Tokyo has such an amazing public transport system YAPC attendees can stay in most parts of the city and given that so many of them come from Tokyo they probably just go home in the evenings.

Registration took place outside the main auditorium. This area was big enough to allow registration without blocking entry to the room. The auditorium comfortably held the full conference. It was a tiered room which I always like as I’m small and find it hard to see over hundreds of people in a room that isn’t tiered. Each chair had a small table though most people weren’t using these. I used mine as I needed somewhere better than my lap to balance my laptop when I was typing up the talks. The only problem that I can think of was that the seats were quite close together and I don’t know how comfortable they would have been if you had been quite large.

The other rooms were smaller but seemed adequate for the number of people going to them. The DeNA room did appear very packed at one point but not uncomfortably so. Additional chairs were brought in and more space could have been made by laying the room out without tables.

The venue was close to a variety of restaurants and coffee shops providing options if you didn’t want to eat the lunch provided. The beautiful weather probably also helped. The rooms for the talks were in three separate buildings and I don’t know what it would have been like to move between these in torrential rain – though umbrellas were provided in the conference bag. But as the weather was beautiful it was possible to sit under the trees and eat lunch in the sun. And the walk between buildings was really pleasant.

The conference dinner was also held at the venue. This did cause a problem as it couldn’t hold all the attendees. I’m not sure, however, if there is anywhere suitable in Tokyo to host a dinner for 500 people. In the end 300 people got to attend based on a variety of criteria such as how far you had travelled to get to the conference. I didn’t hear anyone complain about this but then given my limited Japanese this isn’t really surprising.

I suppose I should say something about the wifi. This was difficult to use and even though I have a Macbook I wasn’t able to set this up myself. Some of this was because the instructions were in Japanese and some of it was because of the security required by the university. It also didn’t appear to support the number of people trying to use it in the main auditorium. When I finally got connected on the first morning, after both Marty and Emerson had played around with my computer, I wasn’t able to stay connected for long. I needed to be connected as I wanted to help with transcribing and eventually the problem was fixed by asking people who didn’t need to be connected to log out. I didn’t have any problems using this on the second day of the conference.