First of all, I should be clear that the Agile200x submission system is the best I have used, so the following comments are picking hairs. I’ll also confess to having a bias becuase I had nothing accepted this year!

1. I thought the comment/review distinction was unclear. As a stage reviewer, I was unsure which to use sometimes. It took me a while before I figured out that reviews were visible to fewer people. So I’d prefer to have a single review type. The Stage committees can use all reviews to make their decisions.
2. There was some talk that there was less gaming with this years system. I’m not convinced. Its possible any gaming was just less transparent due to the hidden reviews.
3. I wonder whether early submitters were penalised – their sessions ended up being at the bottom of the list. How can we encourage and reward early submissions? These should be the ones that get most feedback and get the most iteration.
4. Why was the deadline extended when there were already plenty of sessions? Again, this doesn’t encourage early submissions with lots of iteration. I’d like to see a fixed deadline for submissions, followed by a further fixed period for reviewing and iterating, before stage teams make their decisions.
5. Should there be a higher bar for submissions? A minimum number of words? This would encourage people to submit well thought through ideas, and discourage people from submitting dozens of bare ideas at the last minute.
6. It didn’t always seem clear which stage to submit to, and when people submitted to an inappropriate stage, then the process seemed a bit random. As a reviewer, I tended only to look at what had been submitted to the Frontier Stage, and quite likely missed Frontier Friendly topics which had been put on other stages, but didn’t get accepted. Could reviewers suggest alternate stages? Then as a reviewer I could filter for sessions suggested for my stage? Or as a submitter I could have the option not to specify a stage in the hope that it gets picked up by a stage reviewer?
7. Relate to point 6 – while each stage produced a balanced program, I’m not sure the whole program was balanced. Could there have been more cross stage collaboration to help submissions find the right home?
8. Having to log in to view submissions might have got in the way? I’d like to be able to view all submissions without having to log in, and only log when/if I want to review something.

Agile2009 was still a great conference. These points are with the hope of making Agile2010 even better still!

Karl

In addition my long term hope is that we don’t need a Frontier stage at all. I would hope that edgy sessions far from the mainstream are found on every stage. That certainly happened on the Manifesting stage this year where we embraced Neuroscience (Linda and myself), Cognition, Psychology and Agile outside of software.

I have one: more content about System Thinking (yes, I know that there was one presentation on that at the 2009 conference).
I know that System Thinking is old stuff. But the fact is that I got interesting in Agile after it was hot. So I rarely get to see presentations on it, and I do not get enough pressure from peers to bother reading books on it. So, a presentation would have been valuable to me.

Possibly contradictory with my call for advanced sessions. No matter.

Several commenters have mentioned that they have no personal experience in managing an open review process, or that Agile 2009 was actually the most open process they’ve personally seen. There have been expressions of fear that people might “game” an open review process. There have been suggestions to resort to illusion-of-control methods like a double-blind review process to try and prevent any possibility of gaming the system. Lions and tigers and bears, oh my!

Those comments indicate that this group of people simply sees no practical way to apply agile thinking to the problem of organizing a large-scale conference. The only way they can see to do it is to revert to non-agile thinking. Okay. It is what it is. The logical conclusion, then, is that we need a different group of people.

Maybe we should ask the organizers of the aforementioned events to take the lead in organizing Agile 2010. Their events haven’t been as large as Agile 20xx before, but the mechanisms they have used can be adapted to a larger event if people really want it to work. Some of the individuals involved in those events also have experience organizing past Agile 20xx events. Among them, there’s ample expertise in both agile thinking and large event planning.

I believe it is possible to organize a large event without abandoning agile thinking. The reason I believe that isn’t that I’ve done it before personally. The reason is that I have confidence that an agile approach offers a practical way to solve problems. Besides, we’re only talking about the review process. Other aspects of event planning remain the same.

Carlton asks, Why reinvent the wheel? The answer is that the agile community is in the business of reinventing the wheel. The whole movement is about changing the way people think about and act upon problems and challenges. If we’re the first to manage a large-scale event in an agile way, that’s a Good Thing; it’s a reason to be excited and enthused about the challenge, not a reason for fear and doubt. If we’re unwilling to live by the values we promote, then we’ve got nothing to talk about at a conference.

Another possibility is something I saw in a solicitation from another conference – they said that papers “will be subject to a double-blind review process”. I’m not sure just what that entails, but I assume it means the submitter doesn’t know who’s reviewing and reviewers don’t know the submission authors.

If we had an anonymous system we wouldn’t need a rule saying you can’t submit proposals and be a reviewer. Could that work?

As for a an open review system, I don’t know. I would want some more information about it and I would want to know how other big, long running conferences select submissions. Other people have run into this problem before and have come up with solutions. We reinvnent the wheel?

I like your suggestion about making feedback available to reviewers. In fact, in a open system all feedback would be public.

It seems some in the agile community fear openness, transparency, and collaboration, and prefer to try and prevent occasional abuses of a system by discarding the fundamental agile value, trust. Interesting. Disappointing, too. Why should others believe in our values if we don’t believe in them ourselves?

In addition I would like to see the previous year’s speakers feedback made available to the reviewers. It makes it easier to accept a session from someone you don’t know if you can see their feedback.

Carleton – cronyism is a risk but here is the counter. My personal interests and research at the moment are in neuroscience, sociology, et al. This made both a naturally reviewer and presenter on the Manifest stage. So how do you solve the problem that the best qualified reviewers are also well qualified to present on their own stage?

BTW I’m happy to send you copies of my feedback so that you see what people thought of my session.