Looking back at Agile 2009
Another conference has come and gone. I’m home. I’m exhausted. I’m glad to have good internet connectivity again, and time to sit alone with my thoughts.
I had a fabulous time. Through conferences such as this one, I now have dozens of friends from around the world that I rarely get to see. I really enjoy getting together face to face. Missa vitussa se poro on. (I hope I’ve spelled that correctly.) And I spent time with friends that I previously only knew from Twitter and email. And I met new friends that I’d not known before. All of that was thrilling. (I also didn’t get a chance to talk with some friends whom I know attended. Life is like that.)
I attended some great sessions. I attended some sessions that were less great, but very interesting to me. I attended some sessions that were great, but less interesting to me. I didn’t attend any that were terrible. I suppose there might have been some bad ones. I heard rumors of such, but I’m not sure whether that’s an indication of the session or the expectations of the person reporting it. I also didn’t attend sessions. I didn’t attend some sessions because
- there was another session I attended instead,
- the session was full,
- I thought that I’d done something similar in the past,
- it was a “talking head” presentation (which I have a prejudice against),
- I wanted to give some energy to the Live Aid project to support Agile Philanthropy,
- I wanted to chill out in Open Jam and relax with casual conversation, or
- I forgot until it was too late.
I didn’t maximize my time. There is no optimization for life. Instead, whether doing something that seemed important or something that seemed trivial, I merely tried to be in the present time, in the present location. “Be here, now.” Or, as Lyssa Adkins said in Build Your Team’s Collaboration Muscle, “be in present time.” I left Slack for myself, rather than fill every moment with The Most Important Thing. I highly recommend this technique.
I could talk about specific sessions and events I found enjoyable or helpful. I could talk about things that I found annoying. Certainly not everything was perfect. It’s a human endeavor, after all. Instead, I’d like to talk about what can be done to make Agile 2010 even better.
Jim Newkirk, the Conference Chair for Agile 2010, invited me to the conference retrospective. This was mostly populated with stage producers and stage assistants. I had been merely a stage reviewer, and was initially hesitant. I was happy to find that I could add a little value to this retrospective, and I found that there was a topic about which I was passionate that was important to many.
There were many concerns about the session selection process for this year’s conference. I had concerns both from the vantage point of a reviewer on the inside of the process, and from the vantage point as a session author on the outside. Finding that others had similar concerns, and different concerns about the same process, I volunteered to collect some of those concerns, organize them and summarize them, and report them to Jim for use in planning Agile 2010. I also want to collect suggestions for improving the session selection process. Please send your concerns and suggestions to agile2010 -at- gdinwiddie.com or you may leave a comment on this blog posting.
Thanks for your help.
I miss small rooms where more focused topics can be discussed. The feedback I received on my proposal was it wasn’t felt it had enough draw for a stage. Several others left comments that my topic was of interest to them. IMO “stages” send the same sort of message I think you mean by “talking head” presentations.
As a newbie presenter, the opportunity to present was just invaluable and so it was really wonderful to see that the conference selection process allowed people like myself to present.
I have to believe things like this help us grow more passionate people in software development that are, in turn, interested in growing others… which is what we’re trying to do, right? Sort of act as a catalyst to greatness.
I did realize at the end that I’d totally missed all the cool stuff going on in Open Space though because I was so focused on the sessions. It might be interesting to try to promote those more as another way to get newbies involved in contributing.
Could you say a bit about how the reviewing was done? I heard each of the following suggested by attendees (one each) and do not know their validity/accuracy:
1) inundating with many proposals might give you a better chance;
2) inundation actually made a reviewer think twice about the value of presentations, i.e., which one (or two) did the proposer _really_ want to do;
3) seemed many producers and assistants managed to have their presentations picked. May not be true, but in absence of knowing selection “standards,” I can see people suspicious. (Don’t know about this. Didn’t count myself.)
I have also previously emailed program folks these observations:
1) Stages should not have competing sessions on their own topic.
2) Require earlier evidence of what speaker truly plans. (Some folks seem to put presentations together late in the game. Brian Marick, for one, clearly did not and had a succinct, clear impact as a result.)
3) Have a tutorial/workshop day for all sessions over 90 mins. Going to one of these means having to choose to miss 3-4 other talks.
And if it seems this all might reduce the number of talks selected, I say this argues for making Open Space/Jam a bigger part of the Conference like it used to be. Allow unselected sessions that reviewers still thought were good to be pre-announced as Open Space/Jam choices, for example.
Scott, each stage had its own review committee and set its own criteria. I was on the Distributed Agile review committee. We tried to ensure space for newcomers and limit overexposure to the well-known usual presenters. Even so, that’s hard to do, especially from the perspective of a single stage and with such a large number of proposals.
I’m leery of a day dedicated to long sessions, as that would limit the number of long workshops that one could attend. Often these are the most powerful.
Hi George,
Thanks for volunteering to do this. First of all, it was an awesome event. I think that is due in part to the high degree of interest in continual improvement (for instance your blog post).
Before getting into my thoughts on the Agile 2xxx process, let me just say that it is 5x better than any other process I’ve been part of. The system itself, the degree of transparency (though low), and the ability to iterate one’s proposal are second to none.
This was my second time attending and also my second time presenting. My experience last year was that the submission process itself was almost as good as attending the conference itself. My experience submitting this year was that that submitting a proposal was just submitting a proposal.
I don’t know why, but it seems like the 2008 process was vibrant: full of people and feedback pretty much all hours of the day and night. In contrast, it seemed like there was very little feedback on very few submissions from very few people for 2009.
One other large difference was the lack of the (public) rating system. Perhaps some people didn’t like seeing the red marks from non-official-reviewers, but I appreciated the feedback. My first submissions last year were terrible, but the ton of feedback I received inspired me to create completely new and different proposals.
Thanks for your efforts to make an even better system.
Cheers!
Damon
Hi! I was on one of the review committees, and also submitted to a couple of other stages, so I saw both sides of it. Or two sides, rather, as there are surely more.
I think the number one thing I’d like to see in the software is richer support for communication and interaction feedback loops. Unless you went and opened a bunch of pages repeatedly, you couldn’t see when people were saying things in response to your comments.
Another is some guidelines for what makes a good review. Some stages did a great job of giving rich feedback to proposers. Others tended toward one-line responses that I suspect others found as disappointing and unhelpful as I did.
It would also be great to shorten up the lead time. I’m not sure what the cause of the long lead times is, but I suspect it’s not necessary to lock down every single session so far in advance.
As a reviewer, I really wanted richer submitter profiles. I often had to ask for info on previous speaking experience, sample material, links to videos, or anything else that would help me evaluate a stranger. If they’ve presented before, I’d love easy access to previous evaluations.
Is it worth thinking about splitting the conference into two or three slices? One would be the large session done by well-known people for novices. That could be planned early, so there’s plenty to announce. Another would be a big chunk of open space, organized dynamically at the time of the conference. And perhaps in the middle are the small-batch talks, which we could plan closer to the conference and under less pressure.
And definitely, make open space a bigger part of the conference.
Thanks for working on this, George.
> I left Slack for myself, rather than fill every moment with The Most Important Thing.
🙂 Beautifully put.
I have no gripe with the submission/review process (but having had 3 sessions accepted may bias me). I’d have liked some producers and official stage reviewers to have been more verbose with their feedback –some were literally 2-3 word subject headings with no content– but most of the other feedback was pretty full, and very helpful to me in fine tuning my sessions.
Knowing why a session is rejected would be helpful, especially if the session received mostly good reviews.
Agile2009 still has the best submissions process I have ever encountered. Sure there is room for improvement, but let’s not overlook just how transparent and fair it already is.
As a reviewer, I really felt like I was missing some critical info: is this person a good session-giver? Eval forms won’t tell me that, I need something richer. One such thought is that prospective speakers use something like Speaker Rate when they give talks so that listeners can say what was good and not. Even newbies can use it: give a talk in your company or at a local agile group event and have people rate it. I tried out Speaker Rate for the first time this year and, while only one person took the time to rate me, I think its still a pretty good idea. Check out how it works: http://speakerrate.com/lyssaadkins.
Hey, maybe someone can build a phone app that lets people type in a rating and their thoughts about a session (ala short Tweets) into something like Speaker Rate. If LiveAid can build a donations app in four days, it’s completely do-able. Nothing is beyond this community.
I’m quoting feedback I received from a reviewer just as we were nearing the end of the selection process:
I dig the review system we are using. I think it provides more feedback and
transparency. I also know many people who are frustrated with the process.
Some simply do not like having their sessions visible to everyone, while
others (including me) find the review process an inconsistent experience:
– not knowing where to submit
– not knowing what to do when a submission moves from one stage to the next
or knowing when it has moved
– having wildly different review processes from stage to stage
– asking reviewers to provide more than a one word review (I had many of
these for sessions I submitted)
Personally, I like the stages which provided three questions as a guide. It
felt more acceptance driven that other stages where the review I received
was one word – not very helpful.
I suggest that the committee gives the stage producers some guidance on
review criteria. I think this allows the producers to select a process which
works for their stage while also helps the submitter have a more consistent
experience. I also think it is in line with our community’s outcome driven
approach to delivering value.
I am not trying to quell innovation (for each stage). Instead, I think
providing some acceptance tests for reviewing submissions would help provide
some consistency. I think the producers need to be more aware of the
submitter’s experience, for each stage and across stages.
I have heard frustrations from other submitters about this last year and
this year and I think it would help for the committee to give a bit more
consideration to this missing persona (the submitter).
Lyssa’s post reminds me: where did the feedback from the ThoughWorks application go? Anybody know if they will be sending it to the presenters??
Damon
Damon, I don’t know, and I don’t even know who to ask. I asked around, and they asked around, and I’ve seen tweets that this is in progress.
My only request for improvement is to publish the # of sessions of each length per stage that will be accepted. If only 2 longs talks are going to be accepted, it would be good to know that going in
Thanks for the comment Megan. I can’t answer for other stages, but for the Distributed Agile stage that wasn’t known going in. There was a bias toward shorter sessions in order to accommodate more. I lobbied hard for the one 180 minute experiential workshop on that stage, and having attended it, I’m glad I did.
I will provide more thoughts later. First a quick reply to Megan: It was not known at the beginning how many 180 minute sessions we would take. However for Manifesting Agility a single 180 represented 1/6 of our allocated time. Sessions had to pretty amazing to get the nod. In addition for a session of that takes that much of our time I had to be convinced we had top notch presenters who could make it work and make it very interactive.
BTW George this tiny comment box is a hassle 🙂 Maybe wordpress gives you some options for this?
[I looked around, but couldn’t find any options for comment box size. You can paste in text from a text editor, though. –George]
I think that Lyssa is bang on it would help to have speaker ratings of some form. Many speakers I know, some I trust already – but for the unknowns it would help to know how others perceive them. It would make it easier in taking a very big leap of faith.
More feedback to submitters! Not sure what happened between 2008 and 2009. In 2008, I got lots of detailed feedback before the deadline which helped me improve a poor proposal into one that was eventually accepted. In 2009, very little feedback before or after the deadline. It would be nice to know why proposals were rejected. I got glowing remarks during review, then rejected. Lack of space? or flaw in proposal?
Or flaw in my character?! 🙂
I have a long-ish history with the Agile20XX series of conferences. I attended ADC & XP/Agile Universe before there was an Agile20XX series. I was on the committees for Agile2005, Agile2006, and Agile2007. So I remember the bad old days of the completely opaque, command-and-control process in place in 2007 and before. It was excruciatingly painful and at odds with the Agile values.
Thus my hat is off to the Agile2008 & Agile2009 conference committees for creating the current open, community-driven, highly visible process. I am inordinately grateful that any interested community member can see submissions as they come in and comment on them. As a submitter I found the early feedback invaluable in shaping my submissions. As a community member I appreciated the insight into how the conference was shaping up, and how easy it was to get involved.
Yes, I understand that there are still spots of opacity, including the actual selection process. But the current process is such a vast, revolutionary improvement over the past process that the remaining small bits of opacity don’t bother me.
In fact, not only do the small bits of opacity not bug me, instead I find my cranky-old-person voice muttering in the back of my head: “Harumph. Kids these days. They have no idea how good they have it. Back in MY day there was no feedback. There was no opportunity to improve a submission. There was only ‘accepted’ or ‘rejected.’ The track chair had to ASSIGN each and every selection committee member submissions to review.” Cranky-old-person voice then retreats with a dismissive, “Now get off my dang lawn.”
That said, I honor those who restlessly seek further improvements to the process: greater visibility, more feedback, improved communication. I respect their concerns. I hope that those who want things to be different will be spurred by their dissatisfaction to get deeply involved, volunteer on the conference committee, and make things even better.
Personally, I’m thrilled with the outcome of Agile2009 and I eagerly await Agile2010 even if absolutely nothing changes about the submission and review process.
Elisabeth, you look great with that Clint Eastwood snarl on your face (from Gran Torino). 🙂
You’re absolutely right that the process is far better than most conferences. It also appears to have slipped back in some ways from 2008, even while moving forward in others. And, as a first-time reviewer, I found some parts of doing that job harder than they might be.
The one strong suggestion I’d make is limiting peoples votes. Make it scarce commodity so they have more value. A stage manager could have unlimited resources, but a regular person should only have N votes or comments. I think this would eliminate a great deal of the casual (and useless) chitchat, political endorsements of “my friends”, etc.
To be fair, I’ve not been to one of Agile 2XXX conferences, but I speak extensively in other venues (like the No Fluff Just Stuff Java/Agile tour).
Jared’s comment above reminds me: I volunteer to help build a manipulation detection system, at least a crude one. There was definitely some dubious voting I saw, and it would be great to make it easier for reviewers to spot that.
Erik, the stages are really just intended to bundle presentations vaguely similar in subject matter, but done in different styles. It’s a big improvement over sorting by Talking Head, Experience Report, Workshop, etc. I think the choice of stages could be better, however. And I agree that there should be space for small talks, too. Perhaps that’s appropriate for Open Jam?
Gil, when you say “I like the stages which provided three questions as a guide.” What stages? What questions?
For what it’s worth, they deleted the public “voting” system that the Agile 2008 submission system had from the Agile 2009 one. Instead, they allowed comments from everyone, and reviews from just the reviewers. The text of reviews were visible to everyone, but the yes/no/maybe votes were only visible to the stage producers.
Much of this was not apparent to me, as a reviewer, until some time into the process.
About manipulation of “voting” – the thing I’d like the conference website to make clear is that the conference is not constructed by democratic process. The community comments are input for the selection committee and feedback for submitters. But they are not votes.
The selection committees are entrusted with making the decisions about what’s accepted and what’s not for their stage, much like a Product Owner. They are not obligated to bend to the will of popular opinion. While I have not served on a stage committee for Agile2008 or Agile2009, I expect that each committee considers the input from the community, and weighs that input along with all the other considerations for putting on a good show on their stage at the conference. And so I also trust the committees to ignore mindless advocacy and listen to anyone who takes the time to make substantial comments.
To that end, I have no interest whatsoever in seeing any system put in place that limits anyone’s involvement by restricting how many comments they may make. Someone who cares passionately enough about the conference to comment substantially on every single submission should be allowed to do so.
This comment is not in response to any particular other comment. In any case, I wanted to elaborate a bit on how I found the public voting system useful. As a presenter, I liked being able to see how people rated my proposals, whether they were an official reviewer or not. As a presenter, I’d like to get a quick sense of how my proposal is being perceived so that I can rapidly iterate it. While the final word comes from the official reviewers, it is very helpful to get feedback from anybody that found their way to my proposal and feels strongly enough to say something/rate it. I thought that the rating system added an very useful dimension to the feedback. For instance, somebody might say something that sounds critical but then provides a very high rating. Another person might say something that seems supportive, but comes with a very low rating. Good to know.
I the concept of “Stages” was a fine idea for helping people to process the large number of sessions.
But, I think it was taken a little too far when it was then physically tied to an actual Stage. This implies that all talks on the same topic would garner the same sized audience.
I sat in multiple talks in very large rooms with less than 10 people in the room. I also was unable to attend a couple of sessions that were busting at the seams in smaller rooms.
I would personally prefer a session booking process be implemented. That way I have the freedom to finish the important conversation I am having and arrive right on time for the next session and now there will be a seat for me, rather than having to end a conversation prematurely just so I can go stand outside a closed door in a hallway and wait for the previous session to finish so I can rush right in and get a seat.
That said, at least if there was an informal online conference organizer (like the one in the iPhone app), the conference organizers would be able to get a feel for the # of people who intend on attending a session and select the right size room.
I will follow-up on the ThoughtWorks application. I believe it was the intent to share the data collected.
ThoughtWorks is still gathering feedback data. Tweets were sent earlier today, encouraging everyone to complete their reviews. The entire set will be sent to Agile Alliance later this week.
This year was my first involvement with organizing this conference although I have been attending this conference since 2003. We somehow made the Agile Frontier stage appear and Eric Willeke and I acted as Stage Producers. In order to prevent even the suggestion of bias I didn’t submit a proposal on purpose.
Anything I say here about the selection process is only valid for the Agile Frontier stage as this is the only one I was involved with.
We had a great team of very active reviewers: Brandon Carlson, Chris Matts, Karl Scotland, Marc Evers, David Anderson and Joshua Kerievsky, Eric Willeke and I were also doing reviews (I hope I didn’t forget somebody, sorry if I did). We tried to provide feedback as soon as a proposal was submitted to the system. Some general and completely personal observations:
* The sooner a proposal was submitted the more time you have to perfect it. And you’d get multiple feedback cycles.
* Sessions submitted within 24 hours before the deadline received less attention because the reviewers are volunteers and have a thing called real life in which they work and sleep.
* If a proposal was submitted so close to the deadline that no feedback cycle was possible I doubt if the submitter really gets Agile and its many ways of collecting feedback.
* Bombarding the system with many proposals ( > 3 ) reduced your chances to be accepted on the Frontier Stage as you are not able to select your most valuable session so how would we be able to do that.
* If reviewers and producers from a stage recommend a different stage for your session it’s best to do that as your chances of being accepted at your current stage are slim.
* Unfortunately there was no email warning system otherwise we could have been even more responsive.
At the Frontier stage we had a preference for shorter sessions to allow more ideas to be presented. Discussion can then be continued at the OpenJam. As a stage producer I would love a session taking a 45 minute slot for a 15 minute presentation that is very compelling. We care about the ideas presented not the form.
The final session for the stage were selected by the stage producers taking in account many factors among which topic, presenter, reviewer comments and votes, community comments and personal preference. They were specifically not decided by democracy or popular vote as that would be completely inappropriate for the Frontier stage (If you want to know why read the Agile Frontier stage description).
I was very pleased with the conference and the submission system. Of course there is still plenty left to improve, but except for the final decision almost everything is transparent.
I was a participant and presenter at the conference, but was not involved in its organization or planning. FWIW, my observations are:
1. 2009 seemed better organized than 2007 or 2008, from my individual perspective.
2. All the sessions I attended were valuable for me. No duds.
3. I liked that there were more interactive sessions this year and fewer talking head sessions.
That’s the good news.
4. I’ve never understood the concept of “stages”. There doesn’t seem to be any real difference between them. I just choose sessions individually. For me, chopping everything up into “stages” only adds confusion.
5. It was very hard to find which sessions were scheduled at what times, either on the website or in the printed guide. There was no alternative but to read all the material sequentially until I found something of interest. A single web page with a simple list would have been simpler. Maybe a keyword search, or something like that. The large boards in the hallways at the venue were the easiest-to-read presentation of the schedule.
6. The only conference I know of that has a truly open and “agile” submission/review process is XP Days Benelux. If you want to learn how an open process works, ask the organizers of that event. Not sure if it would work for Agile 20xx because of the dependency on commercial sponsorship. Session content has to support whatever the sponsors are selling, at least to a certain extent. I don’t mean to suggest the conference is a direct sales channel for sponsors. I just mean that the conference has to attract an audience that might be interested in the sponsors’ products, and the content of sessions (or at least a significant subset of it) has to lead participants to think the products on display are relevant to their needs. With a truly open review process, the organizers relinquish control of content to the participants themselves. It then becomes a matter of chance whether the sponsors’ product offerings will support the practices being proposed in the sessions. There’s the potential to lose vendor support for future events. So, although some people are calling for a more open review process, it may simply be impractical for this particular conference.
7. The “frontier” stage didn’t accept a number of submissions that truly addressed the leading edge of practice or that raised questions about status quo practice. IMHO this is also because of the dependency on commercial sponsors. They have to attract an audience that is likely to purchase their products. That means the conference must target early adopters and not innovators. This is probably unavoidable short of breaking the conference up into a series of smaller events that can be organized on a smaller budget and supported by registration fees alone.
Great feedback, Dave.
Stages: Do you think a different selection of stages could be more helpful?
Frontier: Do you have examples of these submission topics?
Only problem was picking a stage, and received guidance only after it was too late to change it.
I think it ended up in the right place, but timing made it iffy. Would hate to have been rejected because I picked the wrong stage and didn’t know I did it wrong.
I found the comment dynamics around submissions to be confusing. It was unclear how to respond to reviewer comments vs. open comments.
I found the review dialog to be lacking in general. Responding to comments with a change to the description elicited no feedback from my reviewers.
It was frustrating to not be able to make changes after submissions were closed. I was not able to revise based on comments even though reviewing had not started yet.
As a stage assistant in 2008, I know that there will always some degree of arbitrary in the selection. Someone once complained to me that one of his sessions were not selected in 2008 (another was), even though it got a relatively good grade, and I painstakingly tried to list our thought process. The final step, though, was still “because we did not have a good feeling about it”.
Now, since you are asking for feedback, here are my observations:
* I agree with Dave that the Frontier stage did not really address the leading edge of agile. If you recall, there were two stages in 2008 that did (Breaking Acts and Questioning Agile).
Here is an example of a session that did not get accepted: “Is Scrum Evil?” (http://agile2009.agilealliance.org/node/571). I was *very* disappointed that it did not get accepted (while another, very mundane, session of mine was). In the end, the session took place during Open Jam, but it didn’t get the visibility it deserved. Also, I am very surprised that it got no real reviews (though it did get comments). Where reviewers afraid of others reading their opinion on Scrum?
* I felt there was a curiously low number of sessions selected from non-US presenters. The general feeling of the persons I talked to in France is that this is because the stage producers and the reviewers were mostly North American, and therefore did not know non-Americans well enough.
FWIW, I count 23% non-NA producers & assistants this year, compared to 40% last year.
* I think a major criteria for rejection is when a session goes back and forth between stages. I believe this is what killed the “Is Scrum Evil?” session faster than anything else. Because it was maintained in the Coaching stage for a while, it got mixed comments (in the sense “why is it in this stage?”), and when it arrived in the Frontier stage, reviewers probably noticed that it has enough comments already and neglected it.
* a final problem for me was the levels of the sessions. The main problem was that there were too few Expert-level sessions, and many were not for experts at all (for example, the session on Scala -which I enjoyed a lot- was certainly not for Agile experts; not for Java ones). I’ve been quite disappointed by this.
I’m not sure how to address all these points. Here are a few things that might help:
* restore the notation system by the common man, in addition to the notations by the reviewers, but make clear that it is not the only criteria
* ask the commentators and reviewers to provide their own evaluation of the complexity level
* arbitrarily mark 25% of the sessions as introductory, 50% as intermediate, 25% as expert, based on the grades for the complexity level
* appoint a “well-connected” & authoritative person to salvage sessions there were not selected, based on personal taste. The Program Chair would seem like the right person for this, but being quite busy already, I’d rather she appointed someone else as an aid. One of the original Agile Manifesto signatories / Gordon Pask Award winners could be an ideal choice for this task. I can see something like a “Joe’s choice” stage. Or a “Gordon Pask’s selection” stage.
* enforce greater diversity in the reviewers for a given stage. Maybe impose that 50% must be unknown to the producers. Or make the Conference & Program Chairs appoint some of them directly.
George, thank you for setting up this discussion. I enjoyed meeting you finally after seeing your postings many times on various lists.
I participated in this year’s conference as an organizer, attendee, and presenter. I found a few truly great sessions, many good ones, no duds.
I would like to have seen more variety of advanced sessions, meant for those of us who’ve been coaching or who’ve been practicing Agile a long time.
Like others I found the stages to be somewhat more confusing than helpful. I count 20 stages from the conference website. What if we limited the number to 7? We limit the size of agile teams for good reason. We could try limiting the number of stages.
At first I was going to say that’s what we should do but thinking about it further brought up deeper concerns. I can see 2 reasons to have categories at all (whether you call them stages or something else): it helps attendees select appropriate sessions for themselves, and it helps the conference organizers to know who should review the submissions. A third reason might be that it tells you which sessions you can schedule at competing times. OK, let’s separate out these 3 concerns and think a little on each…
First, a way to help attendees select appropriate sessions. This can be served by tagging sessions according to intro, advanced, etc., job role, industry, etc. All this is do-able. But the only real choice for attendees is set by what sessions are offered when. So if you’re in a venue where 5 things are offered simultaneously, you can only support up to 5 “tracks” without conflicts. You can take the personas that you’re appealing to and aim for a session schedule that minimizes clashes for them. So the 3rd item (scheduling) is closely tied to this one.
Looking at the other end – the mechanism to get the submissions reviewed – gives a different set of concerns. Here I would not want to artificially limit the number of categories. That would dampen the quality of the conference. Outstanding talks might get rejected just because they don’t fit a predefined category. And with truly new ideas, you don’t know what you were looking for till after you find it.
From a practical standpoint it makes sense to have a team of reviewers – a small them, therefore multiple small teams. That necessity imposes the need to select these reviewer teams somehow. For that purpose the 20 stages seems more ok to me. It was confusing to reviewers though, so maybe we just need a better coordination system for passing things from one stage to another.
I wonder if it would work better to not have the submitters choose a stage. Maybe not even have all those stages be visible to the public. What if the submitters were instead told (in real time) that we’ve got too many of this this type or length talk and are looking for more of that topic by someone with such-and-such experience. At the beginning they see the conference theme, and a statement about the general types of content wanted. Then they start getting the real-time reading on what we’ve got too much of and what we’ve got too little of.
Even though we attempt to cover all possible session types by the way reviewer teams are constituted, suppose a great session proposal comes in that doesn’t really fit any – or fits 3 of them. The real need is to get the right reviewers on it (borrow some from different review teams?) and decide whether it should get presented. All that should happen without the submitter having to worry about it. All they need is some feedback, guidance, and a decision. They should feel like they’re in a dialog similar to what an agile team member has with a good product owner.
I know I haven’t proposed a solution here. I’m trying to just explore the issue a little. We’ve got a lot of things working well – let’s not toss that out with the parts we found frustrating. Ideas?
I like most of Nancy’s ideas. I think I disagree about one point, though: Tracks or stages don’t really offer a good basis for avoiding schedule conflicts. I think the Personas would be a better basis for that.
For instance, consider a person who has an interest in growing agile beyond the single-team level. He/she might be interested in sessions that are categorized on different tracks or stages, such as Planning and Tracking Large-Scale Projects (management track), Common Components and Solution Architecture in Multi-Team Projects (technical track), and Challenges in Remote Collaboration (a track having to do with human factors). The titles are made up, of course. This person would not like to find all three sessions scheduled concurrently, especially after having paid a great deal of money to register and travel to another continent to attend the conference.
Remember our closing keynote speaker, Jared Spool? He was all about User Experience. Why not think about the program from the point of view of attendees, rather than trying to dream up arbitrary categories, tracks, or stages?
Several commenters seem to think this year’s review process was “open.” It wasn’t. With different reviewers acting autonomously for different stages, there was no coordination of the program and, as Eric pointed out, a proposal might be batted back and forth between stages only to be neglected in the final analysis. Some comments carried no weight while others (the reviewers’) were decisive. No public explanation of the rationale for accepting or rejecting each submission was offered. Having any portion of the review/decision process done in secret renders the whole process “not open.”
Let’s not break our arms patting ourselves on the back for having an open process. I suggest once more to look at the way XP Days Benelux is programmed. That’s an open process. This isn’t.
@dave nicolette Commercial interests had nothing to do with the selection of the sessions for the Frontier stage. We tried to select a range of sessions on topics that were either underexposed or questioning agile or just not agile. And we tried to balance the topics on this stage.
Have we left out great sessions? I am absolutely certain we did, we only selected less than 14% of the proposals. And more than half of the overall proposal were of such great quality I would have loved them at the conference.
Could any of these sessions have replaced a selected session? Definitely!
We just could not select all the sessions we would have liked. We chose to go a certain way and you may have made different selections if you were in the same situation. I actually may make different decisions if I have to select sessions now. At that point in time we made the decisions based on best effort with the knowledge we had at that time.
If you don’t agree, blame me. I was responsible.
@ Eric Lefevre-Ardent I looked up your session and I agree there is something wrong here. I remember checking the list of submissions over and over, making sure each got reviewed and a chance of being accepted. I can’t explain why I have missed your proposal, I apologize for that. I think it would have been a good session for the Frontier stage.
What topics would you have liked more of on the Frontier stage?
@olav, I’m sure you didn’t take any direct orders from commercial sponsors. Sorry to have given you that impression. I do think it’s fair to consider the interests of the financial sponsors, because we would have no conference without them, and they have to get some sort of return on investment. Targeting early majority adopters more than innovators seems perfectly logical.
“Blame” is not interesting. It’s not even relevant, since the conference turned out quite well. The only interesting question is what can we learn from Agile 2009 that we can use to make Agile 2010 even better.
I agree that we need to see what we can learn to improve the next conference.
This year there wasn’t supposed to be a questioning agile stage or something similar. Because of a blog post by David Anderson and many comments to that blog post the Frontier stage was added.
I wish for next year’s conference to include something similar to the frontier stage to the programme. Maybe more focussed on sessions that are over the leading edge.
I hadn’t thought to add this before, but speaking on behalf of a sponsor, here are some thoughts. I’ll also mention that I’ve been involved in many vendor relations types of things, including an Agile roadshow that a bunch of vendors are participating in right now.
First, I wouldn’t worry too much about linking sponsorship and session content/process. Being associated with something as great as Agile 2009 is the key draw for a sponsor.
I think it is good for all concerned to have “vendors” be treated the same as everybody else. After all, a good session is a good session. If you don’t have something useful to say in general, who wants to hear it anyway?
My thoughts on how to make vendors happier have more to do with expo hours and stuff like that.
In summary, my advice would be to not worry about sponsors when it comes to the session submission process, stages, etc. Focus on creating great content and a great event and sponsors will be excited to be a part of it.
@olav well, I’m sure the reason it was not selected is really a combination of factors.
I don’t know exactly what sessions I would have wanted to see. But take the “Question Agile” stage in 2008 (http://agile2008.org/stage-questioning.html). It did had a number of advanced sessions.
I also would have liked to see more of sessions bridging agile and some other domain (theater, say).
I’m not blaming you, though. The fact that the stage was a bit late to come didn’t help the quality of submissions.
I do wonder why there were more controversial/advanced submissions last year, though. Different comfort zone for submitters? but why?
Oh, I have another suggestion: we could let the submitters do more lobbying. A simple button “Talk to the stage producers” might do; in the case of “Is Scrum Evil?”, it would have made a difference. And I’m quite sure it won’t be abused: what do you think stage producers will do when received to many requests from a submitter?
@Damon: I don’t think anybody is consciously worrying about the sponsors. However, I can imagine that submitters and reviewers and producers will unconsciously walk on eggs. With 1500 participants, there is a risk to expose yourself as “not a team player” in the industry, which is not good for potential customers, partners or employers.
From that point of view, I understand Brian’s reaction on our sessions: http://agile2009.agilealliance.org/node/571.
I guess everyone has been mulling over lessons learned and thinking about how we might do things differently for Agile 2010. Here are a few ideas for purposes of discussion:
These suggestions can be summarized by the phrase, “Do the simplest thing that could possibly work.” Perhaps you’ve heard that expression before.
1. Problem in 2009: Stages
Submissions had to specify a stage. Submissions for each stage were reviewed by a different set of reviewers. There was no coordination across stages. Submissions that were moved between stages tended to be ignored or slip through the cracks. The stages didn’t really make much sense. They seemed to be someone’s predetermined idea of categories, and those categories didn’t always line up with the actual submissions. Many submissions could be thought of as pertaining to multiple stages. In some cases, reviewers just pushed the proposal away claiming it seemed to fit on some other stage.
Proposed solution: Have all submissions reviewed together. Don’t predefine any tracks, stages, or categories. If submissions and feedback cause categories to emerge naturally, then use those. Otherwise, they aren’t needed.
2. Problem in 2009: Closed review process
Submissions were publicly visible and anyone could post comments. However, only the comments of the official reviewers carried any weight in the selection process. The actual selection process was closed. Different standards applied to different stages. No rationale or explanation for selections/rejections was published. Submitters had no mechanism through the conference website to collaborate with one another to refine, combine, or coordinate their content. Even when submitters modified their proposal in response to reviewer comments, there was (usually) no reciprocal feedback to let them know whether they had made things better or worse.
Proposed solution: Provide a mechanism that facilitates self-organization and collaboration to review and refine submissions. Define a few simple guidelines for the information that should be included in a submission. Avoid arbitrary criteria that are disconnected from the quality of the content of submissions, such as how early or late the submission was made, how many other submissions were made by the same person, whether the speaker is well-known or not, whether the same session was presented the previous year, and so forth. The selection process itself can be totally transparent and lightweight: Those who are interested enough to take the time can vote on the submissions, or score them, or some such thing. IMHO people should offer reasons for voting a submission up or down.
3. Problem in 2009: Insufficient manpower to review all submissions carefully
Reviewers complained that with hundreds of submissions to read and only a handful of part-time volunteers to assist them, it was not possible to do justice to most of the submissions. Some submitters complained that their submissions did not receive the attention they deserved.
Proposed solution: Dispense with the idea of an official review committee, except for the purpose of coordinating the program to provide a meaningful experience for participants, and instead let those who are interested in the conference collaborate through a website to suggest improvements to proposed sessions and activities. Let the community self-organize to select the content for the conference. Take a page from the Open Space book and trust that whoever “shows up” will be the right people. If you aren’t comfortable with “trust” when so much is at stake, remember that it is one of our community’s fundamental values. If “trust” doesn’t work or if we don’t honestly believe in it, even among ourselves, then we’ve got nothing to offer at a conference anyway and we may as well not bother.
4. Problem in 2009: Participant experience
While many individual sessions were excellent and Open Jam afforded numerous opportunities for ad hoc activities, and while the organizers used Personas to help submitters focus their content to specific participants, the program itself was not structured in a way that offered a cohesive experience to any of the Personas. Whether you had a good experience at the conference was largely a matter of random chance.
Proposed solution: Encourage those who contribute to the review process to take a “user experience” view of the conference as a whole. When assessing the program as it emerges, imagine walking through the conference experience from the perspectives of various Personae, or at least from the point of view of a particular Persona with which you identify. If you find a “hole” in the experience, then collaborate with others to fill it. A key point is to ensure that from the point of view of any given Persona, there are no cases when several sessions of interest are scheduled concurrently. Of secondary importance is to ensure there are few times during the conference when a Persona has nothing interesting to do. With respect to that last point, I think we should remember George’s comments about Slack and not trying to “maximize your time.” But the point about avoiding concurrent presentations of interest to the same Persona is critical, IMHO.
5. Problem in 2009: Website software
Some people complained about functionality of the conference website. I don’t recall their specific complaints, but they are all in the nature of “Why can’t I do this or that on the site?”
Proposed solution: Use a simple wiki as the submission site. This enables interested parties to collaborate freely and without running into arbitrary functional barriers just because the authors of custom software didn’t think of it in advance. Everyone knows what a wiki can and can’t do, and they can work with one to collaborate. This will also save effort and money since there’s no software development to be done. Registration functions can be outsourced to an external service.
Responding to Dave Nicolette’s comment “Tracks or stages don’t really offer a good basis for avoiding schedule conflicts. I think the Personas would be a better basis for that.”
We’re not in disagreement. I only meant that the organizers need to structure the sessions looking at it from the attendee’s perspective (as you also stated). Personas is a more flexible way than tracks. But whatever you choose, there is still a judgment involved in deciding who this conference is aimed at.
I’d like to add an item to your posting no. 43 as follows:
6. Problem in 2009: Many attendees’ cell phones had no reception at all on the floors of the conference sessions. This made it difficult or impossible to meet up with people they wanted to see.
I have two issues I would like to bring up about the conference:
1) Too many long sessions – I attended many great 90 minute sessions that also could have been edited to 45 minutes. As a result, I missed sessions that were of interest to me. Yes, this is a problem at every gathering, but the producers have a responsibility to ask the presenters to get to the meat of their presentation quickly.
2) Producers are also presenters – this bothered me in 2008 and more so in 2009. It is really odd that the person reviewing my submission to a stage is also in competition with me. As a community, we need to be mindful of the appearance of cronyism and insider behavior or we will become stale, insular and detached from new ideas. I feel if you are going to review or produce a stage, you cannot submit a proposal for it. It just looks bad.
Carlton, WRT producers also being presenters, I think it would be hard to find producers if it disqualified you from presenting. OTOH, I’ve heard other comments that being a producer or asst. producer seemed to be a sure-fire way to get your sessions accepted. Even if not true (it wasn’t for the producers of the Live Aid stage), the perception is troubling. What to do?
Carlton and George, it occurs to me that my suggestion to move to an open review process would eliminate the perceived problem of producers who are also presenters. Decisions would not be made by a small set of people behind closed doors, but by the community at large in an open forum. Thoughts?
Dave – such an open system is easily gamed. In a system like that Jeff, Mary, Ken, et al get any/all sessions approved. You and I have a fighting chance and new people are screwed. In addition if I have a enough friends they can vote on my sessions and game the system. So open voting systems are good indicators of popularity but not quality.
Carleton – cronyism is a risk but here is the counter. My personal interests and research at the moment are in neuroscience, sociology, et al. This made both a naturally reviewer and presenter on the Manifest stage. So how do you solve the problem that the best qualified reviewers are also well qualified to present on their own stage?
BTW I’m happy to send you copies of my feedback so that you see what people thought of my session.
One idea that appeals to me is limiting the number of sessions that you may submit. Perhaps we should be limited to 5 submissions and 2 accepted sessions (as lead presenter). If the resource is limited there will be fewer scattershots.
In addition I would like to see the previous year’s speakers feedback made available to the reviewers. It makes it easier to accept a session from someone you don’t know if you can see their feedback.
Mark, “such an open system is easily gamed.” Please see my earlier comment about “trust.” I’ve seen “such” a system work very well; granted it was for a much smaller event.
I like your suggestion about making feedback available to reviewers. In fact, in a open system all feedback would be public.
It seems some in the agile community fear openness, transparency, and collaboration, and prefer to try and prevent occasional abuses of a system by discarding the fundamental agile value, trust. Interesting. Disappointing, too. Why should others believe in our values if we don’t believe in them ourselves?
It seems like there already is a perception problem with producers also being presenters on a stage – I hadn’t heard the comment the surest way to get your proposal accepted is to be a producer\assistant producer.
As for a an open review system, I don’t know. I would want some more information about it and I would want to know how other big, long running conferences select submissions. Other people have run into this problem before and have come up with solutions. We reinvnent the wheel?
The problem with grouping proposals into stages after the fact ignores the fact that there are so many submissions that it would be difficult for even a team of people to do well.
I have no experience managing a completely open review process – especially for a conference as large at this one. Can it work for a big conference?
Another possibility is something I saw in a solicitation from another conference – they said that papers “will be subject to a double-blind review process”. I’m not sure just what that entails, but I assume it means the submitter doesn’t know who’s reviewing and reviewers don’t know the submission authors.
If we had an anonymous system we wouldn’t need a rule saying you can’t submit proposals and be a reviewer. Could that work?
XP Days Benelux has used an open review process, and XP Days London pretty much so. These are much smaller events than the Agile 20xx series, of course.
Several commenters have mentioned that they have no personal experience in managing an open review process, or that Agile 2009 was actually the most open process they’ve personally seen. There have been expressions of fear that people might “game” an open review process. There have been suggestions to resort to illusion-of-control methods like a double-blind review process to try and prevent any possibility of gaming the system. Lions and tigers and bears, oh my!
Those comments indicate that this group of people simply sees no practical way to apply agile thinking to the problem of organizing a large-scale conference. The only way they can see to do it is to revert to non-agile thinking. Okay. It is what it is. The logical conclusion, then, is that we need a different group of people.
Maybe we should ask the organizers of the aforementioned events to take the lead in organizing Agile 2010. Their events haven’t been as large as Agile 20xx before, but the mechanisms they have used can be adapted to a larger event if people really want it to work. Some of the individuals involved in those events also have experience organizing past Agile 20xx events. Among them, there’s ample expertise in both agile thinking and large event planning.
I believe it is possible to organize a large event without abandoning agile thinking. The reason I believe that isn’t that I’ve done it before personally. The reason is that I have confidence that an agile approach offers a practical way to solve problems. Besides, we’re only talking about the review process. Other aspects of event planning remain the same.
Carlton asks, Why reinvent the wheel? The answer is that the agile community is in the business of reinventing the wheel. The whole movement is about changing the way people think about and act upon problems and challenges. If we’re the first to manage a large-scale event in an agile way, that’s a Good Thing; it’s a reason to be excited and enthused about the challenge, not a reason for fear and doubt. If we’re unwilling to live by the values we promote, then we’ve got nothing to talk about at a conference.
@Olav you’ve asked what sessions I’d have liked to see in the Frontier page.
I have one: more content about System Thinking (yes, I know that there was one presentation on that at the 2009 conference).
I know that System Thinking is old stuff. But the fact is that I got interesting in Agile after it was hot. So I rarely get to see presentations on it, and I do not get enough pressure from peers to bother reading books on it. So, a presentation would have been valuable to me.
Possibly contradictory with my call for advanced sessions. No matter.
Eric in my mind there is nothing Frontier about Systems Thinking – in fact after talking to Rachel, Liz, Declan and Mike Sutton at the conference I’ve got a whole lot of reading backlog on the subject. In my mind this is far from edgy these days.
In addition my long term hope is that we don’t need a Frontier stage at all. I would hope that edgy sessions far from the mainstream are found on every stage. That certainly happened on the Manifesting stage this year where we embraced Neuroscience (Linda and myself), Cognition, Psychology and Agile outside of software.
This is a bit of a late response – not too late I hope.
First of all, I should be clear that the Agile200x submission system is the best I have used, so the following comments are picking hairs. I’ll also confess to having a bias becuase I had nothing accepted this year!
1. I thought the comment/review distinction was unclear. As a stage reviewer, I was unsure which to use sometimes. It took me a while before I figured out that reviews were visible to fewer people. So I’d prefer to have a single review type. The Stage committees can use all reviews to make their decisions.
2. There was some talk that there was less gaming with this years system. I’m not convinced. Its possible any gaming was just less transparent due to the hidden reviews.
3. I wonder whether early submitters were penalised – their sessions ended up being at the bottom of the list. How can we encourage and reward early submissions? These should be the ones that get most feedback and get the most iteration.
4. Why was the deadline extended when there were already plenty of sessions? Again, this doesn’t encourage early submissions with lots of iteration. I’d like to see a fixed deadline for submissions, followed by a further fixed period for reviewing and iterating, before stage teams make their decisions.
5. Should there be a higher bar for submissions? A minimum number of words? This would encourage people to submit well thought through ideas, and discourage people from submitting dozens of bare ideas at the last minute.
6. It didn’t always seem clear which stage to submit to, and when people submitted to an inappropriate stage, then the process seemed a bit random. As a reviewer, I tended only to look at what had been submitted to the Frontier Stage, and quite likely missed Frontier Friendly topics which had been put on other stages, but didn’t get accepted. Could reviewers suggest alternate stages? Then as a reviewer I could filter for sessions suggested for my stage? Or as a submitter I could have the option not to specify a stage in the hope that it gets picked up by a stage reviewer?
7. Relate to point 6 – while each stage produced a balanced program, I’m not sure the whole program was balanced. Could there have been more cross stage collaboration to help submissions find the right home?
8. Having to log in to view submissions might have got in the way? I’d like to be able to view all submissions without having to log in, and only log when/if I want to review something.
Agile2009 was still a great conference. These points are with the hope of making Agile2010 even better still!
Karl