Zimmerman urges Metro to embrace “beta testing”

Metro is “more nervous than [it] need[s] to be” about opening up Web tools like NextBus to the public, said Arlington board member Chris Zimmerman at today’s Metro Board of Directors meeting, but Fairfax member Catherine Hudgins isn’t so sure. Metro staff still feel that closing off access to the real-time bus information tool until June’s launch is the best policy. This debate really gets at a fundamental issue which technology firms, too, grappled with mightily in the 1990s.

The issue first surfaced a month ago when DCist and New Columbia Heights’ Andrew Wiseman noticed that you could get into an unreleased beta version of the NextBus system. That system will tell riders, in real time, when their bus will arrive, like the digital displays do in Metrorail stations. Riders will access the information on the Web, via mobile devices, or by calling a hotline. Metro staff plan to release the service in June, but it’s been working, and many riders had found it quite useful and most often accurate.

However, when Metro found out about the public access point, they moved to shut it down, claiming it’s not yet accurate enough. Many riders argued that something is better than nothing, and they’d rather take the risk. Blogs and riders criticized Metro’s decision, but staff and General Manager John Catoe arguing that the service was not ready and that opening it up would trigger a “flood of complaints” that Metro would need to answer.

This issue boils down to two simple questions. First, is it better for Metro to release something imperfect and essentially ignore complaints, or to wall off that imperfect thing so that nobody can access it and complain? And second, should Metro develop software based only on their own internal opinions, or collect input from a wider group before decisions are final?

RAC chair Diana Zinkl presented the issue to the Board today in her monthly presentation. The topic arose in response to the RAC resolution I introduced. Assistant General Manager for Information Technology Suzanne Peck explained Metro’s reasoning to the Board. After listening to Peck’s analysis, Zimmerman said,

I appreciate that very candid appreciation of the staff view. It’s the first I really heard on any public forum. But I think that some of the running dialogue that’s been going on, between a segment of the public that’s particularly attuned to the use of Web tools and this organization, has been about the question of when is it okay to show something.

The critique on the part of that segment, which in fact is now represented specifically by a member of the Riders’ Council who offered that motion, has been that Metro’s view is basically to not risk anything, to get everything perfect before it goes out. And I think we just heard confirmation of that. I think you heard an honest and sincere expression of that viewpoint, that Metro shouldn’t put anything out until it’s really really right and good.

The view of folks in this world is that, you know, it’s really better to not try to do it that way because, in fact, on the Web you don’t get things perfect until you have a lot of input anyway. If you try to wait until get it perfect you’ll never get it out there, and that’s why Metro is slow in implementing a lot of these things.

I just want to say, Mr. Chairman, that I think that’s a valid criticism. I can see for certain things that’s true, particularly where safety is actually involved. [But] it’s hard for me to see what the risk is that I might go onto the Web, look into some site that’s supposedly predicting when the bus is going to arrive, and it could turn out to be wrong. Especially when, right up front, I’m not getting in through the main Metro web site because it’s not ready to be there yet. I have enough sophistication to have found out how to get in. And whatever I do there, it tells me this is being tested, we don’t promise that there’s anything accurate about it.

The worst that happens is my bus doesn’t arrive when it said it was going to arrive. And when I’m standing there at the bus stop now, I dont have anybody telling me when the bus is going to arrive. I just know it’s not arriving when the schedule told me it was going to arrive. I don’t really see that there’s a risk here.

I think we’re more nervous than we need to be. And I think I now understand a little bit better the nature of this discourse.

Fairfax member Catherine Hudgins agreed that Metro ought to be engaging the public in the development process. However, she also echoed Peck’s concern that opening it up in the meantime would trigger a “firestorm of complaints of innacuracy,” distracting staff from the real task of finishing the real service. Hudgins said,

We want to convey that we want our public involved … [in] developing the scope of what you’re going to do. But I think there is a fallacy in the fact that you can put things up and assume that it will do no harm to Metro, because it means that our staff will need to respond and will need the time to respond, and will be putting out fires that they should have put out in the closure of their development process.

Hudgins, Peck, and Catoe are right that launching the service for real is more important. We don’t believe staff ought to launch it now and immediately start spending time answering complaints. That would clearly interfere with the main task. However, riders aren’t asking for that. They’re asking for Metro to open it up and explicitly disavow any responsibility for its accuracy or for supporting it.

If someone waits for a bus which doesn’t come, they might complain to Metro. More often, they don’t, and silently curse Metro. If they could access an unsupported service, then it might help improve their ride. Or, it might give wrong information. If it’s wrong, maybe they would be more likely to complain. But would it be so bad if Metro told them, “sorry, but this is unsupported, and while we appreciate your input, we aren’t ready to handle complaints yet”?

From the IT staff’s point of view, complaints about bus service which don’t go to them are less intrusive than complaints about a beta tool which might go to them. But from a savvy rider’s point of view, it’s better to get the information at their own risk and decide for themselves whether to trust it. As Zimmerman pointed out, being frustrated with Metro because the real-time information is wrong isn’t really worse than being frustrated with Metro because the schedule is wrong. And while not completely accurate, the real-time information is definitely at least more accurate than the printed schedule.

The other reason to open up the data involves Metro’s ability to get useful input. Their IT staff are very intelligent, I’m sure, but more heads are always better than fewer. They’re not going to see every use case. For example, if you go to wmata.com and enter information into the trip planner, get a trip, then hit the Back button to change it, the Javascript on that form clears it all so you have to start over. Why didn’t they notice that? It’s not really their fault; there are always things you don’t notice. However, the bigger question is, why didn’t they let some people try out the site before it was finished, accepted and paid for, in order to find and correct these flaws? Now, it’s slower and more expensive to make changes.

At the RAC meeting, member Kenneth DeGraff pointed out that for people living on bus lines with multiple numbers, like the 90s, the old system required requesting next arrival for each line individually. That’s time consuming. Did Metro think of this when outlining requirements for the new version of NextBus? We don’t know, because we can’t see how the new one works. Who knows how many sticking points there are that we won’t find out until some people start using it?

Peck talked about the way Microsoft does structured beta testing with large numbers of people. Hudgins added that Metro lacks Microsoft’s resources, and added that she doesn’t want unfinished Microsoft software. That’s true. However, there’s a huge difference between Microsoft and the Web. When Microsoft ships a version of Windows, that version is out there for years. It runs on millions of computers with their own different hardware and software. It’s complex, and very time consuming for them to update it. Until recent years, they couldn’t even automatically update it at all. On the Web, on the other hand, the site controls everything. The software isn’t on my computer, it’s on their Web server. They can change things daily, and the best companies do. Microsoft has to step carefully before releasing something. A Web site, however, can simply turn it on and then change it.

In the 1990s, the technology industry grappled with this dilemma. Everyone was used to writing software on a “release cycle” of months or years. Microsoft would release one version of Windows, then start on the next for a few years later. But with the Web, that became far too slow. Web companies couldn’t spend months collecting requirements, building a priority list, changing things, testing, and launching. Competitors would have left them in the dust. Instead, companies like Netflix launch a feature, see what happens, then improve it or delete it. Google always tries out new features on a small subset of users before launching them for real. These companies gain valuable insight that they could never have figured out by just sitting around a conference table talking about the product or from a focus group.

To her credit, Peck agreed to involve the RAC in reviewing the NextBus system. So far, while we received a presentation on the general details, we haven’t gotten to try it out and give specific feedback. I’ll work with Peck and Zinkl to set that up. I’d like to enable a subset of readers and bloggers to attend a meeting and review it as well in a sort of focus group. It’s not as good as letting everyone try it out and harnessing the “wisdom of crowds,” but it should help Metro staff avoid some larger pitfalls they might not have seen.

NextBus will be out in only a few months, and while it’s too bad we can’t all use it in the meantime, those months will pass quickly. However, there will be many more IT projects in the future. They needn’t work out like NextBus or the WMATA Web site. Involved riders should be able to weigh in at the very start, when Metro is working out the scope of work for the project and before contracts are signed. Then, at least some people should participate throughout. And finally, when possible, Metro should turn on the new feature in an unofficial way, with explicit refusals of any support, to reality test the product before it’s set in stone.