Open source and the production of knowledge

Bob McHenry, a former editor in chief at Encycloapedia Britannica and my former boss, has an article in Tech Central Station on Wikipedia (discussed previously here and here).

To put the Wikipedia method in its simplest terms:

1. Anyone, irrespective of expertise in or even familiarity with the topic, can submit an article and it will be published.

2. Anyone, irrespective of expertise in or even familiarity with the topic, can edit that article, and the modifications will stand until further modified.

Then comes the crucial and entirely faith-based step:

3. Some unspecified quasi-Darwinian process will assure that those writings and editings by contributors of greatest expertise will survive; articles will eventually reach a steady state that corresponds to the highest degree of accuracy.

Does someone actually believe this? Evidently so. Why? It's very hard to say. One possibility that occurs to me is this: The combination of prolificacy and inattention to accuracy that characterizes this process is highly suggestive of the modern pedagogic technique known as "journaling."... It may well be that the practice of journaling in the schools, along with the acceptance of "creative spelling" as a form of personal expression not to be repressed, underlies much of the success of Wikipedia.

Superimpose on this intellectual preparation the moist and modish notion of "community" and some vague notions about information "wanting" to be free, et voila!

Leaving aside the question of whether journaling is responsible for faith in Wikipedia, Bob's piece does point out a significant issue about Wikipedia: the lack of peer review. Leave aside the absence of editorial standards or consistency (which are good things, but in my experience most readers don't really notice them), or the lack of systematic fact-checking. As a number of authors have pointed out, successful open source projects incorporate peer review mechanisms, and often operate in environments in which the criterial for measuring success and failure are understood by all participants. Code either works or it doesn't.

Fuzzy content, in contrast, is a different story. Here, even if you get all the basic facts right, experts can argue over how the facts fit together, or which facts matter. The assumption that contributors share a common understanding of what defines a decent article on a subject, what constitutes important facts, and what sources can be trusted to back up claims, which can be used to guide the growth of Wikipedia, may not be warranted. If the open source model is to be extended from software development into areas where debate over ground rules and epistemology are regular affairs-- as in history and encyclopedia writing-- this problem will need to be solved.

Comments

[Greg]

Very good points - and I could almost substitute "journalism" for "journaling" with no loss of meaning or rigor. In this case, the wikipedia model is almost a microcosm for the current "blogging versus mainstream media" debate that's all the rage. It also mirrors the original Dmoz directory issues.

The whole discussion leads me to something I'd come across in Bruce Sterling's excellent novel "Distraction". As I recall, it was a "trust-based" economy of information, where one moved up the ranks by gaining greater degrees of (presumably measured online) "trust".

Perhaps what is needed, then, is a "trust index" for wikipedia contributors (and the blogosphere). Or perhaps that would merely iterate the problem to another level.

In any event, such a script would be interesting.

And fun.

[Franz]

Thanks for pointing us to the McHenry article, quite interesting ... especially from someone involved in the traditional encyclopedia editing process.

I do concur that its hardly reasonable to expect an implied 'darwinian' editing process to end up producing truth.

Yet why do I enjoy the Wikipedia so much? I have begun a habit of using it as a source of links for my blog entries (see the previous post in this blog) In a few cases I even found it stretching and enlightening, in the history of flight, for example, where it provides a less than US-centric view. It appears to be 'reasonably correct', though danger does exist outside my own knowledge domain where I cannot do reasonableness checking on my own.

But just because the facts are correct, as you suggest, can their arrangement be misleading? Scientists, even encyclopedia editors can have their own axes to grind. At least the Wikipedia can be readily edited.

I will continue to use it, the same way I use the Web, with an ample dose of caution. Our schools do already teach that view, neither of my sons (in high school and university) had heard of the Wikipedia, though they are enthusiastic users of the web for encyclopedia type information. They consider the source of the information in each case ...

[mdog]

Presumably the self-selection of entry monitors for particular issues, who then get emailed when pages of interest to them change and are thus able to fact check, is of some use towards these ends.

[Franz]

You could establish a watch group for a set of pages, even construct technical means to make sure you knew whenever a page is changed and have the ability to respond ... I have worked with Wiki software (Socialtext) that can do this. But then its just who is willing to outlast a contrary opinion.

[Alex]

I think the ability to edit/update quickly is a virtue that shouldn't be overestimated. Traditional encyclopedias are revised on an annual basis, and even the electronic ones tend to have long editorial pipelines, dedicated but overworked staffs, and a traditional of thoughtfulness and care (or just eccentric attention to minutiae). Under those conditions, things don't get updated quickly.

The question, I think, is whether there are elements of the Wikipedia model that you can port over into more conventional encyclopedias, giving you a balance between the speed and volume of wikipedia and the authority and consistency of Britannica.

I'm beginning to think that there are: one would be to give a defined group of contributors free reign over their subjects, and let them revise at will. This would essentially multiply the number of editor-authors, while putting some controls on who writes.