IFTF's Future Now

Webscience

via Langreiter,

a commentary at the Journal of Neuroscience speculates on the future of research journals and peer-review:

Research studies appear on databases, not in journals
First, I don't think that it makes any sense to continue with paper copies of research articles. Instead of the "quasi-legal" document that is the current scientific article, we should be moving to full data being available on the web together with the software that might have been used to manipulate the data, as well as multimedia presentations to back up the data. Research papers are primarily of interest to other researchers in the same area, and they usually don't need the introduction and certainly not the discussion, which mostly degenerates to hype anyway
...
If an absence of peer review (or post-publication review, as I call it) is a step too far, then we should have an author (or rather funder) pays model. These fees could support a peer review mechanism, which should be open in that both authors and readers would know who was reviewing studies. It's ethically unacceptable that such important judgements should be made an unidentified judge. Like it or not, we live in a world where what is not transparent is deemed to be biased, corrupt, or incompetent until proved otherwise. Plus I believe that peer review should be a scientific discourse rather than an arbitrary judgment. This is far from radical: it's simply science returning to its roots when science was presented and discussed at meetings rather than published in journals.

Perhaps we will invent new forms of peer review by learning from innovations like Wikipedia. It is in some ways a form of peer review, only reviewers make changes directly rather than simply commenting.
...
Another worry from the conservative about such a system is to wonder how credit would be allocated. At present credit comes from publishing in prestigious journals. Often the impact factor of the journal (a dubious and manipulated statistic) is allocated to the paper, which is wholly unscientific because there is little correlation between the citations to studies and the impact factor of the journals in which they are published, because the impact factor of a journal is driven by a small number of highly cited studies (Seglen, 1997). In the new world I'm imagining, credit would come from the buzz from researchers and hits on the study. These hits can be disclosed in real time, unlike citations, which come years after studies are published.

The part about multimedia presentations makes a lot of sense as we expect simulations to play an increasingly important role.  I'm eager to see researchers experiment with new ways of publishing and finding each others work. I met someone from Georgia Tech who had setup some kind of display to notify researchers when they were working on portions of the genome that overlap with someone in the building.  It's impossible to know what will work or not beforehand - what would encourage research and what would bring the whole system to a screeching halt. For more webscience speculation, look at these Machine Learning (Theory) posts:

9/18/2006 - What is missing for online collaborative research?
12/1/2005 - The Webscience Future

Experts Rate Wikipedia Higher Than Non Experts

from arstechnica,

A new Wikipedia study concludes that experts rate an article in their field of study more highly than do non-experts.

This less-than-intuitive finding is the conclusion of a study in which Chesney had 55 graduate students and research assistants examine one Wikipedia article apiece. Each participant was randomly placed into one of two groups: group one read articles that were in their field of study, while group two read randomly-assigned articles. Respondents were asked to identify any errors that they found.

Those in the expert group ranked their articles as generally credible, higher than those evaluated by the non-experts. Chesney admits that this is unexpected, but has a possible explanation: "It may be the case that non-experts are more cynical about information outside of their field and the difference comes from a natural reaction to rate unfamiliar articles as being less credible."

Collaborative research, Wikipedia or elsewhere

There is a post at the Machine Learning (Theory) blog which discusses online collaborative research and how the right system could take the lead with well-designed mechanisms for credit and controversy.  There was a previous post on this blog December 2005 that noticed the progress year over year of the Machine Learning page at Wikipedia, suggesting a possible future:

Wikipedia becomes better to the point where it is a completely comprehensive listing of current research in machine learning. At some point, we-the-community realize this and begin to emphasize (and credit) information placed in wikipedia.

The more recent post at MLT lists necessary ingredients for an online collaboration system for scientific research, highlighting dicussion, credit, and disagreement mechanisms.

There does exist a discussion framework on Wikipedia/MLpedia, but the presentation format marginalizes discussion, placed on a separate page and generally not viewed by most observers. The discussion, in fact, should be an integral part of the presentation.
...
Wikipedia intentionally anonymizes contributors in the presentation, because recognizing them might invite the wrong sort of contributor. Incentives done well, however, are one of the things creating [community buy-in]. One of the existing constraints within academia is that the basic unit of credit is coauthorship on a peer-reviewed paper. Given this constraint, it would be very handy if a system could automatically translate a subset of an online site into a paper, with authorship automatically summarized.

A comment by Aaron Hertzmann suggests how online collaboration could make life easier for people doing research.  He breaks down the function of a discussion page for papers new and old:

With more recent papers, we want to have discussion in order to understand the work and its significance, and to hear what other people think of the paper. Often, the most interesting aspects come out later in discussions with colleagues, and its a matter of luck as to whether these discussions happen (and whether you happen to be present). With older papers, its really useful to find out the history and signficance of the paper — the oral tradition associated with the paper — which is otherwise not written down. The authors (or other implementers) often have hindsights about the work which are not written in the paper.

Also related and good to read: (via BoingBoing) Clay Shirky's recent remarks on experts, Wikipedia and Larry Sanger's Citizendium.

Lanier on Collective Intelligence

Jaron Lanier, well known promoter of the idea of virtual reality, talks about the 'hive culture' and wikipedia. A provocative view worth reading. Nicholas Carr calls it a collectivism fetish and comments on the article. Lanier discusses the idea of 'collective intelligence' and its uses and abuses. A very thought-provoking piece.

Amazon ProductWikis Re-Emerge

I previously observed that Amazon had started to include product wikis on many offerings. These can be modified by the consumer to reflect their thoughts on a product. Productwikis then vanished from Amazon, so it appeared this was some sort of selective test. They are back again, on a number of products. Previously it was hard to find any edited Wikis, but now they have a most edited link. Still unclear how different this is from their linear comments, which are more blog-like. Should manufacturers start by providing fundamental information? What kind of wiki war might ensue? Still unclear, too, if all Amazon users will see ProductWikis. Also, Amazon is starting to host author blogs.

Open source and the production of knowledge

Bob McHenry, a former editor in chief at Encycloapedia Britannica and my former boss, has an article in Tech Central Station on Wikipedia (discussed previously here and here).

To put the Wikipedia method in its simplest terms:

1. Anyone, irrespective of expertise in or even familiarity with the topic, can submit an article and it will be published.

2. Anyone, irrespective of expertise in or even familiarity with the topic, can edit that article, and the modifications will stand until further modified.

Then comes the crucial and entirely faith-based step:

3. Some unspecified quasi-Darwinian process will assure that those writings and editings by contributors of greatest expertise will survive; articles will eventually reach a steady state that corresponds to the highest degree of accuracy.

Does someone actually believe this? Evidently so. Why? It's very hard to say. One possibility that occurs to me is this: The combination of prolificacy and inattention to accuracy that characterizes this process is highly suggestive of the modern pedagogic technique known as "journaling."... It may well be that the practice of journaling in the schools, along with the acceptance of "creative spelling" as a form of personal expression not to be repressed, underlies much of the success of Wikipedia.

Superimpose on this intellectual preparation the moist and modish notion of "community" and some vague notions about information "wanting" to be free, et voila!

Leaving aside the question of whether journaling is responsible for faith in Wikipedia, Bob's piece does point out a significant issue about Wikipedia: the lack of peer review. Leave aside the absence of editorial standards or consistency (which are good things, but in my experience most readers don't really notice them), or the lack of systematic fact-checking. As a number of authors have pointed out, successful open source projects incorporate peer review mechanisms, and often operate in environments in which the criterial for measuring success and failure are understood by all participants. Code either works or it doesn't.

Fuzzy content, in contrast, is a different story. Here, even if you get all the basic facts right, experts can argue over how the facts fit together, or which facts matter. The assumption that contributors share a common understanding of what defines a decent article on a subject, what constitutes important facts, and what sources can be trusted to back up claims, which can be used to guide the growth of Wikipedia, may not be warranted. If the open source model is to be extended from software development into areas where debate over ground rules and epistemology are regular affairs-- as in history and encyclopedia writing-- this problem will need to be solved.

And Wikipedia Bias?

I have been a big proponent of the cooperatively updated Wikipedia. I use it often and have used it in my posts here and elsewhere to provide definitions and background. Most of what I refer to is technical stuff, where strong agendas would not be expected. But once ideological agendas are in play all bets are off.

Here is an example posted by a knowledgable lawyer, on the term 'Patriot Act' that illustrates just such an example. In the case of this politically charged definition he believes that there is no way that such legal postings, with their potential political leverage, could be maintained as objective, unless you just posted the dry text of the laws involved. That is why legaleze exists, and even that is argued over. Since its been publicized the post has been modified a number of times. Which is correct? Is this where Wikipedia-like efforts will require editors to provide some semblance of objectivity?

Even in the realm of science, opinion creeps in ... recent articles I have researched in the history of computing and flight have clear debatable opinion in their Wikipedia articles. Here, too, you could make the case that Britannica also has the bias of their authors and editors, and making a fix (or distortion) in paper encyclopedias is necessarily slower.

And even in the area of business information, there is a point of view. In the case of (just for example) Procter & Gamble''s entry there is a bland, dated description of the company, some old financial information about performance and articles on controversies about the corporate logo and clashes with PETA. These are interesting things, in the sense that they are sensational, in the tabloid sense, but hardly balanced in a view of the company.

Wikipedia Examined

I continue to watch with amazement the evolution of the user developed encyclopedia, the Wikipedia. It is really a remarkable achievement, now over a million articles in dozens of languages, developed and maintained by its own users. I use it often now and have learned much from it. Is this just the start of a global brain, beginning with simple lists of definitions? The next steps could evolve from semantic Web research. Will we then have a global brain that links knowledge with intelligence? There has also been considerable discussion on the Wikipedia's accuracy and comparisons to paper and search resources. I just read an essay by Taran Rampersad in WorldChanging that provides some useful further insight. Ross Mayfield also comments.

Educause on Wikis

A very good article on Wikis in Educause Review. It provides an overview, and discusses a number of the usual objections to the medium. Although this is mostly about academic uses, much of it is universal. Lots of links to related information. Recommended.

Blog and Wiki for SiGGraph

Ross Mayfield informs us that he has set up a test of a Wiki and a Blog covering this week's SiGGraph Conference in Los Angeles. Nice idea. The Wiki is open for editing. A comparison of the two forms of communication, though not with the same content, and as you would expect for a graphics conference, there are lots of pix. It also speaks to how much information you can get remotely from these approaches. I have attended this conference in the past, and will compare my remote experience this year with actually being there.