Prediction Markets vs. Delphi: Old and New Methods of Forecasting Go Head-to-Head in German Study

Over the last few days, I've been corresponding with a young researcher named Andreas Graefe at the German Institute for Technology Assessment and Systems Analysis who is working on how to use prediction markets for participative political decision-making. As a testbed of his methods, he has launched a prediction market called TechForX which aims at predicting long-term developments in the media industry.

At first I wasn't that impressed - there are lots of these media prediction markets popping up recently, such as MSNBC's iPredict and Touchstone Books' MediaPredict. And O'Reilly and Yahoo!'s BuzzGame has been around for some time now.

What caught my attention is that Andreas will compare the results to a 200-person Delphi survey that is running in parallel to the prediction market. Ever since I started looking into prediction markets last year as part of IFTF's Open Ex Forum, I began thinking about the similarities to the Delphi method. As Andreas project describes:

While in the Delphi a small number of selected experts are asked to judge predefined theses, it is the goal of TechForX to enhance traditional foresight activities by involving a preferably broad public. Thus, TechForX invites every internet user to participate by revealing his / her opinion on a subset of the same theses that are asked in the Delphi.

Ironically, though, there's little know about the relationship between experts and non-experts within markets and prediction markets especially. Though theoretically "the wisdom of the crowds" comes from aggregating -all- of the groups knowledge about what will determine future events, not everyone contributes equally - some people have more information to give than others.

So if this project finds that Delphi is just as good as prediction markets, it will tell us two things. First, prediction markets aren't so much about crowds as they are about a place to convene experts. Second, IFTF abandoned Delphi decades ago due to the time- and labor-intensive process of mailing out and collating survey results on paper. We might have to re-think that decision in the age of instant electronic communication!

I encourage you to check out the market at http://www.techforx.org. Here are some of the futures being posed:

• More money is spent on internet advertising in search engines and navigation tools than on television advertising
• On average, users spend more time watching personalized TV on demand than fixed programs.
• A value added tax (VAT) for transactions within virtual worlds (like Second Life) is raised.
• Online self-publication of books (without involving publishing houses) is the predominant way of commercial distribution, even for established authors.
• Virtual visits to museums over the internet are more popular than physical visits.

Comments

[David Scott Lewis]

I seriously wonder whether these public prediction markets will be dominated by hyperenthusiastic early adopters who don't at all represent the public at large nor give any indication as to adoption rates.

Podcasting, video podcasting (as it relates to video podcast feeds)and the Twitter Jihadists come to mind. Remember when Scoble was reading over 1,000 blogs a day? No longer. Remember when Rubel was touting Twitter, but not he's no longer on the receiving end? Remember how Rice was touting video podcasts when in fact video podcast feeds that are done by the "We, the Media" crowd have almost no subscribers? But these are the guys that will likely skew the public forums.

Some may claim they have foresight. Some may claim they live in an alternate reality. (Funny that a lot of them are active in Second Life.) They're not the norm; they're statistical outliers. They're needed to push forward new technologies, to open discussion, but they don't represent the masses. Yet, I suspect that they'll skew the public prediction markets.

[Alex Forshaw]

I don't care what the academic studies say; play-money prediction "markets" are ludicrous. So are "internal" prediction markets, which destroy 90% of utility by shutting outsiders out of the process of information arbitrage. These concepts are high on buzz and negative on actual utility. And thanks to insane US regulations, real-money PMs aren't going anywhere as far as the United States is concerned.

It was very unfortunate that play-money prediction markets got so much academic sanction early on. As a result, the immense utility of REAL-money PMs has been effectively blown off by people who think they can get the forecasting power of real-money markets and the zero regulation of play money. It's really sad.

[Øyvind]

Alex:

A large body of evidence (empirical studies) clearly show that incentives do not matter for increasing prediction accuracy.

This means that companies do not have to be restricted by government regulations. They could just use play money instead of real money.

So your claim "negative on actual utility" is false.