Chance of Wasting Time Piloting Prediction Markets? 100%
I’ve been asked to participate in a “prediction markets” pilot to help inform decision making regarding IT in higher education. Say what? Well, the idea is cool, in theory. We have a simulated market with each trader given “funny money” and a web interface to the markets and our performance dashboard. Folks make various propositions and then we lay our money down, see how the pool changes, sell, buy, etc. Like an eBay for ideas.
Unfortunately, many of the propositions are poorly stated, i.e. participants don’t necessarily “get it,” and the concept is genuinely hard to understand. Here are examples of what I would say are two poorly formed propositions: “Our curriculum is our biggest problem” and “Likelihood that over the next 3 years technology will be instrumental to the success of collaboration with external partners.” The first is rather difficult to “prove” and the second is such a no-brainer that I’m not sure of the point of contention. I mean, isn’t it instrumental NOW?
The second concern is that, as the CTO, can’t I influence the outcome? I mean, isn’t this like insider trading? In fact, isn’t that what insider trading is all about? Maybe you don’t want the CTO to bet (funny money or no) on whether or not technology is “instrumental”? What’s the point again?
Perhaps I’m the one who doesn’t get it.
bendy said:
Jun 12, 08 at 6:52 amYeah, I thought the efficacy of prediction markets was that it drew upon people who had no vested interest in the outcome, other than “profiting”; just as one might use Linux and Postgres, but invest their retirement in Microsoft and Oracle.
So to work, topic of the market has to have some sort of general interest. Higher Ed IT Planning doesn’t strike me as a big water cooler topic.
“How ’bout dem Blackboards? I think they’re gonna run the tables. Made them Moodles look like a buncha amateurs!”
As an alternative, you could suggest a Higher Ed IT Management fantasy league.
cec said:
Jun 12, 08 at 8:17 ambendy – I love the idea of a higher ed IT Management fantasy league! you could print up little baseball cards w/ a manager’s picture on the front, stats on the back. You could explore the idea of trading (U Chicago’s CIO) to CMU or see how he would do at MIT. It would be excellent.
admin – a good chunk of the work i’m doing now is using information markets as a source of probabilities. whoever put yours together and decided on the questions seems to have no idea what they are doing. you are exactly right. questions in an information market should be easily understood and fairly concrete. less “our curriculum is our biggest problem.” and more, “project X will be completed on time.”
fwiw, we’re currently using something called market scoring rules in which participants directly alter probabilities (instead of buying shares). the scoring rules make more sense for our purposes and seem to work better when you don’t have many (less than 40?) active participants.
oh, one other thing. to ensure proper incentives, the money needs to be more than just funny money. at the very least, prizes or time off credits. give each participant a pool of time off credit and let them actually win anything they have in their portfolio above the initial pool.
jezebel said:
Jun 12, 08 at 8:43 amScrew it all and get down to playing poker — I mean get down to the work: http://www.planningpoker.com/
I have adopted “Tra la la, fuck it/them” as my personal motto.
admin said:
Jun 12, 08 at 4:38 pmJez: planning poker! This is great. It might be a fun to do at least once. CEC: can you give me more info about market scoring rules, e.g. an url? I saw this: http://blog.oddhead.com/2006/10/30/implementing-hansons-market-maker/ but the math formulas seem, um, scary. But poker, that’s fun!
The IT Mgmt fantasy league is awesome. Or we could make it like Magic cards. Or maybe like those “most wanted” cards the military dropped all over Iraq?
essprit said:
Jun 12, 08 at 5:05 pmLove the IT management fantasy league idea. I wonder how you’d deal with insider info, though. Or maybe you just wouldn’t care. Hey, maybe it could even lead to helping that favorite (not) IT manager in one’s own institution a better job somewhere else, so they could be someone else’s problem, er, opportunity?
cec said:
Jun 13, 08 at 6:35 amadmin: let me see if I can dig up a URL. The math may look a little scary, but that’s mostly because you can’t get published if something looks simple. Imagine a market where you’ve got two participants and a simple yes/no question. The initial probabilities are 50% for yes and 50% for no.
Each of us is given a certain amount of money for each answer, say $100.
Each of us can adjust the probabilities. We gain money when we increase the probability and we pay to decrease a probability. The amount we gain or lose is proportional to the logarithm of the change. So, (approximately) a change from 50% to 55% might earn as much as a change from 90% to 99%.
We are each limited by how much we crank down the probabilities. Since a decrease costs money and we can’t go below $0.
If you and I disagree on a question, we are going to continue to adjust the probabilities until we’re out of cash (in either the yes or no answer) and we’ll have to come to a compromise (in a sense). If there are more than 2 of us, then we can capture the overall sense of the group.
The advantage to the scoring rules is that a lack of market participants doesn’t “starve” the market for buyers or sellers.
admin said:
Jun 13, 08 at 1:31 pmThanks for helping me understand, c. I now feel that I have a 75% chance of eventually comprehending this but for now I’m ignoring your phrase “the logarithm of the change” because it’s just not helping… 🙂