Team Human vs. Team Watson : Round II

Since Watson is doing so well there has been some confusion about what is actually going on as we watch the game. There's been some talk that the Jeopardy challenge is unfair, and that's true it is, but both sides have some unfair advantages.  Here's how it is as far as I understand it.

Team Human Advantages (#teamhuman)

  • Using the most advanced computational system ever encountered, which has been under intense development for millions of years, the Human Brain. It has more raw computational power than Watson, can handle almost infinitely more parallelization and dynamic linking, is incredibly robust to new information and has pattern recognition heuristics which we are only barely beginning to comprehend. Its hard to underestimate how big an advantage this is and its hard to judge it, this is why it seems like a good problem for AI research.

  • They understand language - Watson does not understand language at all. It knows some things about language patterns and has learned how to match words and phrases for answers to other words and phrases which are questions for Jeopardy, and only Jeopardy. If a topic shows up which is has not seen much then it does not know what to do. Since it doesn't understand language it can't make the kind of leaps of reasoning that the humans can. This is why the topics are restricted to types of topics that have shown up on Jeopardy before, nothing that requires Trebek to explain the meaning of the question.

Watson Advantages (#teamwatson)

  • A huge memory database of facts which are relevant to jeopardy questions - its hard to say if this is more facts than Ken Jennings has in his head, it's represented very differently but humans have amazing heuristics for accessing data quickly and linking it together. But perhaps Watson has an edge here.

  • A totally focussed system designed and optimized for years just to play Jeopardy (against most of us this is an advantage, against Ken Jennings and ... its questionable who has spent more time training for these games)

  • Question sent as a text file - this really could be a bit of an advantage, the computer still needs to scan the text, parse it and analyze it.  The humans need to analyse the visuals and simultaneously listen to Alex Trebek describe the question, of course humans are very good at that kind of thing, Watson is not.

  • No video questions - ya, that's just not on, you want to see a machine fail at something? I'll show you my toaster trying to cook lasagna, we're just not there yet.

  • Button pressing - it's not clear exactly how Watson's button actuator works. And how does Watson know it is allowed to press the button if its not listening to Trebek? It's possible it has an 'unfair' fast reaction time to the humans, I don't know.

So while this is a fascinating challenge and should demonstrate to everyone how far Artificial Intelligence research has come, everyone should keep in mind that Watson is not Data or Skynet, in fact it's not even Wall-E.

Watson has been trained to play Jeopardy. It has the ability to answer questions and find data in a much more natural manner than was possible even 5 years ago. But it is not playing the same game that Ken Jennings and Brad Rutter are. Maybe next year.

My advice for next year's challenge

Oh you know there will be one, Jeopardy ratings are through the roof! The engineers at IBM should make efforts to remove these complaints of unfairness in the following ways:

  • (1) Let us see Watson's button - Ahem, you know what I mean. Put a robot out there or something more visual to let Watson press the button. Also, do some work with people who understand the human body to make sure it isn't unfairly fast. How long does it take for a human being to physically press a button from the moment they 'decide' to press it? It may seem like a handicap to add this delay to Watson but it really would seem more fair. After all, we want to know that Watson is winning on the questioning answering part of the game, not these physical details, so remove them as issues.

  • (2) Give Watson a camera - Watson really could visually parse the questions to know what they are. At the beginning of the round it would scan the topics visually and build a database to start planning.  This shouldn't be hard as visual text analysis is quite advanced, it wasn't added because it was a needless complication of an engineering problem. But the optics, excuse the pun, are not good in terms of fairness.

  • (3a) Get rid of all talking - Lock each player in a room, they wouldn't hear Alex Trebek, they'll all just read the questions.  When another contestant answers this should be sent via text file to all the other players. Then it would be fair...but boring and weird.

  • OR

  • (3b) Give Watson speech recognition - This will be a real problem as speech recognition is one of those areas that really has turned out to be harder than anyone imagined. Vision? No Problem. Text analysis? Give me enough text and I will move the Earth. Robotic control? Are you kidding? Easy. But understanding human speech transmitted through a vibrating air column, damn that's hard.

    But they should do it, they could use the latest technology available for this, which IBM is generally recognized to lead anyways, and just let it be the machine's achilles heal.  It could train on Alex Trebek's Canadian accent (no French Alex!) It could train on its opponents and it could nail the common and simple prompts that the host gives when it is it someone's turn to play or to break for commercials. It could just ignore Trebek reading the question out except for the cue that it is time to press the button. It may still not do much better at taking advantage of  wrong answers by opponents and it would likely make some entertaining mistakes, but it would make it more realistic. Paradoxically it might get more credit for losing in this way than it will in winning the way it is currently set up.

So if humanity loses tonight, don't fret. This machine is just a step along the way and sometimes things aren't as smart as they first seem. Then again, you could also say its holding by not even trying to do everything at once. Would you be more scared/impressed if Watson really did everything its human opponents did and still faired well?

On to round three.








1 comment:

  1. [...] you need some ego to be an intense competitor like Jennings, but I think he has a point. As I pointed out yesterday the quick reaction time between making the decision to buzz and registering a button press is [...]