Team Human vs Team Watson : Round I

Just in case watching a computer play humans on Jeopardy wasn't your idea of a romantic evening last night, here's my summary of what happened.

They spent a lot of time explaining the way Watson was built and even gave a high level discussion of the fact that the system maintains a belief distribution over possible answers.  When the computer answers a question we see its top three picks with the probability weight on each answer and the threshold for buzzing in.

The first round was impressive with Watson dominating the first few minutes.  It answered quickly and flawlessly until the first commercial break. It seemed that the humans just weren't fast enough.

But after the commercials it got interesting. Ken Jennings seemed to modify his strategy to simply press the button as early as possible.  It was clear he buzzed in several times having no idea what the answer was, then stalled for a moment and guessed, usually correctly.  This was a smart adaptation to Watson's algorithm.  The computer won't answer unless it is confident in its answer. But a human can keep thinking after they buzz in and gamble that they can come up with something.

Interestingly when Ken buzzed in very early the answers showing on the screen for Watson seemed to be lower quality, it still hadn't converged on a good answer and froze at the buzzer.

At one point Ken buzzed in and got it wrong, then Watson buzzed in. We could see that its top answer was the same wrong answer Ken just gave, but it repeated it anyways.  This seems to indicate the algorithm does not keep computing after the buzzer and can't take into account the answers of other players. A minor thing to change really.

The round ended with Watson answering the last question, correctly identifying the Event Horizon of a black hole. Fitting I think.

We'll see what happens tonight.

What do you think about the challenge, leave your observations in the comments. If you are on twitter make sure to pick a side: Are you rooting for #teamhuman or #teamwatson?