Bug lets humans grab Daily Double as Watson triumphs on Jeopardy
By Casey Johnston

Note: In this article, Jeopardy's "answers" are referred to as "questions" and vice versa.

The humans tried to hold on in the second game of Jeopardy against the IBM computer, but ultimately were no match. Watson finished with a two-game total of $77,147 to Ken Jennings' $24,000 and Brad Rutter's $21,400. Jennings and Rutter managed to make a larger dent in Watson's progress in the second game, but the computer managed to take both Daily Doubles away from the human contestants, not affording them enough of an opportunity to make up for Watson's $25,000 lead from the first game. Still, there were a few aspects of the game that gave the humans some ins, including a bug that let Ken Jennings score the first Daily Double.

During a panel at Rensselaer Polytechnic Institute's Experimental Media and Performing Arts Center, Dr Chris Welty, a member of Watson's algorithms team, noted that the start-and-stop nature of filming the episode got Watson mixed up and allowed a bug to surface. Watson begins every round looking for Daily Double clues, because they are crucial to progress in the game. After one filming pause in the first round when Watson had been made to stop and then pick up again, Welty said Watson began again thinking the Daily Double had already been found. So it stopped looking for the clue, allowing Jennings to find it first.

"They were having a lot of problems in that particular round and they kept stopping," Welty said. "There was still a Daily Double left in that round, and the front end that keeps track of the game state had thought the Daily Double was already revealed." Because Watson thought the Daily Double was gone, it started working its secondary strategy of selecting the lowest level clues to allow it to learn about a category. This left Jennings free to sort through the remaining higher value clues where the Daily Double was, allowing him to pick it up while Watson was cherry picking the top rows.

Another of Watson's biggest weaknesses was laid bare by a category from the first round, "Actors Who Direct." The questions in the topic were shorter than standard clues, usually only the names of two movies pointing to one man, and didn't give enough time for Watson to process and hit the buzzer first. "The answers were not ready in time because the questions were so quick," said Chris Welty. "One of the things that Watson actually doesn't know is that it's losing the buzzer because its answers aren't ready."

Not only was this bad from a score standpoint, but it formed a vicious circle for Watson's clue selection. Welty pointed out that Watson will select clues from categories based on where it's getting responses correct, which it was in the case of Actors Who Direct, but Watson doesn't get any information on whether its right answers are actually allowing it to buzz in first and get the points."It's going to keep going back because it's getting all the right answers," Welty said.

Aside from issues of timing, Watson's algorithms worked well in the sense that it was very rarely certain of a wrong answer. On answers it was certain of, it nearly always beat Jennings and Rutter to the buzzer; if the answer didn't turn up a high-confidence response, as was often the case with subtly worded questions, Watson would remain silent.

That's not to say there weren't outliers—Watson was occasionally unsure of answers that were correct. For example, in a Daily Double question on art from the first game, Watson came up with the correct answer, Baghdad, but with only 32 percent confidence. And as happened with the infamous Final Jeopardy question from the first game, Watson seems to struggle with the relationship that categories can have to a correct response. In the topic "On the Keyboard" during the second game, the clue "A loose-fitting dress hanging straight from the shoulders to below the waist," prompted Watson to ask "What is a chemise?" The correct response was the dress shape and keyboard key "shift."

But in regular Jeopardy rounds, Watson was able to learn during the game based on previous answers in the category what type of answer was required. For example, in the first Jeopardy game, Watson eventually figured out—albeit a bit late—that the "Name that Decade" category did, in fact, want a decade as the answer. Even Watson's handlers were impressed: "It actually kind of figured out on its own that decades were important," Dr. Adam Lally, a senior software engineer from IBM, said. "It kind of in a way had figured out on its own how to solve this problem."

Towards the end of the panel, Welty and Lally were prompted to discuss the choice of gender for Watson's voice, which is currently of the smooth, genial male variety. "We did experiment a lot with female voice as well," Welty said. "But the speech software we had, the way you could change the settings of the voice, and I mean this in the best possible way, it just was not possible to get a female voice that wasn't a little bit grating." This drew sounds of ire from the crowd, but Welty added that having the voice operate in lower ranges made it easier to soften, and that both men and women on the development team preferred the male voice.

Watson's machine learning may come in handy in the future that its creators are envisioning for it, which include medical diagnoses and tech support. Of course, phone or voice input is currently out the question, as parsing sounds isn't something Watson can currently do. But with text input, Watson could be able to do great things from an information standpoint, especially given that it is able to find high-level connections between tiny details.

As a result of Watson's two-game win, 100 percent of its prize money, $1 million, will be donated to charity. Jennings and Rutter walk away with $300,000 and $200,000, respectively, and each is donating half of their prize to a charity of his choice.

Computerworld - While Watson certainly impressed the nation with its sweeping victory on the game show Jeopardy last week, the medical community -- which IBM hopes will be first to use the technology -- may eventually become even more impressed with its affordability.

After showcasing Watson's ability to ingest Jeopardy questions and spit out near real-time answers, IBM is now preparing the supercomputer for a full-time gig as a data analytics engine for the medical community.

IBM announced this week it is working with speech and imaging recognition software provider Nuance Communications to produce a system that can help physicians and other healthcare professionals cull through gigabytes or terabytes of patient healthcare information to determine how to best treat illnesses.

"Combining our analytics expertise with the experience and technology of Nuance, we can transform the way that healthcare professionals accomplish everyday tasks by enabling them to work smarter and more efficiently," said John E. Kelly III, senior vice president and director of IBM Research. "This initiative demonstrates how we plan to apply Watson's capabilities into new areas, such as healthcare with Nuance."

For example, a doctor treating a patient could use Watson's analytics technology, in conjunction with Nuance's voice and clinical language understanding software, to rapidly consider all the related texts, reference materials, prior cases, and latest knowledge in journals and medical literature. This could help medical professionals confidently determine the best options for diagnosis and treatment.

IBM is working with Dr. Eliot Siegel, professor and vice chairman of the University of Maryland School of Medicine's department of diagnostic radiology, to bring that Watson project to fruition in the healthcare industry.

Siegel told Computerworld that patient information tends to be written in free form by physicians, who use abbreviations and short-text explanations. So it could take well over 10 minutes to an hour for another physician, radiologist or specialist to understand the intricacies of a patient's malady.

Now multiply one person's medical record by the thousands that a hospital or medical group might have, and the difficulty in finding best practices from healthcare trends becomes even more daunting.

Enter Watson.

IBM hopes that in about two years, Watson can be tweaked and go commercial to help hospitals and physicians take data from electronic health records (EHRs) and churn it into predictive modeling to determine the most likely outcomes from various treatments.

While there are many hurdles to achieving that goal -- such as the continuing lack of widespread EHR deployment -- Watson could one day save untold dollars and lives, IBM hopes.

Cost shouldn't be a significant factor as Watson is relatively cheap compared to medical technology routinely purchased by healthcare organizations.

The Watson supercomputer that appeared on Jeopardy last week was made up of 90 IBM Power 750 Express servers powered by 8-core processors -- four in each machine for a total of 32 processors per machine. The servers are virtualized using a Kernel-based Virtual Machine (KVM) implementation, creating a server cluster with a total processing capacity of 80 teraflops. A teraflop is one trillion operations per second.

According to Tony Pearson, master inventor and senior consultant at IBM, a Power 750 server retails for $34,500. Thus the 90 that make up Watson would cost about $3 million.

"That's not bad. You're going to spend $3 million on an MRI machine," Pearson said. "If you look at how expensive hospital equipment is, cost is not the issue."

A hospital, or even physician's office, doesn't necessarily have to buy the full clustered Watson computer system. The original compute algorithm single threaded on a single core processor took two hours to scan memory and produce an answer to a question. IBM technologists just added 2,880 CPUs, which produced the ability to answer the Jeopardy questions in three-seconds.

If a hospital or physician is willing to wait 30 seconds for an answer, then you'd only need one-tenth of that compute power or nine machines.

"So you're in the $300,000 range," Pearson said. "It's quite possible [to wait two hours] if you run it on your Power 750 at home. I'd bet there are some people who'd say, 'heck, I can't even get my doctor to call me back in two hours.' I think it's reasonable that larger hospital systems will have the bigger machines and smaller hospitals might settle for waiting a little longer for an answer."

"Watson seems amazing, but I'm not sure how it can take all that unstructured data and process it," said Marc Probst, CIO of Intermountain Healthcare in Salt Lake City, which services close to half the population of Utah.

Probst said he's skeptical because Watson's structured database is very dissimilar to the unstructured data in an EHR format.

"I don't know how well Watson works with Nuance, but there's so much detailed data in healthcare," Probst said. "It's known that the human mind can process 7 to 9 data items and consistently. The average clinical decision application uses over 40 data items to make a decision. So, there's much more data involved."

Last week, Intermountain Healthcare opened a 10,000-square-foot informatics research center supported by two data centers. Intermountain's Homer Warner Center for Informatics Research staffs 65 physicians and PhDs charged with providing decision support functions to clinicians, as well as provide input on the best possible care options.

For example, several years ago the Journal of the American Medical Association (JAMA) published the findings of a study that showed a correlation between the number of babies that wind up on ventilators in neonatal intensive care units (NIC) and at what point physicians induced labor.

"Using the practices and technologies in place at the Homer-Warner Center, we were able to change behavior. It had a dramatic impact on the health of babies," he said. "We were at about 30% of births induced prior to 39 weeks." Now, he said, "about 3%" are induced that early.

Probst said reducing the number of babies in NIC units saves "millions and millions" of dollars per hospital in his system. "We've got hundreds of such examples," he said.
http://www.computerworld.com/s/artic...mputerworld%29