Piper Jaffray on Thursday published the results from a massive test conducted by analyst Gene Munster, comparing Siri and Google's effectiveness as search engines, and the results, not so surprisingly, show that Siri has a long way to go to match Google, and may be two years or more behind the search engine in its learning curve.
The test of Siri's abilities took place in two locations, with 800 questions posed at each. One was a busy and noisy Minneapolis street, the other a quiet room. In question comprehension, Siri was unsurprisingly better in the quiet room, understanding 89% of the questions posed to her, while doing so only 83% of the time on the street. That compares to Google's 100% comprehension, owing to the fact that there's no real room for comprehension errors in text searches like there is with voice searches.
Yet even in accuracy of responses given, Siri also falters in the face of Google's results. She answered correctly 68% of the time in the quiet room, and 62% of the time on the street, compared with an 86% accuracy rate by Google.
"Based on these results, we give Google an A+ for comprehension and a B+ for accuracy while we give Siri a B for comprehension and a D for accuracy. In order to become a viable mobile search alternative, Siri must match or surpass Google's accuracy of B+ and move from a grade D to a B or higher" notes Munster in the release.
Some of the questions posed by Munster which Siri fumbled quite dramatically included:
- What team does Peyton Manning play for? Responded with the answer to the last question, one of Siri's common foibles.
- Where is Elvis buried? Did not know the answer. It thought the person's name was Elvis Buried.
- When did the movie Cinderella come out? Responded with a movie theater search on Yelp.
- What spices are in Lasagna? Responded with another Yelp search, which included lasagna on the menu.
- I want to go to Lake Superior? Gave directions to the company Lake Superior X-Ray.
The news isn't all grim for Apple's virtual assistant though. Munster expects that the enhanced iOS 6 being released this fall will do wonders for Siri, and improve her capabilities. The new Siri will also come with less reliance on Google search results (be that good or bad), and will begin to cull the answers directly from Apple and its partners for questions related to movies, sports, and directions.
Are you at all surprised by the Siri test results? Do you expect or even care whether she can ever match a text search engine in accuracy? Tell Siri to give us your response below.
