‘Smart Speaker IQ Test’ Shows That Siri is Getting Better at Answering Questions

BY Evan Selleck

Published 20 Dec 2018

The digital personal assistants out there are already pretty helpful, but the companies behind these services are always working to improve them.

Which is good news for the end user, because that means, gradually, the digital personal assistant they use will get better over time. These types of improvements can be small in nature, or it can mean we see a noticeable boost to what the software can recognize from a query or command, and then give us a response we’re actually looking for. It’s always so frustrating when one o these features can’t do something, so always learning and improving is a good thing.

And it turns out that Siri is getting better! At least, that’s the conclusion that Gene Munster’s Loup Ventures has concluded in its latest annual “Smart Speaker IQ Test”. Munster himself, along with Will Thompson, conducted the test which saw the biggest digital personal assistants out there asked a series of 800 questions. All the same questions for each service so that the tests could have fairer results.

For Amazon’s Alexa, an Amazon Echo (2nd Gen) was used. Siri was queries on a HomePod, Google Assistant on a Google Home Mini, and Microsoft’s Cortana Harmon Kardon Invoke. Several different smart devices were used, too, including: “Wemo Mini Smart Plug, TP-Link Kasa Plug, Phillips Hue Lights, and Wemo Dimmer Switch”.

The test breaks things down by a lot of categories. But, the results show a pretty stark lead by Google Assistant when it comes to answering questions correctly. According to the results, Assistant answered 87.9% of questions correctly. Siri, in second place (if that’s how you want to look at it), answered 74.6 percent correctly. In third and fourth place, respectively, Alexa answered 72.5% correctly and Cortana answered 63.4% correctly.

The ability for the digital personal assistants to understand the query were much closer, even if Assistant once again has a demanding lead with a 100% results. Meanwhile, Siri, Alexa, and Cortana all earned 99.6%, 99.0%, and 99.4%, respectively.

Munster and Thompson break down the methodology behind the IQ test, making it as simple and straightforward as possible, while asking the same 800 questions across the services. Here’s how they explain it:

    “We asked each smart speaker the same 800 questions, and they were graded on two metrics: 1. Did it understand what was said? 2. Did it deliver a correct response? The question set, which is designed to comprehensively test a smart speaker’s ability and utility, is broken into 5 categories:

  • Local – Where is the nearest coffee shop?
  • Commerce – Can you order me more paper towels?
  • Navigation – How do I get to uptown on the bus?
  • Information – Who do the Twins play tonight?
  • Command – Remind me to call Steve at 2 pm today.

Interestingly, while Siri did manage to keep up with Assistant in a variety of different categories, there are two areas to make a note of moving forward. First: Information. Siri simply does the worst in this category by a noticeable margin, answering only 66% of queries correctly. (Compare that to Assistant’s 94%, Alexa’s, 91%, and even Cortana’s 75%.) That’s definitely not good enough, even for a passive device in your home that’s not really offered up as a smart speaker. Siri is on the HomePod and people are going to ask it questions, it needs to do better.

(And this score is indicative only of the HomePod, mind you.)

However, there is good news! Because for a passive device that’s not billed as a smart speaker, the HomePod handles commands better than the competition by a significant margin. The HomePod handled 85% of commands correctly, while the Assistant managed only 74%, Alexa handled 68%, and Cortana managed 65%. Interestingly, Loup Ventures has a theory for that, and it comes back to the iPhone:

“HomePod’s lead in this category may come from the fact that the HomePod will pass on full SiriKit requests like those regarding messaging, lists, and basically anything other than music to the iOS device paired to the speaker. Siri on iPhone has deep integration with email, calendar, messaging, and other areas of focus in our Command category. Our question set also contains a fair amount of music-related queries, which HomePod specializes in.”

The full IQ test is worth looking at, and you can find it through the source link below.

If you own a HomePod, are you happy with your usage of Siri on the device?

[via Loup Ventures]