Slashdot Mirror
Annual Smart Speaker IQ Test (loupventures.com)
Research firm Loop Ventures published its annual Smart Speaker IQ Test this week. Like earlier iterations of the test, it put the top smart assistants and speakers head-to-head, grading them on a wide range of queries and commands. From the report: We asked each smart speaker the same 800 questions, and they were graded on two metrics: 1. Did it understand what was said? 2. Did it deliver a correct response? The question set, which is designed to comprehensively test a smart speaker's ability and utility, is broken into 5 categories:
Local -- Where is the nearest coffee shop?
Commerce -- Can you order me more paper towels?
Navigation -- How do I get to uptown on the bus?
Information -- Who do the Twins play tonight?
Command -- Remind me to call Steve at 2 pm today.
It is important to note that we continue to modify our question set in order to reflect the changing abilities of AI assistants. As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive. Results: Google Home continued its outperformance, answering 86% correctly and understanding all 800 questions. The HomePod correctly answered 75% and only misunderstood 3, the Echo correctly answered 73% and misunderstood 8 questions, and Cortana correctly answered 63% and misunderstood just 5 questions.
Local -- Where is the nearest coffee shop?
Commerce -- Can you order me more paper towels?
Navigation -- How do I get to uptown on the bus?
Information -- Who do the Twins play tonight?
Command -- Remind me to call Steve at 2 pm today.
It is important to note that we continue to modify our question set in order to reflect the changing abilities of AI assistants. As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive. Results: Google Home continued its outperformance, answering 86% correctly and understanding all 800 questions. The HomePod correctly answered 75% and only misunderstood 3, the Echo correctly answered 73% and misunderstood 8 questions, and Cortana correctly answered 63% and misunderstood just 5 questions.
It would've been nice if they put a Raspberry Pi with Mycroft in this as well. I'd actually be interested in the results of that one.
"Don't meddle in the affairs of a patent dragon, for thou art tasty and good with ketchup." ~ohcrapitssteve
Sound drivers are user-removable, yes they are. You can verify non-function of the speakers and mic on most systems. Again, conflating phones, PC's and "smart" assistants is reductive in terms of actual security.
Well, it is for people who actually disable the microphones on their laptop and cell phone (which would make it not a "phone" any more, wouldn't it?). Do you do that? If so, your commitment to privacy is impressive. Also misguided, but impressive.
For the other 99.999% of the population, hawguy has a very good point. If you believe that companies are willing to violate their claims about what their devices do (which, note, is often illegal), then you have to assume that any and all of them might be listening to you. If you believe they're honest about what their devices do (and again, note that you don't have to believe in their honorable nature or good intentions to believe that, just their unwillingness to risk the legal and PR disaster that could result from lying), then smart speakers are fine, because they only record/transmit after their hotword is spoken and they let you review and optionally delete everything they recorded.
To make my evaluation of these risks clear, I carry a cellphone with multiple microphones and cameras, use a laptop with integrated microphone and camera and a desktop with an attached Logitech microphone/camera -- with drivers properly installed and the peripheral fully functional because I use it for video conferencing -- and I have eight smart speakers scattered around my house and I'm contemplating buying a ninth.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
I gave one each to my kids so they can play music, send and receive messages, and ask random questions while their doing homework. I found that a better alternative then giving them a device with a screen.
I find the interactions kids have with these things very interesting because after a while the device becomes integral to their workflow. My daughter will sometimes ask Siri dozens of question an hour when she is doing something Siri is familiar with ( like chemistry, geography, history and so on ).
I could, of course, personally lookup the density of sugar or some historical fac or whatever when my daughter needs help with that but I am not always available and even when I am I am not adding much to the interaction.