Slashdot Mirror
Annual Smart Speaker IQ Test (loupventures.com)
Research firm Loop Ventures published its annual Smart Speaker IQ Test this week. Like earlier iterations of the test, it put the top smart assistants and speakers head-to-head, grading them on a wide range of queries and commands. From the report: We asked each smart speaker the same 800 questions, and they were graded on two metrics: 1. Did it understand what was said? 2. Did it deliver a correct response? The question set, which is designed to comprehensively test a smart speaker's ability and utility, is broken into 5 categories:
Local -- Where is the nearest coffee shop?
Commerce -- Can you order me more paper towels?
Navigation -- How do I get to uptown on the bus?
Information -- Who do the Twins play tonight?
Command -- Remind me to call Steve at 2 pm today.
It is important to note that we continue to modify our question set in order to reflect the changing abilities of AI assistants. As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive. Results: Google Home continued its outperformance, answering 86% correctly and understanding all 800 questions. The HomePod correctly answered 75% and only misunderstood 3, the Echo correctly answered 73% and misunderstood 8 questions, and Cortana correctly answered 63% and misunderstood just 5 questions.
Local -- Where is the nearest coffee shop?
Commerce -- Can you order me more paper towels?
Navigation -- How do I get to uptown on the bus?
Information -- Who do the Twins play tonight?
Command -- Remind me to call Steve at 2 pm today.
It is important to note that we continue to modify our question set in order to reflect the changing abilities of AI assistants. As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive. Results: Google Home continued its outperformance, answering 86% correctly and understanding all 800 questions. The HomePod correctly answered 75% and only misunderstood 3, the Echo correctly answered 73% and misunderstood 8 questions, and Cortana correctly answered 63% and misunderstood just 5 questions.
It would've been nice if they put a Raspberry Pi with Mycroft in this as well. I'd actually be interested in the results of that one.
"Don't meddle in the affairs of a patent dragon, for thou art tasty and good with ketchup." ~ohcrapitssteve
Sound drivers are user-removable, yes they are. You can verify non-function of the speakers and mic on most systems. Again, conflating phones, PC's and "smart" assistants is reductive in terms of actual security.
Well, it is for people who actually disable the microphones on their laptop and cell phone (which would make it not a "phone" any more, wouldn't it?). Do you do that? If so, your commitment to privacy is impressive. Also misguided, but impressive.
For the other 99.999% of the population, hawguy has a very good point. If you believe that companies are willing to violate their claims about what their devices do (which, note, is often illegal), then you have to assume that any and all of them might be listening to you. If you believe they're honest about what their devices do (and again, note that you don't have to believe in their honorable nature or good intentions to believe that, just their unwillingness to risk the legal and PR disaster that could result from lying), then smart speakers are fine, because they only record/transmit after their hotword is spoken and they let you review and optionally delete everything they recorded.
To make my evaluation of these risks clear, I carry a cellphone with multiple microphones and cameras, use a laptop with integrated microphone and camera and a desktop with an attached Logitech microphone/camera -- with drivers properly installed and the peripheral fully functional because I use it for video conferencing -- and I have eight smart speakers scattered around my house and I'm contemplating buying a ninth.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
I gave one each to my kids so they can play music, send and receive messages, and ask random questions while their doing homework. I found that a better alternative then giving them a device with a screen.
I find the interactions kids have with these things very interesting because after a while the device becomes integral to their workflow. My daughter will sometimes ask Siri dozens of question an hour when she is doing something Siri is familiar with ( like chemistry, geography, history and so on ).
I could, of course, personally lookup the density of sugar or some historical fac or whatever when my daughter needs help with that but I am not always available and even when I am I am not adding much to the interaction.
>"you can't reliably lock down a laptop."
Yes you can, to the highest degree of what is even possible, when it is running Linux. You are in control of which distro, what things are loaded, what services are available and running, how it is configured, have 100% root control, when and how it is updated, and all the code is open source.
My kids have found the smart speakers especially helpful for their foreign language classes.
Our house is all smart lights and "smart" stuff.. heck even the dishwasher talks with alexa. Does it make us more productive? probably not.. However, being able to ask when the dishwasher and clothes dryer will be done, or have it turn on the office lights or bedroom lights while walking down the hall is nice, same with turning off the lights.
Seeing the front door camera and the backyard cameras are nice (backyard cuz we have bears and the dogs lose their shit if they can corner a bear) anyway, it's all just convenient. Plus all the random day to day stuff, like before the "smart" stuff we just went to google and typed a question, now for the most part we just ask Alexa.
We also have August smart locks in our house, and i've heard about all the security risks that bings, but like, before we had an august, the security risk was a rock smashing the window and someone getting inside. TBH, before we had the august, half the time we forgot to lock the door (canada :p) The screened versions of echo are great for digital photoframes as well. Also, they work pretty good as whole house speakers. I didn't think we would use the "list" type features, but here we are 6 months later and when I go grocery shopping I'm looking at our list made though an echo. Downsides are right on.. this shit was expensive, but it's pretty cool! There is a snoop risk, but there is a snoop risk in our all devices. Devices with cameras are covered, but they can still listen.
They of course are not directly accessible via the internet, but yeah, it would suck if they got hacked. Anyway, all is great for now.. if something shitty happens we'll reevaluate , like everything else!