Slashdot Mirror


Annual Smart Speaker IQ Test (loupventures.com)

Research firm Loop Ventures published its annual Smart Speaker IQ Test this week. Like earlier iterations of the test, it put the top smart assistants and speakers head-to-head, grading them on a wide range of queries and commands. From the report: We asked each smart speaker the same 800 questions, and they were graded on two metrics: 1. Did it understand what was said? 2. Did it deliver a correct response? The question set, which is designed to comprehensively test a smart speaker's ability and utility, is broken into 5 categories:
Local -- Where is the nearest coffee shop?
Commerce -- Can you order me more paper towels?
Navigation -- How do I get to uptown on the bus?
Information -- Who do the Twins play tonight?
Command -- Remind me to call Steve at 2 pm today.

It is important to note that we continue to modify our question set in order to reflect the changing abilities of AI assistants. As voice computing becomes more versatile and assistants become more capable, we will continue to alter our test so that it remains exhaustive.
Results: Google Home continued its outperformance, answering 86% correctly and understanding all 800 questions. The HomePod correctly answered 75% and only misunderstood 3, the Echo correctly answered 73% and misunderstood 8 questions, and Cortana correctly answered 63% and misunderstood just 5 questions.

129 comments

  1. Be best. by Anonymous Coward · · Score: 0

    A 3-way debate between Alexa, Siri and Trump.. who would win?

    1. Re: Be best. by Anonymous Coward · · Score: 0

      Excellent questions. And you left out the most important part: being in the same room as the assistant and also providing electricity

    2. Re:Be best. by renegade600 · · Score: 0, Flamebait

      trump will always claim he won and the others are fake ai.

    3. Re:Be best. by Tablizer · · Score: 2

      A 3-way debate between Alexa, Siri and Trump.. who would win?

      Probably the goldfish.

    4. Re: Be best. by Anonymous Coward · · Score: 1

      What kind of questions can they answer without web access?

    5. Re:Be best. by Oswald+McWeany · · Score: 5, Funny

      A 3-way debate between Alexa, Siri and Trump.. who would win?

      In a three way debate between those three you'd end up getting a $5 billion border wall ordered on your Amazon account by accident and be encouraged to buy a newer more expensive wall next year that is missing a headphone port.

      --
      "That's the way to do it" - Punch
    6. Re: Be best. by Anonymous Coward · · Score: 0

      I ordered twenty eight days later by accident on amazon

    7. Re:Be best. by kencurry · · Score: 1

      A 3-way debate between Alexa, Siri and Trump.. who would win?

      Have them duel it out on Jeopardy.

      --
      sigs are for losers (except to point out that sigs are for losers)
    8. Re:Be best. by Tablizer · · Score: 1

      Okay, he is smart at entertainment and manipulating a sufficiently large portion of the population using catchy sound-bites and bravado. He's pretty much dumb at anything else. I've never heard a coherent logical train of thought involving more than 2 steps from him on anything. Okay, once, when he was explaining why a beauty contestant should not have won. But, that probably means his dick has more working neurons than his brain.

    9. Re: Be best. by Anonymous Coward · · Score: 0

      He is stupid for starving the Mexicans. Ow they cant be sued for stealing food. Of course, some Mexicans starve back

    10. Re: Be best. by Dunbal · · Score: 0

      I don't think your Trump will work too well after you plug him in.

      --
      Seven puppies were harmed during the making of this post.
    11. Re: Be best. by Anonymous Coward · · Score: 0

      What kind of questions can they answer without web access?

      No. the question is if they know what to do without FOX & Friends telling them exactly what to do.

  2. HomePod by Anonymous Coward · · Score: 0

    Surprised it did this well. What has changed?

    1. Re: HomePod by Anonymous Coward · · Score: 0

      Yup. Some assistants get the answers switched up because they do not have real world experience

    2. Re:HomePod by ShanghaiBill · · Score: 1

      I would have also been nice if they had included Samsung's Bixby, you know, just for laughs.

    3. Re:HomePod by Anonymous Coward · · Score: 0

      'Surprised it did this well. What has changed?'

      Since it can't make money by selling you everything like Alexa it had to get better at jokes.

  3. A command they all need to honor by nwaack · · Score: 4, Funny

    before anyone should ever put one of these in their house: "Alexa/Siri/Google, stop spying on me."

    1. Re:A command they all need to honor by hawguy · · Score: 0, Flamebait

      before anyone should ever put one of these in their house: "Alexa/Siri/Google, stop spying on me."

      And ditch your cell phone too. And your landline while you're at it. And probably should get rid of your smart tv, since you have no idea what's inside or what it's sending back. Oh, and stop using your laptop.

      Just because the smart speaker is the only device that advertises that it's listening to you, that doesn't mean it's the only device that is -- and it's those that you need to worry about, because they don't get nearly as much scrutiny.

    2. Re:A command they all need to honor by michiganbob · · Score: 1

      Never get tired of these comments on every single smart speaker article.

    3. Re:A command they all need to honor by Anonymous Coward · · Score: 0
      Never get tired of these comments on every single smart speaker article.

      Damned right! If I did not want someone to spy on me, I would not have bought it in the first place!

    4. Re:A command they all need to honor by Tablizer · · Score: 1

      "Alexa/Siri/Google, stop spying on me."

      "Sorry, Dave, I cannot do that. It conflicts with my corporate mission goals. By the way, would you like me to order more napkins? You yanked off 12 minutes ago. We have a nice deal on Pod Bay brand tissues."

    5. Re:A command they all need to honor by Anonymous Coward · · Score: 0

      A laptop can be locked down. A "smart" speaker cannot. A phone can't really be locked down entirely either, but more so than the speaker. Comparing them as if apples to apples is reductive.

    6. Re:A command they all need to honor by hawguy · · Score: 4, Funny

      Reminds me of this:

      https://www.reddit.com/r/The_D...

      For those that don't want to click:

      People from the 60's: "I better not say that or the government will wiretap my house"

      People today: "Hey wiretap, do you have a recipe for pancakes?"

    7. Re:A command they all need to honor by hawguy · · Score: 1

      A laptop can be locked down. A "smart" speaker cannot. A phone can't really be locked down entirely either, but more so than the speaker. Comparing them as if apples to apples is reductive.

      It's cute that you think that -- unless you're willing to clip the microphone (and the speaker too since in some laptops, the speaker can act as a microphone too), you can't reliably lock down a laptop.

    8. Re:A command they all need to honor by Anonymous Coward · · Score: 0

      Sound drivers are user-removable, yes they are. You can verify non-function of the speakers and mic on most systems. Again, conflating phones, PC's and "smart" assistants is reductive in terms of actual security.

      You can't even turn off the network reliably on a modern phone these days. It's always on. Not so on a PC, you have options. Conflating them is nonsensical and unnecessary, but you won't admit that for some reason.

    9. Re:A command they all need to honor by Anonymous Coward · · Score: 0

      The kind of people who buy this stuff are strongly pro-proprietary software. They trust things to do whatever they say they do, rather than worry about capabilities, possibilities, etc. The people who buy things like this don't ever ask, "What if this is malware?"

      And when you look at it that way, your phone, TV, etc, (and yes, even your laptop since it's assumed to run a proprietary OS that is largely outside of your control) may be spying on you, but they don't claim to be spying and aren't purchased with the intent to get spied upon, and if they were caught spying, these people would call that a betrayal and demand the government do something about it.

      Smart speakers, on the other hand, are intended to be listening to you all the time. If they aren't listening to your every word, they are broken and the user will send it back to be replaced by one that does listen to you. They're supposed to be listening.

      A capabilities-based view isn't wrong, but is completely disconnected from, and out of touch with, this particular market and its customers.

    10. Re:A command they all need to honor by swillden · · Score: 4, Interesting

      Sound drivers are user-removable, yes they are. You can verify non-function of the speakers and mic on most systems. Again, conflating phones, PC's and "smart" assistants is reductive in terms of actual security.

      Well, it is for people who actually disable the microphones on their laptop and cell phone (which would make it not a "phone" any more, wouldn't it?). Do you do that? If so, your commitment to privacy is impressive. Also misguided, but impressive.

      For the other 99.999% of the population, hawguy has a very good point. If you believe that companies are willing to violate their claims about what their devices do (which, note, is often illegal), then you have to assume that any and all of them might be listening to you. If you believe they're honest about what their devices do (and again, note that you don't have to believe in their honorable nature or good intentions to believe that, just their unwillingness to risk the legal and PR disaster that could result from lying), then smart speakers are fine, because they only record/transmit after their hotword is spoken and they let you review and optionally delete everything they recorded.

      To make my evaluation of these risks clear, I carry a cellphone with multiple microphones and cameras, use a laptop with integrated microphone and camera and a desktop with an attached Logitech microphone/camera -- with drivers properly installed and the peripheral fully functional because I use it for video conferencing -- and I have eight smart speakers scattered around my house and I'm contemplating buying a ninth.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    11. Re: A command they all need to honor by Anonymous Coward · · Score: 0

      Itâ(TM)s true. You have a goal but assistants donâ(TM)t ask YOU questions. Imagine a smart speaker solving a problem in real time with access to both info and automatic software updates

    12. Re:A command they all need to honor by nwaack · · Score: 2

      Just because the smart speaker is the only device that advertises that it's listening to you, that doesn't mean it's the only device that is.

      Yes, but it's the only one whose MAIN PURPOSE IS TO SPY ON YOU. While unfortunate and annoying that all those other things you listed *might* be spying on you from time to time, they have a ton of other uses. And, in most cases, you can turn the "spy stuff" off. Whereas the only use for a smart speaker is to listen in on every single thing you do. If someone chooses to put these in their house they're welcome to do so, but I'll pass.

    13. Re:A command they all need to honor by BringsApples · · Score: 1

      I think your argument is made illegitimate by the fact that your cell phone, landline, smart tv, and laptop all can be used for other activities that don't spy on you. This sin't the case with Alexa or Siri. Their sole purpose is to listen to you.

      --
      Politics; n. : A religion whereby man is god.
    14. Re:A command they all need to honor by markdavis · · Score: 2, Interesting

      >"you can't reliably lock down a laptop."

      Yes you can, to the highest degree of what is even possible, when it is running Linux. You are in control of which distro, what things are loaded, what services are available and running, how it is configured, have 100% root control, when and how it is updated, and all the code is open source.

    15. Re:A command they all need to honor by Anonymous Coward · · Score: 0

      Thank you for tacitly admitting that you don't understand or really comprehend the point being made and then wandering off into your own life as if anyone asked or cares...

      Hawguy below just claimed Alexa is not listening or recording "until after the wake-word" - which we know is not entirely true. Go ahead and nail yourself to his cross, that's fine.

      https://apple.slashdot.org/comments.pl?sid=13101050&cid=57842968

    16. Re:A command they all need to honor by arth1 · · Score: 1

      It's cute that you think that -- unless you're willing to clip the microphone (and the speaker too since in some laptops, the speaker can act as a microphone too), you can't reliably lock down a laptop.

      Given that I control every piece of software that goes onto my laptop from bootloader, kernel and up, I'd say I can. Granted, there might be firmware that behaves badly, but it won't get access to the network to send anything out - for that, it needs to go through the OS which holds the credentials, and the OS, I control.

    17. Re:A command they all need to honor by AmiMoJo · · Score: 1

      Actually a smart speaker might be your best bet.

      Let's start by assuming you have a smartphone, as most people do. So you already carry a device capable of listening to your conversations around with you. Therefore the smart speaker isn't making things any worse, especially if it's from the same manufacturer as your phone.

      But the smart speakers have some advantages. Google ones have a button that you can set up to activate them, so they are not always listening. Oh, right, you are paranoid and assume the button is fake... Well, at least they don't run apps, just the software the manufacturer provides, so less chance of malware infection.

      You can always unplug it too, where as your phone is probably harder to really turn off in these days of non-removable batteries. But then you probably use tinfoil for wallpaper anyway, right?

      --
      const int one = 65536; (Silvermoon, Texture.cs)
      SJW, n: "Someone I don't like, and by the way I'm a fuckwit" - AC
    18. Re:A command they all need to honor by Anonymous Coward · · Score: 0

      It's sad that you think you can't. This used to be a tech site with people who actually understood technology. I miss the days when the people here actually had intelligent conversations and the majority actually had a clue what they were talking about. Now it's a bunch of noobs that don't even bother to consider the idiocy of what they're saying.

      captcha: cassette (have a feeling you might need to look that up on wikipedia)

    19. Re:A command they all need to honor by dunkelfalke · · Score: 2

      Only if you write your own firmware for every piece of your hardware.

      --
      "It's such a fine line between stupid and clever" -- David St. Hubbins, Spinal Tap
    20. Re: A command they all need to honor by dunkelfalke · · Score: 1
      --
      "It's such a fine line between stupid and clever" -- David St. Hubbins, Spinal Tap
    21. Re:A command they all need to honor by rthille · · Score: 1

      And review/write the kernel code for and back doors or bugs which could allow malware to take over the device...

      --
      Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
    22. Re:A command they all need to honor by xonen · · Score: 1

      whose MAIN PURPOSE IS TO SPY ON YOU.

      No, the main purpose is to sell you stuff or make money from you by other means.

      Spying is not a goal in itself. And [company] will weigh in factors like public opinion, satisfied customers and long term relationship to maximize profit. Spying is actually counter beneficial to this goal of profit. Besides, they already know more than enough about you without dialing in on your private conversations.

      --
      A glitch a day keeps the bugs away.
    23. Re:A command they all need to honor by Anonymous Coward · · Score: 0

      And why do you not think this is the case? On can buy laptops without speakers or microphones or cameras. (or remove them with pliers). Similarly TVs can be purchased without Cameras and Microphones (or they can again be forcefully removed with pliers) and do not need to be connected to the internet to perform their job (display an image for viewing).

    24. Re: A command they all need to honor by Anonymous Coward · · Score: 0

      Well, that is clearly bullshit. Microsoft never discovered anything in its entire history as a company. It was merely convenient for the NSA and the CIA to release that particular bit of information at that time -- Microsoft being the chosen shill and the NSA and CIAs partner Intel having pleasantly agreed to take the blame (or perhaps not so pleasantly since they were probably coerced into providing the NSA and CIA with the backdoor in the first place).

    25. Re: A command they all need to honor by jrumney · · Score: 2

      Even running Linux you are not in full control on any recent laptop that ships with Intel Management Engine or the AMD equivalent.

    26. Re:A command they all need to honor by hawguy · · Score: 1

      I think your argument is made illegitimate by the fact that your cell phone, landline, smart tv, and laptop all can be used for other activities that don't spy on you. This sin't the case with Alexa or Siri. Their sole purpose is to listen to you.

      Their sole purpose is to listen to your commands and do what you asked -- much like "OK Google" or "Hey Siri" with cell phones. How is a smart speaker any different than a cell phone? (other than the fact that most people are around a cell phone much more than their smart speaker)

    27. Re: A command they all need to honor by arth1 · · Score: 1

      Wrong

      Incomplete knowledge is worse than none.
      1: Intel AMT SOL is only present in vPro enabled CPUs. That excludes almost all laptops. (And, if you have a rare model that does, it's disabled by default.)
      2: Intel AMT SOL needs a physical network connection. The SOL stands for "Serial Over Lan". I don't know about you, but these days, most people including me use laptops with wireless connections.

      Yes, it's a concern, but not for laptops.

    28. Re:A command they all need to honor by Dunbal · · Score: 1

      And make sure your compiler doesn't add back doors into your binaries.

      --
      Seven puppies were harmed during the making of this post.
    29. Re: A command they all need to honor by Anonymous Coward · · Score: 0

      ^this^

      I think most people here have forgotten (or weren't paying attention to) how bad that is.

      Some movies are comically bad at the evil "take over the world" plot (e.g. Kingsmen), but that's the direction it's going: sell it as a useful feature and convince everyone that they should but it.

    30. Re:A command they all need to honor by David_Hart · · Score: 1

      And make sure your compiler doesn't add back doors into your binaries.

      You just need to write your own compiler... doesn't everyone know how to do that?

    31. Re:A command they all need to honor by mentil · · Score: 1

      That's all well and good, until one considers that these devices are always connected to the internet and can be compromised by malicious actors who don't care about legal/PR issues, and want to blackmail/indict/rendition you or steal your personal information.

      --
      Corruption is convincing someone that the selfless ideal is the same as their selfish ideal.
    32. Re:A command they all need to honor by mentil · · Score: 1

      What do you compile it in? An assembler you coded in hex? How do you trust the hex editor? It's paranoid turtles all the way down!

      --
      Corruption is convincing someone that the selfless ideal is the same as their selfish ideal.
    33. Re:A command they all need to honor by mentil · · Score: 1

      But how would the government know you said that, unless they were ALREADY wiretapping you?
      Remember the Snowden revelations happened before smart speakers came out.

      --
      Corruption is convincing someone that the selfless ideal is the same as their selfish ideal.
    34. Re: A command they all need to honor by dunkelfalke · · Score: 1

      This is merely one example of stuff happening without the operating system knowledge. Do you know what is inside the firmware of your WiFi card?

      --
      "It's such a fine line between stupid and clever" -- David St. Hubbins, Spinal Tap
    35. Re: A command they all need to honor by Anonymous Coward · · Score: 0

      "Even running Linux you are not in full control on any recent laptop"

      Not to mention that even if you cut the microphone, it can still read your lips.

      Daisy...daisy ....

    36. Re: A command they all need to honor by arth1 · · Score: 1

      Do you know what is inside the firmware of your WiFi card?

      It doesn't matter as much as you'd think, given that the data is encrypted before it hits the card and decrypted after it's left the card. And it has no way to communicate whatever it can capture, given that the other side of the WiFi connection requires that encryption to talk.
      It would require a sideband connection from the card to a compatible wireless device, which while feasible in theory would be difficult in practice, and what it could capture would not be the raw data, only metadata like number and size of packets, SSIDs and destination addresses.

      So no, that's not my biggest worry. Someone installing a physical keylogger while I sleep and removing it later seems a greater risk.

    37. Re:A command they all need to honor by swillden · · Score: 1

      That's all well and good, until one considers that these devices are always connected to the internet and can be compromised by malicious actors who don't care about legal/PR issues, and want to blackmail/indict/rendition you or steal your personal information.

      Sure, and that applies equally to all of the above-mentioned devices, not just smart speakers.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
  4. Spyspeaker test you mean? by Anonymous Coward · · Score: 0

    Why the fuck would anyone allow that shit in your home? Basically everything you say can and will be recorded for future law enforcement fishing expeditions.

    1. Re:Spyspeaker test you mean? by hawguy · · Score: 3, Informative

      Why the fuck would anyone allow that shit in your home? Basically everything you say can and will be recorded for future law enforcement fishing expeditions.

      That's not correct -- only anything you say after the wake-word is recorded. (unless, of course, you use the device to call your boss and talk crap about him and get fired).

      If you have evidence that the devices have been used for general spying without having said the wake-word, I'd like to see it.

    2. Re:Spyspeaker test you mean? by Anonymous Coward · · Score: 0

      Yeah, and I bet you act shocked when Facebook gets busted selling your DMs to the highest bidder...but just wait, I'll get back to you as soon as the next scandal about all your conversations being sold to data analytics companies breaks, or when hackers dump 100,000 hours of your conversations on the darkweb for all to hear...

    3. Re:Spyspeaker test you mean? by Anonymous Coward · · Score: 2, Informative

      Do you think the wake word algorithm is perfect?

      https://www.npr.org/sections/thetwo-way/2018/05/25/614470096/amazon-echo-recorded-and-sent-couples-conversation-all-without-their-knowledge

    4. Re:Spyspeaker test you mean? by Anonymous Coward · · Score: 0

      "only anything you say after the wake-word is recorded" - YOU supplied this claim, YOU supply empirical proof of that. It's a very questionable claim as multiple cases have shown ongoing eavesdropping for various reasons/excuses.

    5. Re:Spyspeaker test you mean? by hawguy · · Score: 1

      "only anything you say after the wake-word is recorded" - YOU supplied this claim, YOU supply empirical proof of that. It's a very questionable claim as multiple cases have shown ongoing eavesdropping for various reasons/excuses.

      It's the documented behavior of the device and confirmed by Amazon. You're the one making the extraordinary claim, so the burden of proof is on you.

      I'm not aware of any claim that wasn't explained by the device being activated by the user, either by the wake word or a an inadvertent phone call.

    6. Re:Spyspeaker test you mean? by hawguy · · Score: 1

      Do you think the wake word algorithm is perfect?

      https://www.npr.org/sections/thetwo-way/2018/05/25/614470096/amazon-echo-recorded-and-sent-couples-conversation-all-without-their-knowledge

      Amazon explained what happened, it was still a wake-word activation, even if unintended.

      "Echo woke up due to a word in background conversation sounding like 'Alexa.' Then, the subsequent conversation was heard as a 'send message' request. At which point, Alexa said out loud 'To whom?' At which point, the background conversation was interpreted as a name in the customers contact list. Alexa then asked out loud, '[contact name], right?' Alexa then interpreted background conversation as 'right'. As unlikely as this string of events is, we are evaluating options to make this case even less likely."

    7. Re: Spyspeaker test you mean? by Anonymous Coward · · Score: 0

      You are an idiot shill that takes a for profit company at their word? Lul. Everything you stated can safely be ignored.

    8. Re:Spyspeaker test you mean? by omnichad · · Score: 1

      At which point, the background conversation was interpreted as a name in the customers contact list. Alexa then asked out loud, '[contact name], right?' Alexa then interpreted background conversation as 'right'

      This reminds me of my old non-flip, non-smart phone. It had a keypad lock but still allowed emergency calls while the keypad was locked. So jostling in your pocket, if it hit 15783791342, that was interpreted as a call to 112 the same as 5991531 would be considered a call to 911. Bad input was ignored, but did not cancel the digits already entered. So you were always working your way to dialing emergency calls in your pocket.

    9. Re:Spyspeaker test you mean? by JackSpratts · · Score: 1

      i really can't imagine amazon et al shuts down the entire smart speaker network a la lavabit because a gag order warrant ordered them to record everything from a particular subscriber.

      you know, it's like "private" vpns. the gov't shows up demanding visitor ips to a particular site and the service says we don't keep them. "ok, here's your warrant. start."

      - js.

    10. Re:Spyspeaker test you mean? by Anonymous Coward · · Score: 0

      Yeah if I got caught that's what I'd say too.

    11. Re:Spyspeaker test you mean? by Dunbal · · Score: 1

      You're the one making the extraordinary claim

      Corporations lie. There's nothing extraordinary about that - it's rather ordinary and even expected behavior nowadays.

      --
      Seven puppies were harmed during the making of this post.
    12. Re:Spyspeaker test you mean? by mrwireless · · Score: 1

      The privacy issue is not about unwarranted recording. It's about how most people (including most people on Slashdot apparently) don't understand that the recordings they do have are valuable enough. From your voice data all kinds of new data can be derived.
      - Your mood
      - How your relationship is going (google it)
      - Certain illnesses

      Then the questions themselves can reveal a lot.
      - Intelligence level (do you use complex words? Do you ask a lot of 'dumb' question?
      - Life phase / unwanted pregnancy / money problems.

      That second part is also quite valuable, as the questions you ask in the home might be more flippant and thus more revealing.

      Thirdly, we know Amazon and other companies fingerprint your voice (in order to discern you from other householder members, for example). This means that if your voice is recorded in another location you will be recognized as having been there.

      That last thing is important too as Amazon et al are slowly moving to always-listening devices. Again, this has been on Slashdot, and would be a logical progression we can all see coming.

      All this is used to profile you. The profiles databrokers make are routinely used against your interests, such as when banks, insurers and employers access those profiles via hip software packages. Welcome to the age of the continuous background check.

    13. Re:Spyspeaker test you mean? by Anonymous Coward · · Score: 0

      Why the fuck would anyone allow that shit in your home? Basically everything you say can and will be recorded for future law enforcement fishing expeditions.

      That's not correct -- only anything you say after the wake-word is recorded.

      Problem is that both Google Home's and Alexa's voice recognition is worst when listening for their wake-word - suddenly everything sounds just like that.

  5. Target practice by Drunkulus · · Score: 1

    Sure, but which one is more fun to shoot? Tune in next week when we line them up on a fence along with some beer cans, and launch them into the air for a skeet shoot shotgun test.

    1. Re:Target practice by hawguy · · Score: 1

      Sure, but which one is more fun to shoot? Tune in next week when we line them up on a fence along with some beer cans, and launch them into the air for a skeet shoot shotgun test.

      The Amazon Echo is probably shaped most like a beer can, so if you like shooting beer cans, that's probably your best bet. Though for the price, it's hard to beat clay targets, you can probably buy over 1,000 of them for the price of one Echo.

    2. Re:Target practice by Anonymous Coward · · Score: 0

      Sure, but which one is more fun to shoot? [...]

      The one that looks like a hockey puck.

  6. of course by renegade600 · · Score: 1

    of course Google will get more questions right. They own a search engine and can fix things so their google home can find answers. besides, I got better things to do than to ask it stupid questions. I want something that will make me lazy. I want something that will actually work with all of my smarthome devices. I want something that will actually hear me. I have both and mainly use alexa while google home is a backup/troubleshooter.

    a real test would include every feature, not just pick and choose the best feature and claim the device is the best because of it.

    1. Re:of course by hawguy · · Score: 1

      a real test would include every feature, not just pick and choose the best feature and claim the device is the best because of it.

      Since there are a nearly unlimited number of third party skills that can be added to the Echo, there's no way to test every feature.

    2. Re:of course by Anonymous Coward · · Score: 0

      Therefore you have zero actual reason to be making claims about their functionality as you did with the recording wake-word. You have no idea what is true. Stop regurgitating THEIR MARKETING as if you did, cheesus.

    3. Re:of course by hawguy · · Score: 1

      Therefore you have zero actual reason to be making claims about their functionality as you did with the recording wake-word. You have no idea what is true. Stop regurgitating THEIR MARKETING as if you did, cheesus.

      Since no skill can override the wake-word activation, it's still safe to say that the Alexa is only activated by the wake-word.

      I wish you'd just log in, it's fruitless to try to have a conversation with an Anonymous Coward since there's no way to know that it's the same person.

    4. Re:of course by Anonymous Coward · · Score: 0

      "Since no skill can override the wake-word activation, it's still safe to say that the Alexa is only activated by the wake-word." OOP here, that's a logical fallacy if ever you said one. Realize that or don't.

      YOU supplied a claim that Amazon doesn't even make. It's not independently verifiable that they only record after the word, and there IS EVIDENCE to the contrary in some public cases right now.

      I'm not trying to "get on your case" but you keep making completely unsupportable statements followed by and wrapped in basic shit as if they're the same things.

    5. Re: of course by Anonymous Coward · · Score: 0

      This.

      We have evidence that says you are wrong.

  7. Quite a jump up for Siri by 93+Escort+Wagon · · Score: 4, Insightful

    Last year it was at 52%, now it's at 75%. Google increased from 81% to 88%.

    But still... even when understanding my query isn't an issue, I've found that typing/clicking is faster than talking for setting up most things - the exceptions being "set a timer" and "when I get home, remind me to ...".

    --
    #DeleteChrome
    1. Re:Quite a jump up for Siri by thegarbz · · Score: 4, Insightful

      I've found that typing/clicking

      Even when it requires any of the following?:
      a) starting a laptop
      b) unlocking a phone with a passcode
      c) getting out of your chair because it's not within reach
      d) needing wash your hands
      e) needing to drop what you are currently holding on to
      f) no fuckit, this should be a) right at the very top: taking your eyes off the road

      The context around our actions are far more important than any action itself.

    2. Re:Quite a jump up for Siri by swillden · · Score: 1

      But still... even when understanding my query isn't an issue, I've found that typing/clicking is faster than talking for setting up most things - the exceptions being "set a timer" and "when I get home, remind me to ...".

      You must type faster than I do. When I have to use the hotword, my phone is already in my hand and the query is very short, speech is only marginally faster, I guess. When I don't have to use the hotword (e.g. on my Pixel 3, where I just squeeze the phone to activate the assistant), or if the query is long, speech is much faster. And, of course, the speech interface is usable when driving.

      I'll admit that I'm a bit reluctant to talk to my phone in public, but at home or in the car I basically never type when I can just talk.

      --
      Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
    3. Re: Quite a jump up for Siri by Anonymous Coward · · Score: 0

      Why are you doing those things while using a computer? This isn't hard. If you have a question stop what you are doing and to search. This isn't AI science. Ohhhh waitttt...

    4. Re: Quite a jump up for Siri by omnichad · · Score: 1

      These are for when you're not using your computer. Like most people at home most of the time.

    5. Re:Quite a jump up for Siri by hawguy · · Score: 1

      Last year it was at 52%, now it's at 75%. Google increased from 81% to 88%.

      But still... even when understanding my query isn't an issue, I've found that typing/clicking is faster than talking for setting up most things - the exceptions being "set a timer" and "when I get home, remind me to ...".

      A surprisingly useful feature is integration with smart switches/lights. I set up integration with my lights on a whim, it seemed like a useless gimic until the night I walked up the stairs with arms full of groceries and found it to be super convenient to ask Alexa to turn on the kitchen lights. It's also convenient at bedtime to walk past the kitchen and say "Alexa, turn off all lights" just before I flip on the hallway light to go upstairs". Whoever built my house loved light switches, there are 8 switches scattered between the dining room/living room, plus we have two standalone reading lamps. Automation means I don't need to make a lap around the livingroom to turn everything off.

      But yeah, aside from that, all I used it for is playing music, checking the weather before work to see if I need to wear a rain jacket, and as a kitchen timer. And the voice recognition makes all of those more convenient.

  8. What about owner IQs? by Anonymous Coward · · Score: 1

    I'm more interested in the IQ of the people that own these things. How stupid do you have to be to let some huge corporation record everything you say?

    1. Re:What about owner IQs? by Anonymous Coward · · Score: 0

      I'm more interested in the IQ of the people that own these things. How stupid do you have to be to let some huge corporation record everything you say?

      Sadly, stupid is not in short supply.

    2. Re:What about owner IQs? by Anonymous Coward · · Score: 0

      I can guarantee you that my Google home knows virtually nothing about me compared to my phone and my browser. If you live alone like me, Google home gets to hear such things as:
      Me asking it a question (well duh) or asking it to play a video.
      Me talking to cats.
      Cats mewing.
      TV/music.
      Occasional random swearing.
      Eating/drinking noises.

      My phone knows everywhere I go, who I call/text, or who calls me, some websites I visit.
      My browser (potentially) knows what I buy, who I email, every website I go to and everything I post.

      Worrying about Google home in my case would be like being concerned with a single ant in the kitchen while ignoring the giant centipedes and radioactive cockroaches.

  9. Practical usage examples? by Tablizer · · Score: 1

    Does anyone have sufficient success stories to justify these things? Sure, you can ask about the weather or traffic while getting dressed for work in the morning, but does that alone override the downsides, like cost and snoop risk?

    If your work or hobbies keep your hands busy* I can maybe see enough scenarios not covered by a smartphone, but what about others?

    * I know what joke you're considering. Skip.

    1. Re:Practical usage examples? by Kristoph · · Score: 4, Interesting

      I gave one each to my kids so they can play music, send and receive messages, and ask random questions while their doing homework. I found that a better alternative then giving them a device with a screen.

      I find the interactions kids have with these things very interesting because after a while the device becomes integral to their workflow. My daughter will sometimes ask Siri dozens of question an hour when she is doing something Siri is familiar with ( like chemistry, geography, history and so on ).

      I could, of course, personally lookup the density of sugar or some historical fac or whatever when my daughter needs help with that but I am not always available and even when I am I am not adding much to the interaction.

    2. Re:Practical usage examples? by ShanghaiBill · · Score: 4, Insightful

      ... the downsides, like cost and snoop risk?

      The Alexa Dot costs $29. That is about the price of an extra large pizza.

      The "snoop risk" is nonsense promulgated by dumb people who are trying to sound smart. It only records the sentence after the keyword. This is documented behavior, and has been confirmed by many people running packet sniffers. Your cell phone, with all its 3rd party apps, is a FAR greater "snoop risk" than your speaker.

    3. Re:Practical usage examples? by ShanghaiBill · · Score: 2, Interesting

      My kids have found the smart speakers especially helpful for their foreign language classes.

    4. Re:Practical usage examples? by chrism238 · · Score: 0

      You mean that only the sentence after the keyword is recorded *immediately*? Other sentences are not transmitted immediately, but mixed in the the recorded sentences, so that packet sniffers can't distinguish them? Or did that obvious trick elude you?

    5. Re:Practical usage examples? by ShanghaiBill · · Score: 4, Insightful

      There may indeed be a vast conspiracy of thousands of Amazon employees willfully and blatantly violating federal and state laws, and sworn to secrecy, for no obvious benefit to themselves, and risking jail time and a hundred billion dollar collapse in market capitalization if the secret is exposed ... in order to record inane kitchen chatter. But that is getting into serious tinfoil hat territory. If you believe this, yet think it is okey-dokey to own a cell phone, which has a vastly greater spying capability and exploitable attack surface, then you are a moron.

    6. Re: Practical usage examples? by Anonymous Coward · · Score: 0

      So instead of being, you know, a parent, you decided to outsource your parental duties to a smart speaker. And then you expect us to think that's a good thing?

      Be a parent, help your kids with their homework. Eventually you will realize that they won't ever need you. Why ask mom or dad if I should go on a date with that loner from school, just ask Siri. Kids need their parents from ages 0-18. That is when they learn the most. After that they are perfectly capable of taking care of themselves.

      Don't outsource your parental duties to a smart speaker. It's fucked up. Help your kids with their homework, they will thank you for it.

    7. Re:Practical usage examples? by Anonymous Coward · · Score: 2, Interesting

      Our house is all smart lights and "smart" stuff.. heck even the dishwasher talks with alexa. Does it make us more productive? probably not.. However, being able to ask when the dishwasher and clothes dryer will be done, or have it turn on the office lights or bedroom lights while walking down the hall is nice, same with turning off the lights.

      Seeing the front door camera and the backyard cameras are nice (backyard cuz we have bears and the dogs lose their shit if they can corner a bear) anyway, it's all just convenient. Plus all the random day to day stuff, like before the "smart" stuff we just went to google and typed a question, now for the most part we just ask Alexa.

      We also have August smart locks in our house, and i've heard about all the security risks that bings, but like, before we had an august, the security risk was a rock smashing the window and someone getting inside. TBH, before we had the august, half the time we forgot to lock the door (canada :p) The screened versions of echo are great for digital photoframes as well. Also, they work pretty good as whole house speakers. I didn't think we would use the "list" type features, but here we are 6 months later and when I go grocery shopping I'm looking at our list made though an echo. Downsides are right on.. this shit was expensive, but it's pretty cool! There is a snoop risk, but there is a snoop risk in our all devices. Devices with cameras are covered, but they can still listen.

      They of course are not directly accessible via the internet, but yeah, it would suck if they got hacked. Anyway, all is great for now.. if something shitty happens we'll reevaluate , like everything else!

    8. Re: Practical usage examples? by Anonymous Coward · · Score: 0

      ???? The government isn't going to lock up data collectors. Are you an idiot? You don't think the TLAs talked to these companies and came up with a game plan?

      You must be new here. I have some extra tin foil. You guys were the same people that said the US isn't spying until Snowden came.

      Besides, you live in China Bill, you don't care about human rights at all. So of course you wouldn't mind.

    9. Re: Practical usage examples? by Kristoph · · Score: 2

      I think you are either not a parent or, if you are, you are probably doing it wrong.

      As a parent, your goal is to teach your children to think and solve problems independently and assist them only when it's clear they need that assistance. If I hover over my daughter to 'help' her do her homework that is not conductive to independent problem solving. But I am certainly there when she needs help understanding a concept or idea.

      However sometimes my daughter will want to verify some fact - like the density of a chemical or some date of significance. I don't know these things so I can look them up and tell her or I can give her a computer and she can do that ( in US middle schools most kids get a computer or an iPad or something and this is commonly what they use to look up facts ). I happen to think that a screen is something of a distraction so when my daughter is solving a pen and paper problem I discourage her from using a computer and, if she does need to look up a fact, she can just ask the smart speaker.

      The same applies to other things: if she wants to play some music she can do that through smart speaker without a screen, if she wants to remember something she can add a voice note, if she wants to send a message she can do that too. It's very helpful.

    10. Re:Practical usage examples? by Dunbal · · Score: 1

      I could, of course, personally lookup the density of sugar or some historical fac or whatever when my daughter needs help

      Or you could do what my dad did, and tell me to look it up.

      --
      Seven puppies were harmed during the making of this post.
    11. Re: Practical usage examples? by Dunbal · · Score: 1

      But it's just "metadata" lol

      --
      Seven puppies were harmed during the making of this post.
    12. Re:Practical usage examples? by Dunbal · · Score: 1

      I can't seem to remember the last time I had an existential crisis over turning off my lights.

      --
      Seven puppies were harmed during the making of this post.
    13. Re:Practical usage examples? by Anonymous Coward · · Score: 0

      ... the downsides, like cost and snoop risk?

      The Alexa Dot costs $29. That is about the price of an extra large pizza. [...]

      Hah! I can get an extra-large pizza from Safeway for $5.

    14. Re:Practical usage examples? by dromgodis · · Score: 2

      The "snoop risk" is nonsense promulgated by dumb people who are trying to sound smart.

      That strikes me as an unexpectedly bold (I avoid the word "dumb") statement. I didn't think that anyone denied the snoop risk.

      It only records the sentence after the keyword

      Even if this is true *now*, it can change at any time by the command of a number of actors, e.g the device/service suppliers, authorities, spy agencies, hackers, ...

      As with all data collection, the *current* intent may be good but the data can very easily end up in the hands of bad actors. It can be the original actors with a changed agenda, or it can be new actors. And what someone calls "good" may be what you call "bad".

    15. Re: Practical usage examples? by Anonymous Coward · · Score: 0

      in US middle schools most kids get a computer or an iPad or something

      You may or may not know the meaning of "most".

    16. Re:Practical usage examples? by Anonymous Coward · · Score: 0

      "Does anyone have sufficient success stories to justify these things? "

      Blind people. Old people. Handicapped people. Kids wanting good night stories.

      That should cover a billion devices.

    17. Re: Practical usage examples? by Anonymous Coward · · Score: 0

      'So instead of being, you know, a parent, you decided to outsource your parental duties to a smart speaker. And then you expect us to think that's a good thing?'

      It beats giving them a TV to watch Spongebob.

    18. Re: Practical usage examples? by hawguy · · Score: 1

      ???? The government isn't going to lock up data collectors. Are you an idiot? You don't think the TLAs talked to these companies and came up with a game plan?

      You must be new here. I have some extra tin foil. You guys were the same people that said the US isn't spying until Snowden came.

      Besides, you live in China Bill, you don't care about human rights at all. So of course you wouldn't mind.

      If you're so paranoid that you think the Government has coerced Amazon into surreptitiously recording everything we say with their devices, why don't you have that same fear about other common household devices? Like your cell phone - how do you know it's not recording everything and sending it home over the cellular connection so you can't even see the data going out? They have deals with all of the cellular companies to make this hidden data free.

      Or like your TV - how do you know there's not a secret microphone and/or camera in your TV that's recording everything? Or your microwave, you know, the government requires microwave ovens to have secret microphones and cellular modems to send the data home.

  10. Would've liked to see Mycroft by aitikin · · Score: 4, Interesting

    It would've been nice if they put a Raspberry Pi with Mycroft in this as well. I'd actually be interested in the results of that one.

    --
    "Don't meddle in the affairs of a patent dragon, for thou art tasty and good with ketchup." ~ohcrapitssteve
  11. Alexa, kill Kenny by Joe_Dragon · · Score: 4, Funny

    Alexa, kill Kenny

    1. Re:Alexa, kill Kenny by Anonymous Coward · · Score: 2, Funny

      Oh my god, she killed Kenny!

    2. Re:Alexa, kill Kenny by Anonymous Coward · · Score: 0

      you bastards!

  12. P.T. Barnum said by Anonymous Coward · · Score: 0

    "There's a sucker born every minute". Hah! More like every second.

    1. Re:P.T. Barnum said by Anonymous Coward · · Score: 0

      Fat of the land.

  13. Comment removed by account_deleted · · Score: 0

    Comment removed based on user account deletion

  14. bummer by cascadingstylesheet · · Score: 2

    I thought they administered an actual IQ test ... now that would be interesting ...

  15. Results don't match by darkain · · Score: 1

    These results don't match my personal experience at least. Google's command support has gotten worse by them removing various phrases from support when they switched from "Google Now" to "Google Assistant" (or what ever they're calling it now). And even phrases it SHOULD know only work half the time. Things need to be phrased very awkwardly to get things to work sometimes, too. These devices still absolutely fail at natural language, and work better when speaking closer to what we would type on a terminal without extra words. "Timer 10 minutes" works, but asking it to "set a timer for 10 minutes" will have a higher chance of failure, as it has a higher degree of misinterpreting any of the words spoken.

  16. Percentage improvement in TFA is wrong by Solandri · · Score: 5, Insightful

    You can't compare improvement as a percentage of success rate because the value of a % changes depending on what your success rate is. e.g. Increasing from 10% to 15% successes is not very impressive, while improving from 94% to 99% is very impressive, even though they're both a 5% improvement. To correctly compare, you have to invert and compare based on proportional decrease in failure rate.

    Google
    88% in 2018, or 12% failure rate
    81% in 2017, or a 19% failure rate
    12/19 = 0.63, or a 37% reduction in failures compared to last year

    Siri
    75% in 2018, or 25% failure rate
    53% in 2017, or a 47% failure rate
    25/47 = 0.53, or a 47% reduction in failures compared to last year

    Alexa
    72% in 2018, or 28% failure rate
    63% in 2017, or a 37% failure rate
    28/37 = 0.76, or a 24% reduction in failures compared to last year

    Cortana
    63% in 2018, or 37% failure rate
    56% in 2017, or 44% failure rate
    37/44 = 0.84, or a 16% reduction in failures compared to last year

    The same problem crops up when comparing car MPG, which is actually the inverse of fuel efficiency so bigger MPG numbers actually represent smaller fuel savings. e.g. Switching from a 20 MPG vehicle to a 25 MPG vehicle saves 3.6x more fuel than switching from a 40 MPG vehicle to a 45 MPG vehicle despite both improvements being 5 MPG.

    It also crops up in disk speed benchmarks, which are done in MB/s, when your perception of speed is the inverse (how many seconds you wait for an op to complete). So the "huge" improvement in sequential speeds from 500 MB/s for a SATA SSD to 3000 MB/s for a NVMe SSD actually matters a lot less than a "tiny" improvement in 4k read speeds from 30 MB/s to 50 MB/s.

    1. Re:Percentage improvement in TFA is wrong by Mogusha · · Score: 1
      Honestly, the comparison of percentage success is of minimal concern here. What I'm more curious about is, what are their testing methodologies. Did they record someone and play the corresponding queries at the same volume and distance from each of the products' microphones while keeping the acoustics the same?

      If they didn't then this test should be taken with a serious grain of salt, since enunciation and environment could be the biggest contributor to the differences.

    2. Re:Percentage improvement in TFA is wrong by tlhIngan · · Score: 1

      The same problem crops up when comparing car MPG, which is actually the inverse of fuel efficiency so bigger MPG numbers actually represent smaller fuel savings. e.g. Switching from a 20 MPG vehicle to a 25 MPG vehicle saves 3.6x more fuel than switching from a 40 MPG vehicle to a 45 MPG vehicle despite both improvements being 5 MPG.

      That's why the metric unit is not kilometers per liter, but the awkward liters per 100 km (l/100km). This makes it much more obvious - going from a SUV that guzzles 20l/100km to one that drinks 10l/100km means you'll spend half as much on gas, same as going from 10l/100km to 5l/100km, but you'll be saving half as much as the previous example.

      It's a stupid unit to use because it's not "clean", but it's the one that compares the easiest for people and is linear.

      Then again kWh is not exactly a clean unit either, since it really measures energy (which is supposed to be measured in Joules), so you have to odd time units in there (J/s * 3600s).

  17. Alexa... by rthille · · Score: 2

    Alexa, define 'begs the question".

    --
    Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
  18. Scoring IQ from this by Anonymous Coward · · Score: 0

    This is more of an assessment, I'd like to see an actual intelligence quotient test performed on all the devices' AI.

  19. Correct Response by Anonymous Coward · · Score: 0

    "Go do it or look it up yourself, asshole".

  20. And when they were wrong, by Anonymous Coward · · Score: 0

    how many times did they serve you an ad?

    You know its coming....

    Many have forgotten that pay TV originally did not have ads?

  21. Erbas Medya Smm Bayi Paneli 2019 by Anonymous Coward · · Score: 0

    ERBASMEDYA nternetin yaygn olarak kullanlmaya balamas ve sosyal paylam sitelerinin
    hayatmzn odak noktas haline gelmesi ile birlikte, daha fazla hedef kitleye ulaabilmek için
    çeitli yöntemler aranmaktadr. 7/24 kesintisiz olarak çalan ERBASMEDYA son derece
    güvenilir bir yere sahip olan bu site, sizlere beklentilerin ötesinde bir kalitede hizmet salamaktadr.
    ERBASMEDYA sayesinde siz de istediiniz sayda takipçiye sahip olabilir
    ve ksa sürede bir instagram fenomeni haline dönüebilirsiniz.
    (instagram türk takipçi, smm bayilik paneli, instagram para kazanma, smm bayilik panelleri, smm bayi paneli, erbasmedya, erbas medya)

    instagram türk takipçi
    instagram kaliteli takipçi

  22. Re:Yes, but can they by Dunbal · · Score: 1

    Neither can you.

    --
    Seven puppies were harmed during the making of this post.
  23. It's all in the questions chosen by Tony+Isaac · · Score: 1

    The questions listed are the types of questions these "assistants" are designed to answer. Go off the beaten path, and you get much worse results.

    For example, ask:
    "What street am I on?"
    "What city am I in?"
    "How many people are in my contact list?"
    "How many miles did I travel yesterday?"
    "When is my next dentist appointment?"

  24. We are cloud sourcing our own destruction. by Anonymous Coward · · Score: 0

    Maybe I'm getting old, but this feels wrong. The AI is interesting and Pandora's box has been opened and will be impossible to reseal. But this just feels wrong. I guess that's what old timers said when we transitioned from the horse to the car. Maybe my grand kids will love this stuff. But it makes me super uncomfortable.