27 Jan Siri, Alexa and Other Virtual Assistants Put to the Test
When I asked Alexa earlier this week who was playing in the Super Bowl, she responded, somewhat monotonously, “Super Bowl 49’s winner is New England Patriots.”
“Come on, that’s last year’s Super Bowl,” I said. “Even I can do better than that.”
At the time, I was actually alone in my living room. I was talking to the virtual companion inside Amazon’s wireless speaker, Echo, which was released last June. Known as Alexa, she has gained raves from Silicon Valley’s tech-obsessed digerati and has become one of the newest members of the virtual assistants club.
All the so-called Frightful Five tech behemoths — Apple, Microsoft, Amazon, Facebook and Alphabet’s Google — now offer virtual assistants, which handle tedious tasks in response to voice commands or keystrokes, on various devices. Apple’s Siri is the best known, having been available since 2011, but Microsoft now has Cortana, Facebook is testing one called M, and Google builds its voice assistant into its search apps.
These companies are presenting scorecards of their progress with quarterly earnings reports in the next few weeks, so what better time to hand out report cards to their artificially intelligent assistants? With that in mind, I set up tests for the assistants and graded their abilities to accomplish 16 tasks in categories that most consumers generally enjoy: music, productivity, travel and commuting, dining, entertainment and interests like sports.
In the end, none of the voice assistants earned a report card that would make a strict parent proud. Here’s how they stacked up in terms of grade-point averages out of 4.0.
• Google’s Google — 3.1
• Apple’s Siri — 2.9
• Microsoft’s Cortana — 2.3
• Amazon’s Alexa — 1.7
Apple was the strongest at productivity tasks like calendar appointments and email; Google was the best at travel and commute-related tasks. Alexa excelled at music, and Cortana was mediocre across the board. Facebook was left out of the grading system because the company denied access to M, though I did hang out with her for two hours on a friend’s account. More on that later.
Apple said that Siri had “become faster and smarter” and was available in more languages than other assistants. Microsoft said that it was “just scratching the surface” with how Cortana could help people. Google said that it wanted smartphones to do more of the heavy lifting, and that users could do a host of things just by speaking to Google. Amazon did not respond to requests for comment.
On the productivity front, Apple’s Siri, which is summoned by pressing the home button on the iPhone or by saying, “Hey, Siri,” was best able to schedule a calendar meeting with a friend in Hawaii, check what was on my calendar tomorrow, send an email and dictate my most recent email. Others could complete only some of those tasks — Google, for instance, could not read your last email out loud, and Alexa could not compose an email or create a calendar event.
Siri also fared well in music-related tasks, but was bested by Amazon’s Alexa. Both assistants could play the song “Hey” by the Pixies, put on the latest episode of the Radiolab podcast and play music in the instrumentals genre. But Alexa, which can be summoned simply by saying “Alexa,” could play a specific music station on Pandora, whereas Siri could only open the Pandora app.