On the other hand, language does. Today, we use words that didn’t even exist a few years ago. That said, it’s our goal to have our vehicles’ Intelligent Assistant feature, the Virtual Agent (VA) found in many 2022 and newer Toyota vehicles equipped with Toyota Audio Multimedia, to keep pace. There’s an art and a science behind it, especially as vehicles become smarter with every voice command – “Hey, Toyota,” “Hola, Toyota” and “Salut, Toyota” serving as a handful of “wake words” that can control anything from radio and climate control functions to telling you a joke when you need a good pick-me-up on a VA-capable vehicle.
This is where Toyota Connected North America’s (TCNA) Labs team comes in, with data linguists that bridge the gap. They’re the human element of natural language processing, helping translate utterances into actions. And data linguists Jeanne Lemieux and Frania Navarro are making it look like second nature – in Canadian French and Mexican Spanish, no less.
Know Your Domain
TCNA is responsible for programming VA to make commands conversational with a goal to lessen cognitive load on the driver and passengers. In the past, a driver might have a robotic set of commands and subcommands to get a car to lower the temperature, for instance. Now, voice commands can be done as easily as talking to a friend.
Managing Machine Learning Engineer Samit Shah explains it like this:
“We have a set of domains. Each domain has a model that raises its hand and says, ‘I can handle this’ or ‘Nope, this is out of my domain’,” he said. “If a domain raises its hand, the user input gets handled by that specific domain and the VA takes any necessary actions.”
So, when you, a loyal Toyota driver, say “Hey Toyota, I’m cold,” your command goes to the cloud, and then the command is fastidiously put into motion until you’re more comfortable. But there are numerous other ways to explain your intent to the VA.
And then, there are plenty of ways to say that in any one of the three predominant languages in North America.
Data linguist Jeanne Lemieux creates text-based data prompts for Canadian French voice commands in Toyota vehicles. In this photo, Lemieux is using software that helps QA engineers ensure accuracy of voice prompts.
Know Your Audience
Lemieux and Navarro are language nerds. Lemieux was born and raised in Quebec City and holds a bachelor’s degree in Translation. She has studied multiple foreign languages, including Spanish and German and lived in France for a few years, so she is well aware of the differences between Quebecois French and European French. Prior to TCNA she has worked in digital marketing and localization for an international hotel chain. When she’s not helping think of every utterance under the sun, she’s also a professional translator and voiceover artist.
Navarro earned her bachelor’s and master’s degrees in applied linguistics, keeping pace with how language influences culture and culture influences language. In her field, it’s not merely about translating; it’s about providing context around the nuances of language. Previously, she had worked in the video game and software industries, sorting out the nuances of Mexican Spanish and other dialects of Spanish spoken in Latin America.
As language continues to evolve, Lemieux and Navarro keep tabs on this evolution and adjust accordingly.
Navarro not only sorts through English commands coded into the natural language recognition of the current Toyota Audio Multimedia system, but she also regularly sorts through data logs and spreadsheets to reference how Spanish-speaking drivers interact with the system.
It also doesn’t hurt that Navarro primarily speaks Spanish, and her father is in automotive retail, so she can quickly find any newer car-related term or ask about real-world experiences with customers.
“You want to be inclusive, but you also need to have some kind of base for generalization,” Navarro said.
Lemieux keeps up with Quebecois newspapers daily and also relies on the human element, regularly chatting with friends and family in Canada.
“When we offer 10 different ways to turn off the AC, we’re not going to list all those utterances how you’d see them written in a book,” she said. “I really have to think, ‘How would my young niece or brother-in-law say something?’ so that’s how we trained the machine.”
Sometimes, even, prompts can be no more than a few words. For instance, in both French and Spanish, nouns are often accompanied by gender-specific articles, while verbs feature complex conjugation patterns. But the data linguists are noticing users eschewing these conventions in favor of speed and practicality. When Lemieux and Navarro hear how users’ preferences are changing, they must adjust, too.
Data linguist Frania Navarro infuses nuance into the voice command controls for North American Toyota vehicles in Mexican Spanish.
Know Your Capabilities
Machine Learning Models can piece together instructions, but they need raw data. Lemieux and Navarro turn language into data, functions into formulas. There’s no way they could think of every way to say something, but their input helps the cloud computing model figure out what the driver wants.
The two receive basic prompts from Shah and the Toyota Connected Labs team that is responsible for voice commands and user interfaces. From there, they iterate. They test. They correct, including AI-generated prompts that iterate through the cloud. They add utterances for more functions as engineers release them through over-the-air updates. The two are working on an enhanced roster of EV-specific commands for next-generation vehicles as well.
Additionally, they record audio files that help QA test automatic speech recognition in English, French and Spanish especially for automation engineers.
After iterating utterances, they input functions a data structure-friendly format, coded multiple ways.
“Lemieux and Navarro have the lingual capabilities and understand the technical aspects as well to work with the machine learning team and contribute to our codebase,” said Shah. “They’re a crucial part of providing our drivers a seamless voice-operation experience.”