Defining the Multimodal HMI for Jaguar Land Rover in the 2015 and 2016 Production Vehicles

Background

Driving is already a complex task that demands a varying level of cognitive and physical load. With the advancement in technology, the car has become a place for media consumption, a communications center and an interconnected workplace. The number of features in a car has also increased. As Voice Recognition has become a standard technology, Jaguar Land Rover want to focus on making VR better, but attaining perfection is not the answer as there are inherent challenges and limitations in respect to the in-car environment and cognitive load. The Multimodal Interaction (MMI)

approach offers a non-intrusive way to the drivers to interact with the features in the car. With the focus on user-centered design, the MMI can potentially help to reduce the distraction.

I have worked on Multimodal HMI for Jaguar Land in the 2015 and 2016 production vehicles. I have worked on designing the voice interface and ensuring that the visual and speech worked in harmony to accomplish the task and prevent driver distraction and discomfort while driving.

Objective of the Project

Jaguar Land Rover want to improvise the voice recognition feature in the upcoming models as compare to their old voice recognition with the Multimodal HMI as it's the new norm in the automotive world. I have worked on the voice feature, not only in terms of core voice pure, but significantly the link between the graphical user interface and the voice user interface and how that harmonized together. That's the next phase of the current 'say what you see' voice feature and bringing a new dimension to the philosophy of multimodal HMI.

The Challenge

As voice recognition and speech user interface is the new dimension for me, I had to start from scratch in terms of building understanding, current trend & technology and how it is adopted. The main challenge I have faced was while understanding the voice flow diagrams which was designed by the nuance for the voice hardware. Based on these flows I had to design the screens and screen flows. These screens also need to have key id's which will be tagged with the nuance ID so that once the speech engine understand any command uttered by the user these screens will appear on the GUI.

As a multimodal feature it also needed to respond to the secondary input, which was touch interaction in our case so I had to look at various instances where the flow might get completed through Voice & Touch. The other challenge was to pull out the commands from the nuance flows which is going to be displayed on top (intent bar) of the voice specific screens. These are 'Say What You See' (SWYS) commands which is provided to the user to complete the flow through Commands.

My Role

I lead the voice recognition & multimodal HMI. I have worked on defining the framework for the multimodality as well as worked on various speech features like core voice pure, dictation, voice note, etc. I was working closely with the speech engineers / developers, nuance team, etc. in defining the framework and discussing on the various use cases which need to be mapped on the speech engine provided by nuance. Apart from this I was working closely with the HMI attribute team of JLR who provides the guidelines "Golden Rules" which is set of rules defined by the team based on the usability factors for in car infotainment systems. The experience Design team who provides the style guides to the visual designers and review the design time to time and lastly with the visual designers to see how the wireframes/proposed design get translated into the visuals.

The Approach

Building Understanding Through Secondary Research

What is Automotive HMI?

The Autmotive Human Machine Interface (HMI) is the medium through which users can interact with the various features and functionalities of their automobile as well as receive various feedback from it to gain better understanding of its workings.

The Main parts of the HMI are:

1. The Instrument Cluster
2. The infotainment Panel

Evolution of Infotainment System

Recent developments in technology have led to an explosion of features in Infotainment and cluster Interfaces. Today these panels have moved from electromechanical contraptions to sophisticated on-board digital computers.

With the Evolution of the consumer world, there is an exponential rise in 'Things' people can do inside the car, resulting in the exponential rise in complexity of HMI. Basic driving is a function which is taken for granted.

Cars are No more About Mobility

The New HMI's

Benchmarking

After building understanding of the domain, I have done a benchmarking and a step count analysis on getting hands-on to into various automotive vehicles. This built more understanding about how voice really work in the car. The step count analysis of the various feature helped me in focusing my thinking on how to optimize the workflows for Jaguar Land Rover cars and make the workflow seamless & effective.

Changing User & Use case Scenarios

With the help of feature area description document (FAD) I have sorted out various usecase scenarios people actively perform while driving : Navigation, Texting and Calling, Listening to Music, Social Networking, Traffic jam etc...

Feature & Functions (Content Mapping)

I have mapped various features and function available in Jaguar cars and can be initiated from various touch points like steering wheels, Console and touch screen. This helped me in thinking about scenarios in which the feature is initiated by voice, but in between interacted with these entry points.

Multimodal Workflows

Nuance team had delivered various MMI workflows, which I have studied to understand the various screens requirements which are specific to voice as well as the modification required in the existing GUI screens. While studying these flows I have to pull out the various voice commands which user can utter while driving to start off the system / activate the feature and tasks.

Voice Key Screens

Based on the requirement of the key screens from MMI specs, I have designed various key voice screens for Media Player, Navigation, Phone and Tuner. I have also specified unique id's, voice related buttons and if there is any Say What You See Command exist for these screens.

Voice Key Screen Flows

Once the voice screens were finalized the next step was to look at the multimodal interaction on these screens and define the workflows. Some of the examples are shown below in which the interaction with the buttons and end to end workflows are shown.

Multimodal Events

I helped the development team on various activities. Among all one of the important activity was defining the touch events on voice screens. Basically on interacting with screens in voice session what event will be called off by the engine and what result will be displayed. Some of the events are shown below:

Voice Commands Prompts

Voice Commands Prompts are the commands which can be uttered by the user / drivers while driving to start various flows for the features. These commands were gathered from the MMI specs provided by the nuance and also used in voice help to showcase list of commands uttered by users for various features. Also user can create their own voice tags and how these can be used.

Visual Design

With the help of visual designers, we have created various voice based screens and also the GUI screens with modifications based on the feature theme. We have worked closely with an experience design team who provide the visual identity.

Videos

Please find below some of the video available online on how to Use Jaguar Land Rover Voice Command Functionality.

The Impact

I have successfully delivered the Multimodal HMI for Jaguar Land Rover Which was greatly appreciated by the team. The various features were well appreciated by the everyone in the team because I have successfully optimized the task flow, which can be performed by the users in much effective & efficient manner. I have learnt a lot throughout the design and development and really built understanding on voice based interface. This next generation multimodal HMI is still used in all the new production vehicles of Jaguar Land Rover.

Next Case Study