Category Archives: NUI

Posts related to Natural User Interfaces

Concept: Maintenance Training 2030

April 25, 2032 …

Ed, a Canadian aircraft maintenance training instructor located in Montreal, gets ready for his upcoming engines maintenance class.

Supposed to be a big class this week, 8 students.

  • 1 from Canada
  • 1 from the US
  • 2 from South America
  • 2 from India, and
  • 2 from China

Ed checks on the equipment in the classroom.

He puts on his Holo-Glasses, which come to life, softly glowing holographic data displays and icons popping up in front of him.  The device recognizes him, launching the virtual assistant to greet him. “Hello Ed! How are you? All set for your class?” “Just fine, thanks. Everything set?” “Yes, Ed. All the students are going to be attending; no cancellations. Everything looks good with the students. One was having some technical issues earlier, but I helped him through it.” Excellent,” thought Ed. “Everything looks alright with 15 mins to go.”

Ed begins cueing up the opening presentation notes, and the multimedia training manual. These pop up in their own windows in Ed’s field of view.

As Ed continues his preparations, the digital assistant relays notifications confirming the status of the students. The assistant is communicating with the students before class so Ed can focus on his preparation. Everything is looking good. Ed checks the 3D cameras and tests out his holopresence projection, seeing what his students will see.

“Loo-king good! Let’s do this!”

A few minutes later, the class begins. Ed welcomes the students as they holopresence in from their remote locations. Ed and the students, their Holo-glasses on,  take their places in the shared virtual classroom. The software places softly glowing holographic representations of the other participants in the shared visual space.  Ed looks out at the students’ faces, and the students see a holographic overlay of the same classroom and the same students from their own virtual perspective. At first, the experience is a little eerie, but as the class gets going, and all the students introduce themselves, the illusion takes hold and it feels like everyone is in the same classroom.

Ed presents the content, asks questions, and listens to the responses. Master teacher he is, he observes carefully, gets a sense of the learners’ body language and expressions, and, much like in a real class, adjusts as he goes. Ed brings up holographic 3D animations and models of the engine and components for the class to see. He zooms, rotates, and takes apart the holographic engine parts. The hologram also appears in the students’ fields of view, and Ed invites students here and there to come up and try for themselves and demonstrate actions to the class. Static images appear on screens in mid air, demonstrating schematics.

In the afternoon portion it is time for the virtual hands-on lab exercises. Ed and the students convene again, once again with beautiful, interactive 3D holographic models of the engine floating in the shared digital overlay. This time however everyone puts on their SureTouch(TM) haptic feedback gloves.

The gloves use sensors to read finger and hand position, the headset measures their hand positions in relation to the digital model’s virtual position, and actuators in the gloves give pressure feedback to simulate handling real objects with substance instead of just weightless holograms. It’s kind of weird at first, and it’s not quite the same as the real thing, but close enough for horseshoes and hand grenades, as they say. And definitely a hell of a lot cheaper than taking an actual engine offline to train.

As always, it took a few years for the technology to perfect itself and a lot of research and proofs of concept before the regulators really believed it could be as effective as the real thing. The Dutch Aerospace Lab did some great research as always, and once EASA signed off, the other regulators followed pretty swiftly after. Regulators came to appreciate virtual maintenance training, just as they came to appreciate the power of full flight simulators decades before.

The company definitely appreciates it too – they save a small fortune in flights, hotels, taxis, and per diems doing virtual classes like this over the course of the year. As do the students’ companies..

Ed for one, appreciates it too. No packing, no airport security, no  cramped 12 hour flight, no hotel room, no taxis, no jetlag, no traffic. Well … scratch that last one. This is Montreal, after all, where the seasons are winter … and construction. Even in 2032, there’s plenty of traffic. (You can’t win ’em all, I guess.) “Oh well, ” thought Ed. “Decent weather today, so at least could read a book on the way in while the autodrive on the car took care of all the unpleasantness.”  And all from the comfort of the Montreal office.

Ed loves it, and his family loves it too – less time away. And besides. even though he felt a little silly to admit it, irrational as it was, Ed had felt a littled weirded out by flying ever since they started the rollout of unpiloted commerical flights in the late 2020s. Hundreds of times safer than human pilots or not, it’s still kind of creepy to have algorithms flying you around instead of people.

“Or maybe I’m just getting old, ” Ed thought. Gets a little jarring after awhile to see the world transform itself before your eyes so quickly. The young seem to take it in stride, unphased, as they always do. And, Ed had to admit that the toys are pretty cool. All this change has its benefits.

Such is the stuff of life in a world of sci-fi dreams made true.





The television as a learning and training space


Recent years have seen the world of training embracing  learning on mobile devices, or mlearning, for short. There are many reasons for this:

  1. Client demand as people more and more browse the internet principally through mobile devices
  2. Clients always having their phones with them, allowing lots of little moments during a day when learning could potentially take place.
  3. Phones having lots of sensors and input methods, allowing for innovative interactions
  4. Phones allowing multiple communciation methods

Designers and developers have been working on designs using mobile learning. At its most basic this has taken the form of  using file formats so that videos or presentations will play on a tablet, or even just an iPad. Or to make the training as an iPad app or playable within some container app.

Others, approaching the matter with some semblance of actual seriousness, have gotten more creative, and tailored training more to the unique affordances of smart phones and tablets. They make learning games that use sensors or activities that use sensors as inputs for motion or touch based interactions. Or they use location information. Others use ideas of informal learning and performance support to break training into small, focused little pieces that can be accessed in a spare moment.

eLearning authoring tool providers advertise their tools as enabling responsive eLearning. They hype the promise of being able to publish content to multiple media and device types, for desktop, tablet, and mobile.

This is good for learning and training. However, in this focus on mobile, we may be losing sight of possibly the next key development of web-based learning and training – the television as a learning and training space.

Television as a new window to Internet content and learning

Sitting on a couch with a tablet is a nice way to watch  a video or presentation. The device is light and comfortable. But, still, it’s a 10 inch screen. It is nice for portability, but it’s still a small screen. The small size is a compromise, trading visibility and real estate for portability.

But across from the couch is what? The TV. Big screen – 30, 40, 50, 60 inches. 1080p HD, easy to see, nice to watch, decent speakers. And you don’t have to hold anything.

Television used to be a box on which we watched traditional television programs, whether delivered over the air, or through cable or satellite broadcast. Then, came VHS players, DVD, Blu-ray, video game consoles. The living room TV became instead the screen in the middle of a home entertainment center.

Now, increasingly, televisions are also becoming just another one of the screens,albeit, much bigger ones, through which to access internet content, whether for entertainment, work, or learning. This takes the form of video, audio, text, and apps. The long promised fusing of internet and television has arrived, with several different options available to make this possible.

Many TVs are now “smart TVs,” combining a TV with a computer. These TVs are WiFi enabled, with built in interfaces and platforms with apps capability. Apps allow straightforward connectivity to content sources like Youtube, Netflix, digital music streaming services, and other streaming media.

Modern TV screens also allow for stereoscopic 3D. While no longer a faddish selling point, most newer TVs are by market standard capable of displaying stereoscopic 3D content, whether accessed over the web or on 3D Blu-rays. TVs stand out notably from the other screens through which we consume content in that many of them today readily allow Stereoscopic 3D media. TVs are the one dependable 3D screen that people commonly own.

TVs are also capable of being connected to gaming systems like PS4 and Xbox One, the second of which includes the Xbox Kinect motion and voice sensor. These systems, while meant primarily for gaming, are also intended more generally for home entertainment, with app platforms and apps like Netflix and Youtube to see internet video content.

As well, set top boxes like Apple TV as well as many WiFi enabled Blu-ray players offer a similar bridge between the television and the internet.

Tablets, phones, and laptops can share screens wirelessly to TVs, either through devices like Apple TV, game systems, or via Miracast / WiDi from enabled devices.

It is easy to get content on the TV. As well, the TV will either be setup with sensors, whether in the TV itself or via something like an Xbox, or the person will be screen sharing from something which has sensors and a touch based control interface. It becomes easier to browse, select, and interact with online content shown on the TV.

Designers, both web designers an instructional designers,  need to think about training and learning possibilities in this space.  just as they should be thinking about that OTHER class of displays that will also be more and more in people’s lives – wearables and augmented/virtual reality tech such as Google Glass and Oculus Rift. (More on this in a future post)


There are a few challenges in this area:


One main challenge is that there are so many different sorts of configurations and ways to connect the internet to the TV:

  • Via game consoles such as XBox One or Sony PS4
  • Smart TVs
  • Set top boxes like Apple TV, Wifi Blu-ray player, or Chromecast
  • Computer connected to the TV to share the screen via HDMI cable
  • Wireless screencast from laptop, tablet, or smartphone to the TV, whether through Apple Airplay or up and coming wireless screencasting standards WiDi (wireless direct) and Miracast.

This makes things difficult for developers, as there is no one clear dominant target for development.

The gaming consoles, which have positioned themselves as not only gaming platforms, but home entertainment hubs, may be one promising avenue, as the multi-billion dollar gaming industry already attracts lots of skilled developers to these platforms. Microsoft’s XBox One in particular runs an operating system related to Windows and uses the same development toolkit. Also, these gaming consoles offer innovative ways to interact with the content on the TV through different types of controller devices. These include body movement and voice based controls. The gaming console option offers interesting possibilities in terms of innovative learning interactions.

A more straightforward, elegant solution may be through smart TVs, where everything is in one box. This would especially be the case if in the future the telvision included sensors that could be turned on for Kinect-like interaction with cameras and microphones. One challenge, however, is attracting developers to different platforms from different manufacturers. Possibly only a company like Samsung, which is involved in manufacturing phones, tablets, computers, and TVs would be in a strong position to carry over advances in interfaces and interactivity from other devices to TVs. Or someone like Apple.

The other challenge would be emotional reactions from consumers. When early press about the Xbox One suggested that the system would require the Kinect sensor – which includes stereo cameras and microphones – to always be on, even when the system is not in use, people became very paranoid, and there was a backlash.

It is possible that TVs will evolve in coming years to become a sort of all-in-one computer, with web connection, innovative web browsing methods (the concept of adaptive web design will also have to adapt and evolve to cater to TV as a screen), app platforms, and built in SSD memory space, possibly supplemented by cloud storage.

Quite possibly the next stage of the Apple OS – Android – Windows – Linux battles will be fought on the battlefield of internet connected TVs. Ubuntu, for example (A variant of the Linux operating system) has actually been positioning itself as a flexible multiplatform, including TV – OS for some time.

Wireless screen sharing may be the simplest approach, making the smartphone, tablet, or PC the central point of control of what appears on the TV screen. Desktop and laptop computers would have limits though in terms of enabling learning interactions.

Tablets and smartphones, could potentially allow for interesting learning interactions through the accelerometer, gyroscope, and touch sensors.

The scene is probably going to be messy for a few years with a lot of options making it hard for developers to pick. This will make it hard to form development communities that will drive things explosively forward.

Interface and Interactivity

The possibilities for learning and training will depend somewhat on the options available for interactivity. One of the challenges in making the TV a hub for learning content is how the user can control and navigate content sitting or standing from across the room. Good eLearning and online training especially requires rich interactions.

But how do you interface with the TV? A computer you sit right there and control it via mouse and keyboard, and to a lesser extent, microphone and camera. A tablet or smartphone you tap it, swipe it,  turn it, talk to it, because again, you’re up close to it and it fits in your hands.

TV is different. You sit back from it, or stand back from it. You’re not going to stand at your TV tapping the screen like those big maps on CNN election night.

There are probably four major options:

  1. Some modification of a traditional TV remote, possibly one with a touchscreen and accelerometer/gyroscope sensors
  2. Some camera and microphone based sensor like the MS Kinect that lets you control via voice and body gesture
  3. Controlling through a laptop computer, tablet, or smartphone, which shares the screen wirelessly via WiDi, Miracast, or Apple Airplay and lets you control things via touchscreen and motion sensors. The TV simply becomes a screen to mirror content on the other device.
  4. A smartphone or tablet is paired with the TV via an app, and serves as a WiFi-connected touch- and motion-based controller.

All of these could probably be made to work, though options 2 and 4 are probably the most plausible options going forward in terms of usability and in terms of building on existing platforms.


Learning and Training Possibilities

The matter then becomes how to harness this emerging new portal to the internet for learning an training.

A few possibilities come to mind.

  • Any passive consumption of video content. Particularly content in HD or stereoscopic 3D format. YouTube contet, for example. A TV would be the most natural and comfortable way to watch. Everything becomes bigger and more lifelike
  • Educational gaming activities using a gaming controller
  • Web content browsing with voice and gesture inputs enabled by something like the Kinect. Say, for example, a view of different documents or different levels of detail making use of different focal planes in a 3D field of view. This allows information and screen elements to be arranged not just along dimensions of horizontal and vertical, but by depth as well.
  • Interaction with stereoscopic 3D models using Kinect sensor. Such as chemical structures, architectural structures, geographic feature models of an area, or components of equipment.
  • Live, synschronous, life-like teleconferencing via TVs and Kinect sensors using apps like Skype or something like it embedded in a virtual classroom application. Virtual classroom would work very well on an HD television with connected camera and microphone. For live, face to face communications, for conversational practice in language learning, or a live virtual tutoring session.
  • Using the Kinect, the learner practices some psychomotor skill. At the same time, the Kinect camera lets a remote instructor watch the performance and comment. The Kinect could also capture data to assist in analyzing biomechanics.

These are a few sample ideas. Maybe readers can think of others.


The past six years have seen dramatic changes with the coming into the mainstream of mobile devices as a new space for online learning, with unique affordances for interactivity. The mobile web and mLearning have expanded our horizons for entertainment and learning. The television, connected to the internet offers a new field on which we can ply our craft as designers and developers. It’s a developing field with a lot of options that will take some time to sort out and settle down. But for those of us tasked with helping our clients and students to learn and develop, it’s a field we would do well not to ignore.


Once again, feel free to share your comments, either below, or via social media.

On Natural User Interfaces (NUI)


Welcome and an early happy weekend. This article is intended to give a bit of deeper background around trends in what are called “Natural User Interfaces,” or NUIs. This term refers to a computer technology trend related to how we interact with computers. It’s a term that I’ve used in some other articles recently, but didn’t get into too deeply, because it takes a bit to explain it so that you do it justice.

Fair warning that this article is intended to be forward looking. It is NOT about looking at tools that are currently available off the shelf. This is not about immediately applicable information. This is a look at where the technology of human-computer interfaces has come from, where it is, where it is probably going in the next few years to come, and what kinds of possibilities that could introduce for computer based training.

So in that respect, it’s about getting yourself mentally prepared for what will be coming a few years down the road. For those who like to think ahead, to dream about future Instructional Design possibilities using the tools that haven’t been invented yet.

My recommendation: if the preface and introduction pique your interest, bookmark this article, email yourself the link, and maybe set it aside for a quiet Sunday afternoon when you have some time to read and reflect. Then you can process it and reflect on the future possibilities of what you can do with this technology. Anyway, I hope you enjoy the article.

Introduction: What is a Natural User Interface (NUI)?

In a recent article, I talked about the future potential for the Kinect sensor to enable on the fly adjustments to presentation in e-Learning. In that article, I brought up the concept of a Natural User Interface, or NUI (pronounced “noo-ey”). In that article, I introduced the term almost in passing, but I recognize that a lot of people might not be familiar with the concept. The intention of the present article is to go into a little more background, to give some sense of the significance of this new type of human-computer interface, what came before it, how it has already changed how we use computers, and how future developments promise to further shape our interactions with computers. Finally, I will try to look ahead a bit at how these types of interfaces could shape the way we train people using computers.

Let’s get started.

Paradigms of human-computer interaction

So the first question for those unfamiliar with the notion of an NUI would be “what is a NUI?”

Well, to answer this question, it helps to go back a bit into the history of computing.

Computers as we generally know them (electronic calculation devices) have a history going back about 70 years, since the time of the second world war. If you want to be technical, you can trace computing back to Ada Lovelace and Charles Babbage and the Difference Engine and Analytical Engine in the early to mid 1800s, but for simplicity, let’s say 70 years, starting around 1945.

What started as a technology used to automate complex computations for a handful of high-end research and military institutions via massive electrical machines has evolved and grown over these seven decades to become a technology that is an integrated, essential part of the fabric of life (at least for people in relatively developed parts of the world). Along the way, the power, speed, and storage capacities of computers have increased exponentially, while the costs and sizes of components have at the same time shrunk at exponential rates. Computers have gone from machines numbering a handful in the whole world to numbering somewhere in the billions. Some billion powerful computers are carried around in people’s pockets in the form of smart phones, and embedded computing devices appear in almost any electrical device produced today.

Along with these developments, the means through which people interface and interact with computers have also dramatically changed. This change has come both as a result of technological developments, and at the same time as a driver to uptake of computers amongst the general population,  Human-Computer interaction has gone through a number of important paradigm shifts.

A paradigm, for those unfamiliar with the term, is a dominant contemporary pattern or way of conceptualizing and doing things. There have been a few major paradigms of human-computer interaction, with corresponding shifts as the technology moves from one dominant mode of interface to another.

I first want to speak about three major early paradigms of human-computer interaction:

  1. Batch interfaces (1940s to 1960s)
  2. Command Line Interfaces (1960s to 1980s)
  3. Graphical User Interfaces (1980s to 2000s)

I will then speak about the recently emerging paradigm of Natural User Interfaces (NUI). I will discuss some of the different examples of NUIs, and finally look at new possibilities for training opened up by these sorts of interfaces.

First paradigm: Batch interface (1940s to 1960s)

The first computer interface paradigm was the batch interface. In this setup, users entered commands through stacks of punch cards punched by hand and fed into a card reader peripheral, which read the punched holes via optical scanning and turned the entries into electrical inputs. Programmers would carefully enter their code on the punch cards and submit their stack of cards as a batch to be scheduled and run by the administrators of the machine.

Remember, this was a time when computers were huge machines taking up most of a room, and a whole university or department might share one of these machines. It was a scarce, in demand resource, so programmers had to wait their turn for their code to be run. Computers could run one program for one user at one time. This produced a serious bottle neck in performance. Users could not typically just sit at the computer by themselves and use it because the resource was limited and the time could be used more efficiently if the programs were run together one after another as a batch.

This cycle from submission of the program to scheduling to entering it into the computer to running could take days, depending on how busy the administrators of the computer center were. And if there was a bug, something miscoded in the punch cards, the program would fail, and the programmer would have to start again, identifying where the error was without any sort of guidance (“syntax error on line 57,” etc). Such aids didn’t exist. The programmer would try to track down the error in logic by hand, and then resubmit the revised program to the queue. It was a system that encouraged refined first draft work.

In a batch interface, the computer reads commands, coded in rigidly structured messages, carries out commands, and gives output through a printer. The computer would take in the programs of many people at one time, and process them, one after another, as a batch. It was in this time period that the first computer languages were developed.

The frustrations of dealing with these batch processing systems were a major drive for computer science researchers of the day to look into alternate modes of human-computer interaction.

Punch card


Second paradigm: Terminals and Command line interface (CLI) (1960s to early 1980s)

Then followed the command line interface (CLI). This came about along with development of early computer displays and monitors with keyboards used as inputs. Users could input characters through a keyboard and see them displayed on the screen. This would take place at a terminal with a keyboard and  display connected or networked to the main computer machine.

The main computer would be set up to time share between multiple users. The computer basically rapidly switches between carrying out tasks for each user, allowing the central computer to “simultaneously” handle many users at once. To get a sense of how this works, imagine getting your household chores done by doing laundry for a minute, then switching to keeping an eye on dinner for a minute, then switching to attending to your kids for a minute, then switching to tidying the living room for a minute, then switching to sweeping the floor for a minute. Then imagine thistask switching a million times faster. You’re doing one thing at a time in little slices, but to a casual observer, everything is smoothly proceeding all at once. Generally, your computer at home or at work “multi-tasks” in a similar sort of way. The coordination of the time sharing created a certain amount of overhead using up computer resources, but this became less of a concern as computers became faster over time.

So the user no longer had to punch cards, and no longer had to give them to someone else to feed into the machine, and wait. The different programmers and application users could get access to a terminal, and use that to interact directly with the computer in something resembling real time. The user could input text information, and get text output back more or less immediately.

This paradigm also overlapped with the appearance of the first so-called “micro-computers” used as office business machines (e.g. the IBM era). It was also the paradigm under which the first “personal computers” were born. These were standalone computing machines small enough to fit on a desk.

The user of one of these machines could use the keyboard, aided by the feedback visuals from the screen, to type documents, or to enter commands. The user controls the computer and performs actions such as creating, saving, deleting, copying, and moving files and directories using text based commands typed into a a command line. This can still be seen today in the command line in Linux and in the mstsc / Commad Prompt ultility in Windows.  MS DOS, the first Microsoft operating system, worked like this.

This is known as a Command Line Interface or CLI. More advanced computer programming languages were also developed at this time.


Third paradigm: Graphical User Interface (GUI) (1980s to 2000s)

The next paradigm was the Graphical User Interface or GUI (“goo-ey”). This consists of a “desktop metaphor,” with program windows, menus, virtual “buttons” and other controls on the screen with which the user interacts using a mouse and pointer. Associated with this is the acronym WIMP=Windows, Icons, Mouse, Pointer.

The earliest GUI was from research at Xerox PARC in the 1970s. These ideas were later taken up by Apple Computers in the early Macintosh and Microsoft in their Windows OS. Interactions simulated the way a person might interact with a real world machine, by “pushing” (with mouse clicks) virtual buttons, turning virtual dials, etc. It was at this stage, corresponding with a sufficient miniaturization of computer components and fall in price, that the idea of a home “personal computer” took hold. With the desktop metaphor, windows, and mouse pointers, it became much more natural for everyday people to use computers. There were still many rough edges, and certain arcane bits of knowledge to learn, but overall, it became much simpler for everyday people to do basic things with computers. Computers were starting down the road to becoming a household appliance that average people would use as part of their everyday lives.


The emerging paradigm: The natural user interface (NUI) (2000s to present)

The next paradigm of human-computer interaction is so-called Natural User Interfaces, or NUI. This can encompass a variety of types of interaction, but the overarching idea is that rather than having artificial or mechanical intermediary means of input, the user interacts with the computer in ways more like those used to interact with people and objects in the real world, and more directly. This typically means touch, body / hand gestures, facial expressions, speech, and giving queries or commands to the computer in something much closer to the ambiguities of everyday language rather than in rigid computer syntax.

What does this mean? Well, to illustrate, let’s look at the predominant method of computer interaction that we’re just coming from and still wrapped up with. Namely, the mouse.  Or, more precisely, the mouse and pointer as a way of navigating graphical menus and control interfaces on a screen display, with the keyboard for the user to enter in data like on some electronic typewriter. This form of interaction was almost completely predominant from around 1984 right up through to around 2008, a period of 24 years. The 1984 date marks the appearance of the Apple Macintosh (128k), which featured a GUI and mouse. 2008 on the other hand was the appearance of the iPhone 3G, which helped to explode the popularity of capacitive multi-touch smartphones. (As much as I dislike Apple’s closed model and think they’re past their prime, I have to grudgingly give them credit for having been squarely at the center of  both of these technological inflection points.)

The mouse has become so much a part of our daily activities, at home and at work, for so long, that it’s easy to lose sight of how awkward and un-natural a way this is of interacting with with a computer. Or interacting with anything. You sit in front of a computer screen, staring at it.You have a button on the screen. You have to grab this mouse on the desktop, drag it along the horizontal plane of the desk surface in order to move the visual of  a  pointer arrow on the vertical plane of the screen surface. And then you click on a button on the mouse to “click” the on-screen button. Once upon a time, this was the only way to mediate the pressing of a button. It was simply the only way to do it. But what is the most natural instinct to do this, today, given the technology widely available now, namely touchscreens? Well, since 2008, with the iPhone, and since 2010, with the iPad, it’s simple. You reach out your hand to the screen and touch the putton to press it. The whole step becomes much more natural and effortless.

Admittedly, it’s still kind of weird, because you’re still blocked by this 2 dimensional surface as you bump up against it and touch it or move your hands over it. It’s still a little limiting and artificial. But it’s getting there. You’re completing the metaphor at least of the classical graphical user interface or the desktop workspace on which you place things and move things around. Instead of moving them with a mouse, you move them directly with your fingers. You’re still operating something like some old fashioned instrument panel, but that has become more naturally engaging. You move like you’re actually operating an instrument panel in real life.

As mobile computing and mobile internet have taken off, this has impacted web and application design so that even on the desktop, the user interface principles inspired by touchscreen usability – lots of white space, simplified menus and controls, and large button targets – have become predominant. Designers try to build applications that work well on both.

Interacting with the computer in these more natural, everyday ways means that in a sense, the interface fades from attention and becomes invisible to the user. But the idea is that generally the experience is smoother, more realistic, more like a real world interaction. The distance between the user and the computer becomes smaller. In this way the computer becomes a little more like an extension of the user’s body. The user simply smoothly interacts with the computer to do what he needs to do.

We call such an interface a Natural User Interface, abbreviated NUI, and pronounced “noo-ey.” It’s the idea of an interface that drapes itself over us, fits us like a glove by letting us interact with the computer more like we interact with real world objects and people.

In popular entertainment, we see some examples of futuristic concepts of use of NUIs. The computer on Star Trek TNG, for example, which the crew commanded through voice or touch screen control panels as they walked around the ship and did their thing.

Or the gesture interfaces Tom Cruise’s character used in the Pre-Crime unit in Minority Report.

Or more recently in the battle “simulation” in the film Ender’s Game.

Multi-touch touch capacitive screens as seen in modern smartphones and tablets are one good example of an NUI. You interact with screen items by touching them with one or more fingers to stretch items, rotate them, shrink them, etc.

Virtual assistants or agents such as Apple’s Siri or Microsoft’s Cortana are another example, or another aspect of natural user interface technology. Here users interact in a somewhat conversational manner with the computer using speech. Some of the predictive elements of Google Now would also be examples.

Haptics (touch based interfaces) are yet another element to make interfacing more natural by simulating the texture and force feedbacks and resistances you would get interacting with real objects.

Virtual reality would be another example of a natural user interface.The person interacts with the virtual world through head and body movements, receiving visual feedback through some sort of helmet screen.This is a technology going back some decades, but is becoming more affordable and feasible now. An example of a mass product is the Oculus Rift by company OculusVR (In the news of late for having been acquired by Facebook).

Another example is augmented reality, as in Google Glass. Here, important contextual information is projected within the user’s field of view to give continuously present information.

NUIs can also be combinations of these different types of technology. For example, the combination of speech and body / hand gestures is used in the Microsoft Xbox Kinect sensor. Microsoft, has opened the sensor with free APIs and SDK for developing NUI-enabled software for Windows using the Kinect for Windows sensor. The Kinect is a sensor that was previously sold as an optional peripheral for the Xbox and which is now a bundled part of the new Xbox One gaming and home entertainment console.

This particular device features two cameras for stereo machine vision with depth perception. Software in the device can make out limbs, facial expressions, hand gestures, limb and finger movements, face movements, facial expressions, even the pulse of the user, and use these as inputs for control. Multiple microphones are present for noise cancellation and for recognizing directionality of sound. There is software on board for voice recognition and for facial recognition.The user controls the game by voice inputs and by moving his body and hands.

This represents a more natural way to interact and brings to life some of these models of human-computer interaction forseen by science fiction earlier. It is not hard to forsee possible applications to training with this, especially with APIs of the device open to commercial and research development. The following links and the video below give some sense of what is being done with this sensor tool.

The Xbox One with Kinect is probably the hardest push right now for mass adoption of Natural User Interface technology in the home. There is also an Xbox Kinect for Windows sensor coming out that would allow games and software to be written using this device to control a computer.

Another potential route forward might come in the form of the iPad a few generations down the road if/when Apple can put something similar to Kinect’s sensors today in the iPad. The iPad would make a sophisticated control device for the TV, with the iPad mirroring to the TV screen. So this hypothetical future iPad could watch you through twin cameras, to read your eye movements and facial expressions or detect hand gesture based inputs. The microphone inputs, combined with cloud services, could read speech queries or commands from you. The touch screen would detect button presses, finger or stylus drawing inputs. The accelerometer and gyro would recognize if you’re sitting or standing and in what orientation you’re holding the iPad. You could then hold the iPad in different orientations in space as a control surface or workspace. The problem with the Xbox Kinect sensor is that it watches from farther back. So it can’t pick up yet as much nuance of detail as you could with a closer camera. A camera in the iPad could do that.

I wouldn’t be surprised to see Apple to do this, getting everyone used to this method of interaction, and then hitting with the long-predicted Apple TV, integrating something like the Kinect sensor and a slick multiple layers of Natural User Interfaces built in. Bang and bang. It would have a big impact.

Learning and Training Applications

All of this promises to really shake up how we interact with computers. And since interaction is such a key element of computer based training, this has implications for us as designers of instruction.

There are a number of foreseeable learning and training applications for this sort of technology. To name just a few examples:

Speech recognition and text to speech could be useful for language learning.

Gesture based controls could enable more lifelike interaction with 3D models, especially if using stereoscopic 3D image displays. This could potentially be used for a variety of applications in technical training:

  •  to manipulate and examine equipment in maintenance training.
  • to learn structure of machinery by virtual manipulation of 3d models, including assembly and disassembly. Haptic feedback outputs could even simulate the sensation of touching and working with the actual equipment.
  • in biochemistry, to manipulate 3-D models of large molecules like proteins to understand their structure and active sites
  • or to visualize biological reaction steps

Virtual reality could be used to simulate the operation of certain complex equipment, including running through rare or emergency scenarios.

For soft skills, imagine the immersiveness of a training program where you interact with a 3d character in a scenario using simply your body language and your speech. The realism is greatly heightened. Or imagine a training program that can give feedback on your body language, verbal tics like filler words, and your facial expressions while you give a simulated presentation or sales pitch.




Concept: Promoting persistence with exercise equipment through video gaming


Fitness is a big business.

In the US, for example, as of 2009:

  • Health clubs: $20 billion a year, 45 million memberships.
  • Consumer fitness equipment: 3.2 billion

At the same time, there is a lot of concern about public health from diseases related to obesity and lack of exercise.

People spend a lot of money in particular on home exercise equipment. Devices like treadmills, rowing machines, elliptical trainers, and exercise bikes. But people don’t tend to stick with it. The initial motivation comes, but the motivation often doesn’t persist. A common story is that families will buy these for Christmas as part of some intended New Year’s Resolution. More often than not, the box is opened, it’s set up in the basement, it’s actively used for a few weeks or months, and then it’s forgotten about again.

What can be done to help this? Is there a solution to this performance gap?

Gaming and Motivation

One area that excels in creating and then sustaining motivation (persistence) and intensity of engagement is video games. People will spend hours and hours on games, sometimes to the degree of forgoing food, sleep, other activities, and human contact. Games achieve this with a range of different mechanisms: fun and variety, a mix of long term, middle term, and short term goals (game completion, boss or world completion, and minor task or level completion),  continuous informational feedback and rewards in the form of scores and achievements, competition with other gamers, and social communication tools to allow discussion of game strategies and mutual social based motivation.

Could this power of video games be harnessed to encourage people to make more frequent and more effective use of their home exercise machines? Namely through fitness based games that make use of and incorporate the use of the exercise machines?

A solution: fitness based games using the equipment

Fitness based games are something that already exist. There are a number of titles for Wii, Xbox 360 with Kinect / Xbox One, and PS3/PS4. Often, these will make use of the motion based controllers. For the Wii and PS, this involves a handheld motion controller, while for the Xbox with Kinect, this involves simply moving in front of a sensor that captures body movement. The problem with these is that they just involve you moving or jumping around in your living room. There is not a lot of space. This works for people that like yoga and aerobics, but not so much for people that like to bike or run.

As far as I am aware, there aren’t any titles that make use of home exercise equipment. This is, in my mind, a gap just waiting to be filled. Microsoft, the manufacturer of the Xbox, would be in a nice position for this because their Kinect controller doesn’t require you to hold something in your hand to use it.

Microsoft could partner with the exercise equipment manufacturers to build free to download Xbox One game apps that make use of the Kinect sensor and the use of the equipment as part of fun, engaging games.

For example, a biking race game where you control the game, through the Kinect, by pedaling the exercise bike, and there’s nice HD scenery as you go along the race course, any one you like. Mountains, beside the ocean, beside a river. Ort famous race courses like the Tour du France. Ideally displaying in stereo 3D. You could have a training mode and a racing mode, which could offer either a short track race or a longer road race.

Or a running game / road race trainer game tied to major treadmill models. Go for a run by yourself or in some chosen scenery, either in nature or in some city. When you want to test yourself, you play a race mode that puts you in a famous road race course like a big, renowned 10k or the New York or Boston Marathon. Again, 3D rendered and ideally displaying in stereo 3D.

Or for the elliptical, it could be cross-country skiing.

Or a rowing game using a home rowing machine with well known scenes or race scenarios. For example, a game scenario where you train with the Harvard or Cambridge crew, row on an Olympic course, or relive some big race on the Seine in Paris from the early 20th century.

Make sure there’s an interactive display layer menu the user can access for exercise and training analytics. Also with some sort of virtual coaching, maybe something using interactive avatars. In addition, ideally a social network layer to share “achievements” or get encouragement from friends that are also on an exercise program. A space for monitoring vitals like heart rate over time and tools to manage diet and nutrition and suggest meals would also be useful.

Ideally, Microsoft would want to have someone working with the equipment designers and manufacturers to incorporate wifi connections or Bluetooth in the equipment so that the Xbox game, via the Xbox software, can wirelessly control the exercise equipment within some manufacturer and user set safety tolerances. Also, the other way, so that the equipment can wirelessly send the current setting to the game. So, for example, if you’re playing your running game, and you’re on the last kilometer of a 5 k race, and you want to sprint for a PB or to catch someone, and the software calculates that you’re not over-exerting for your age and fitness level, the system raises the speed of the treadmill automatically to match your attempt to go faster. Or for the bike or treadmill to automatically adjust the inclination when the game gets to a hill on a training or race course.

It would also be good for the games to be multiplayer, ideally multiplayer over the internet. Then people could go on at the same time and race each other on their equipment over the Internet. This could help additionally with social based motivation.

With the right gamified elements and incentives and feedback, you could help people make more effective use of fitness equipment in their lives, help them persist at it, and get fitter. The machines would be better used, and health outcomes could be improved over the longer term.

I could even see a nice marketing strategy for Microsoft for the Christmas holiday season. Make a joint marketing arrangement with the major home fitness manufacturers and TV manufactures and the electronics and home appliance stores.

Arrange to set up displays in store. Have the exercise equipment set up facing the biggest TV screen in the store, with the TV hooked up to the Xbox One at an appropriate distance from the exercise equipment and with a well positioned Kinect hooked up to the Xbox. Have some game running in multiplayer mode. People could try it out and have a little low intensity friendly competition right there in the store. And by juxtaposing the Xbox One, the TV, and exercise equipment in a way that shows them working together, you might well increase the sales of all three, benefitting the manufacturers of the devices and the store that sells them. Everyone wins.

Further Reading