Hands-on with HoloLens

I’ve written a lot here previously about the potential for Augmented Reality, specifically the HoloLens, introduced in 2015 and released as a developer edition in 2016. However, the HoloLens being expensive ($4000 in Canada) and a little hard to find, I haven’t previously had the chance to use one hands on. Last week, thanks to a vendor demo at work, I was finally lucky enough to get my hands and eyes on this amazing piece of technology. Here I share some of my observations from this brief hands on session.

Note that I only got to spend something like 30 minutes of time directly with the device, so there was not time for a full and comprehensive review, nor for any deep assessment of how well it would fit into an extended use workflow. But I explored around as much as I could in the time I had.

Strengths

First of all, the strengths:

  • The refresh rate and power seem good. I didn’t notice any lag or stutter while moving around a relatively complex 3d image
  • The device features a decent amount of storage memory. The device has a nominal memory is 64GB. You end up with somewhat less with the operating system installed, but you still have a decent amount of memory.
  • The main interface is similar to Windows with the tiled layout. Menu interfaces are similar to Windows, Xbox One, etc. It runs a special edition of Windows 10.

  • The voice interface and voice recognition seemed to work very well. The interface for the OS uses Cortana voice assistant as one of its main interface modalities (along with gestures). You can use voice commands to launch apps and other basic tasks.
  • The operating system comes with a nice tutorial app to help the user quickly acclimate to the user interface. I didn’t get a chance to go all the way through this tutorial, but what I saw was good.
  • The device has connectivity for Bluetooth and Wifi. Can connect a Bluetooth keyboard. Allows you to use typical apps for productivity, serves as a screen. I did not have the chance to try this out.
  • Battery life is not bad. About 3 hours with active use. I see some reviews online reporting up to 5 hours, depending on the type of usage. This is not enough for a full workday, but enough to say, get you to lunch, you plugin to recharge, and then come back after.  Recharging apparently takes a fair amount of time. However, the HoloLens can be used while plugged in if you’re sitting in one place, which could help extend the length of a session.
  • Video capture was smooth and easy, and will capture your first person perspective, including the apps and holograms in your view. Microsoft calls this “mixed reality capture.” The device features a 2.3 megapixel video and still camera.
  • I found the device to be reasonably “glasses friendly.” I wear glasses, and sometimes VR headsets can be a pain to make sure the two layers of lenses between the digital goodness and my eyes cooperate and align, but this worked well.
  • The device was also reasonably comfortable to wear. It sat well on my head, and was not noticeably heavy. Some VR goggles I find a little front heavy, but the HoloLens is lighter and doesn’t project as much outwards in the front.

Weaknesses

The device is not by any means perfect, however. This is a first generation developer edition for developer experimenters and creative professionals. It’s an R0 product. It’s a lot of the way toward where it needs to be for a widely usable consumer device, but some things could benefit from a few more years to tweak and take advantage of the ever falling prices of hardware to improve the specs and performance. A few shortcomings with the current edition:

  • The vertical field of view is a little narrow.  It’s far enough from your eyes that it only takes up part of the field of view and the edge is visible. If you’re too close to a large holographic object, it will just end at the edge, which breaks the illusion. Perhaps later versions will address this with a screen that fills more of the field of view.
  • Resolution is a little low at a max of 720. It would be difficult to do very detailed 3D models. Hopefully future models can bump this up to at least HD.
  • The device seems somewhat limited in terms of the basic built in gestures. Tap to click on something,tap and hold to drag, resize, or rotate an object and bloom (open your fingers with your palm facing you) to launch a menu. I would be interested to see if the development kit allows you to build new gestures. I believe the Kinect dev kit has a tool for this.  The only thing is that if you build new gestures, part of your work would be to seamlessly educate users on the gestures. All in all though, I did find it relatively intuitive to pick up the basic gestures, even without the built in tutorial.

 

  • The RAM is somewhat limited – 2GB for the CPU, and 1GB for the “Holographic Processing Unit,” a specialized GPU of sorts for handling processing of 3D graphics. The CPU specs are at the level of a smartphone or tablet, so despite the fact you can connect a keyboard, you wouldn’t be able to run very heavy apps. That’s something that would be good to work on for later editions, because one of the places I could see HoloLens ultimately being useful would be as a replacement for a desktop computer and screen. More RAM would be needed to make that feasible.

Apps available

There are a number of apps available. Some of these I was able to try out, others I was not. The apps available include:

  • Netflix
  • Edge browser
  • Camera
  • Movies and Tv
  • Skype
  • Music
  • OneNote
  • OneDrive
  • MS Office
  • HoloStudio
  • Minecraft

Basic Functionality

The HoloLens responds to movement laterally and vertically through space, and to 3 axis rotations of your head.

Windows that are launched initially hover a few meters in front of your eyes, and by default move to stay in your view as you move and turn your head.  However, you can also point and click to pin windows and objects in place. There is some smartness to this; the system software identifies flat surfaces, whether horizontal or vertical. If you turn so that an object or window is hovering above a table, and then click to pin, it will lock to the table top. If you hover a window over a wall, and point and click to pin, it will lock to the wall. You can similarly point and click to unpin. Picture something like pinning an app icon to your desktop or the toolbar in Windows or MacOS, but with the desktop now being now any planar surface in your  surroundings. It’s hard to understand how cool this really is until you try it, to spatially arrange your computing within your environment.

Even more interesting is that the system can remember the placement of Windows and holograms between sessions. For example, you could pin a Netflix window to your living room wall. Voila. Virtual TV. Walk away into another room or to another floor and come back, and it’s still there.  Or rather, it remembers that it should be there. And it will still be there the next time you fire up the HoloLens. There are so many possibilities for this.

Cameras within HoloLens map the environment as you walk and gaze around. This creates a 3d model of the surroundings. If you walk around a building or house, it will map all the rooms visited and store a map of the area.

The HoloLens has built in speakers that allow for an immersive spatial sound experience. Sound sources can have directionality, just as in real life, and sounds attached to holographic objects will rise and fall in volume and adjust in perceived position relative to you as you move closer, farther away, and go around the virtual object. This enhances the immersiveness. And while the speakers in the device manage a good volume to let you hear, it doesn’t seem to project significant noise to those nearby.

Unfortunately the HoloStudio tool for crafting 3D objects did not seem to be installed on the unit I got to play  with, which was a minor disappointment. This is an app I was pretty excited about. Nor did a get a chance to check out the experience of browsing and downloading apps from the store.

Nor, sadly, was the amazingly cool looking Minecraft app installed. (First thing my son asked about when I told about my hands-on)

The Path Forward for HoloLens

All in all, I came out of this brief test of the HoloLens extremely impressed. This is an amazing first draft of this technology. Even with some minor shortcomings it nevertheless reinforced  my earlier perception that Augmented Reality is going to be a key part of the future of personal, creative, and work computing.

This first release of the HoloLens is a developer version is not meant for mass adoption. It’s a first step in a planned series of releases. Previously there was an intention to release an incremental v2 in 2018, and a v3 in 2020.

However, recent unconfirmed reports indicate that Microsoft will instead focus effort on speeding up the v3 release by skipping v2 and refocusing efforts on a big jump with v3. V3 is expected in 2019, and, one would presume, this is where the tech would launch in a more powerful and hopefully cheaper consumer oriented model.

I for one am looking forward with anticipation. Until then, I can only hope I will have the chance to spend more time exploring the the possibilities of this new mode of computing.

Another review:

http://www.vrheads.com/after-24-hours-inside-microsoft-hololens-i-dont-want-take-it-again

Previous Augmented/Virtual Reality Articles on this Blog

Virtual and Augmented Reality and first person media

On the Microsoft HoloLens

Augmented Reality and Wearable Computing: Possibilities for Google Glass in Training

Concept: Maintenance Training 2030


April 25, 2032 …

Ed, a Canadian aircraft maintenance training instructor located in Montreal, gets ready for his upcoming engines maintenance class.

Supposed to be a big class this week, 8 students.

  • 1 from Canada
  • 1 from the US
  • 2 from South America
  • 2 from India, and
  • 2 from China

Ed checks on the equipment in the classroom.

He puts on his Holo-Glasses, which come to life, softly glowing holographic data displays and icons popping up in front of him.  The device recognizes him, launching the virtual assistant to greet him. “Hello Ed! How are you? All set for your class?” “Just fine, thanks. Everything set?” “Yes, Ed. All the students are going to be attending; no cancellations. Everything looks good with the students. One was having some technical issues earlier, but I helped him through it.” Excellent,” thought Ed. “Everything looks alright with 15 mins to go.”

Ed begins cueing up the opening presentation notes, and the multimedia training manual. These pop up in their own windows in Ed’s field of view.

As Ed continues his preparations, the digital assistant relays notifications confirming the status of the students. The assistant is communicating with the students before class so Ed can focus on his preparation. Everything is looking good. Ed checks the 3D cameras and tests out his holopresence projection, seeing what his students will see.

“Loo-king good! Let’s do this!”

A few minutes later, the class begins. Ed welcomes the students as they holopresence in from their remote locations. Ed and the students, their Holo-glasses on,  take their places in the shared virtual classroom. The software places softly glowing holographic representations of the other participants in the shared visual space.  Ed looks out at the students’ faces, and the students see a holographic overlay of the same classroom and the same students from their own virtual perspective. At first, the experience is a little eerie, but as the class gets going, and all the students introduce themselves, the illusion takes hold and it feels like everyone is in the same classroom.

Ed presents the content, asks questions, and listens to the responses. Master teacher he is, he observes carefully, gets a sense of the learners’ body language and expressions, and, much like in a real class, adjusts as he goes. Ed brings up holographic 3D animations and models of the engine and components for the class to see. He zooms, rotates, and takes apart the holographic engine parts. The hologram also appears in the students’ fields of view, and Ed invites students here and there to come up and try for themselves and demonstrate actions to the class. Static images appear on screens in mid air, demonstrating schematics.

In the afternoon portion it is time for the virtual hands-on lab exercises. Ed and the students convene again, once again with beautiful, interactive 3D holographic models of the engine floating in the shared digital overlay. This time however everyone puts on their SureTouch(TM) haptic feedback gloves.

The gloves use sensors to read finger and hand position, the headset measures their hand positions in relation to the digital model’s virtual position, and actuators in the gloves give pressure feedback to simulate handling real objects with substance instead of just weightless holograms. It’s kind of weird at first, and it’s not quite the same as the real thing, but close enough for horseshoes and hand grenades, as they say. And definitely a hell of a lot cheaper than taking an actual engine offline to train.

As always, it took a few years for the technology to perfect itself and a lot of research and proofs of concept before the regulators really believed it could be as effective as the real thing. The Dutch Aerospace Lab did some great research as always, and once EASA signed off, the other regulators followed pretty swiftly after. Regulators came to appreciate virtual maintenance training, just as they came to appreciate the power of full flight simulators decades before.

The company definitely appreciates it too – they save a small fortune in flights, hotels, taxis, and per diems doing virtual classes like this over the course of the year. As do the students’ companies..

Ed for one, appreciates it too. No packing, no airport security, no  cramped 12 hour flight, no hotel room, no taxis, no jetlag, no traffic. Well … scratch that last one. This is Montreal, after all, where the seasons are winter … and construction. Even in 2032, there’s plenty of traffic. (You can’t win ’em all, I guess.) “Oh well, ” thought Ed. “Decent weather today, so at least could read a book on the way in while the autodrive on the car took care of all the unpleasantness.”  And all from the comfort of the Montreal office.

Ed loves it, and his family loves it too – less time away. And besides. even though he felt a little silly to admit it, irrational as it was, Ed had felt a littled weirded out by flying ever since they started the rollout of unpiloted commerical flights in the late 2020s. Hundreds of times safer than human pilots or not, it’s still kind of creepy to have algorithms flying you around instead of people.

“Or maybe I’m just getting old, ” Ed thought. Gets a little jarring after awhile to see the world transform itself before your eyes so quickly. The young seem to take it in stride, unphased, as they always do. And, Ed had to admit that the toys are pretty cool. All this change has its benefits.

Such is the stuff of life in a world of sci-fi dreams made true.