mixed realiry UX Design Challenge

Screen Shot 2018-02-12 at 20.29.20.png


This project is aiming to bring an artist gallery or studio work into the holographic world.

Design Challenge (Click Link)

The Holographic Portfolio: New Ways of Experiencing Art Works on the HoloLens

Imagine a holographic portfolio that enables you to experience and sense art works in a new way. Design a proof of concept for a holographic portfolio that covers the end-to-end experience of an artist's portfolio to be viewed on the HoloLens. This is not about creating an “artwork” for the HoloLens, but rather a way to interact with different works in mixed reality on the HoloLens. The approach should be based on the experience of a studio visit.

Project Goals

During the initial steps for the project, our group members individually went to the Microsoft Store for the 1-hour demo sessions with the HoloLens. From this experience, we learned a lot about the limitation and capabilities of the HoloLens and tried to synthesize these experiences into something that we could use for our project. We also developed these project goals to help guide us:

  1. Create a simple and clear way for a curator to view an artist’s work.

  2. Provide a consistent experience when interacting with holograms.

  3. Design an unobtrusive and intuitive way to navigate the experience.


Worked as UX Researcher / UX Designer in a team of Six.

Group members: Brian Hui, Alexandros Lotsos, Tyler Mechura, Jing Huang, Ankit Ruhela

Instructor: Dana Karwas

Client:  Nicholas Kamuda, Creative Director for Microsoft HoloLens

Joshua Walton, Principal Interaction Designer at Microsoft HoloLens 


Design Process

Researching & Brainstorming

Before we started the project, in order to better understand the technology, we signed up HoloLens DEMO and visited the Microsoft Store of 5th Avenue in NYC.  We were introduced and shown three different HoloLens experiences, and through those introduction, we saw the applications and limits of the technology, which gave us a good sense of the capacity of this technology before we designed the project.

Me trying HoloLens in the Microsoft Store of 5th Avenue in NYC


During early brainstorming phase, we developed a concept map of the many ways we think users could interact with the holograms and other features in HoloLens. This map is probably going to guide us when we start talking to curators and artists. We also sketched a few of the ideas to help us visualize how this would work. From there, we created an initial Table of Elements to help us determine what we wanted to include in our experience.

Concept Map

Table of elements for the design

As for design techniques, we plan on focusing on World Locked Presence, True Scale, and Persistence, as well as Waypoints and Sense Ratios. We hope that with these techniques in mind, we can combine it with a few usability practices (like having minimal UI elements so the user isn’t too distracted and having audio and visual cues available so the user knows when things are happening in the experience) to fully flesh out this design challenge.

We made some sketches for visualizing our ideas.

Comparative Analysis  

During the early stage of design process, we also made the comparative analysis, trying to figure out what are other similar products and the existing programs, with different devices. 

We looked into other art museum, gallery experience in AR, and VR, across different platforms like HTC VIVE, Oculus Rift, and mobile devices. The things we found among the researching were they were even those experience was immersive, but they were all lack of enough interaction, and could not provide enough information. So gaining those insights really gave us more things to think about, and how are we going to integrate those features in our design   

Comparative Analysis Diagram

User Flow of HoloStudio


Prototyping & User Testing  

After the researching and brainstorming stage, we made several sketches of our ideas, and the good way to test them out was to build them out, so we moved on to the stage of prototyping and user testing.

For the prototype we made, I think what we did was really helpful for the project. Before making the first prototype, we still spent much time discussing different approaches for designing the project, even though we already had some ideas last week during the brainstorming.

After that, we came out some simple user flow and based on that user flow we started to make our first prototype. I must say it was a really fun experience prototyping such mixed reality design. Only by using some paper and drawings could clearly show our ideas. I think based on this prototype, we could go talk to more people and get feedbacks for next iteration and further digital prototypes.

1st Prototype

After showing our design idea with the first rough paper prototyping video, we gained some good feedbacks. So this week, we tried to think out of box and made some iteration of the design. The idea we came up was having a digital assistant, floating beside the user during the whole mixed reality experience. In order to cut down the user interface elements, we tried to take the advantage of 3D sounds and the voice control. So basically, the digital assistant would guide the users and give instructions at the beginning, then the users were allowed to walk around to see the art pieces by themselves. If they had more questions, they could ask for help from the assistant at anytime.


2nd Prototype & Iteration 

We made another paper prototyping video of this idea, and also conducted our first user testing this week. The users did give us some feedbacks about the digital assistant, for instance we need to be careful for designing the image of the digital assistant, not to let it distract the users while seeing the art pieces. Over all, the users said they like the idea of having a guiding assistant, and the user flow was quite smooth and easy to understand. This idea made me think of the little alien baby character in the video game in the movie “Her”. I really like how that small guy talked to Theodore. Talking to such a animated avatar was funny but would be really cool to have one. I believe the idea having an A.I. digital assistant was not new, but how to implement this idea into designing a product and let the users enjoy it was another challenge. I think we would conduct more user testing and see if this idea would be our final design. Also, how to develop this concept would be another challenge for us too, and hope we could overcome the technical issues eventually.


3rd Prototype & Iteration 

After having those insights and the feedbacks from our user testing, which we let four different people try our first paper prototype, we started to work on the iteration for second prototype. The major changes we made was taking off the adjust buttons and putting the little signs instead. Also, we made a new version of our digital assistant “Arty”. We drew several possible solutions for Arty, and in order not to be too distractive to the users, we designed it like a flying little ball with two wings eventually. We haven’t tested to other users with the new Arty yet, will like to do that this week to see how people think about it.

Also, we decided how would the further information be shown. We would bring the users to another virtual room, without other artworks, only with the one they select to see further information and its information would be displayed in that room, and they could go back to the gallery anytime when they finished seeing the whole further information, like the progress of that artwork.

We tried to user test our second prototype, and gathered more feedbacks. To be honest, I found it interesting and difficult to conduct such mixed reality experience user testing with only paper prototype. It would be easy to show people your idea through filming a short prototyping video. People only need to watch what happened in the clip and they could get the idea. However, if we want to let users try out by themselves, many details need to be thought and concerned. We need to tell the testers what exactly they could do and what they couldn’t do, and at the same time, we also need to make sure not giving too much information, because what we’re looking for is the fresh experience and feedbacks from them.
I think we received some good feedbacks from the testing and a lot of them we did not think about before. At  this stage we could not make too many changes, but we still tried to make some iteration based on those feedbacks, for instance, the design of how to show the menu buttons. Also, we got pretty good feedbacks from user testing about the digital assistant idea. People love to interact with Arty and they depend on it a lot. That was an interesting takeaway for me, because sometimes even though I think Arty, the digital assistant idea, was cool and fun, I still have some concerns about how willing and comfortable will people have interacting with Arty when designing it. And this questions then lead me to think more about how important it might be for having a digital assistant in mixed reality world. On mobile phone, I don’t think people rely on digital assistant a lot. A lot of big tech players are pushing this idea, for instance: Siri, Google Assistant, Cortana, and Alexa. People are getting more familar with the digital assistant and even though not using them often, but I think in the future, in virtual reality and mixed reality world, people will get more comfortable with using digital assistants and using them more frequently.


Concept Visuals


Painting an Artist Information

Performance and Installation


We made some render images to visualize how our idea of having a digital assistant Arty in holographic studio would be like, and also how the user could interact with the holographic art works, including sculpture, painting, installation, video, and artist information.


Visual Flow Design

During the time we did prototyping user testing, and iteration, we also kept testing our user flow, to see if there's any process that did not make sense to the users. We finalized our user flow then made it into a visual flow (the video below). Instead of the traditional user flow, the visual flow is animated and it could clearly visualize how the whole experience would be like in the space. We used different colors to differentiate the real object and holograms in mixed reality. That was something came along when we were figuring out how to design a mixed reality experience, with simple sketches and colors.   


Interactive Prototype - Unity Development

After paper prototyping, user testing and finalizing our user flow for the HoloStudio. We worked on the interactive prototype in Unity. The goal was to provide an interactive prototype with holographic art pieces in mixed reality and show how our digital assistant could enhance the touring experience. The final version of the prototype is rough but polished enough to feature a full set of gaze, gesture and voice controls, an alpha build of Arty and at least one completed process room.
In the most complete version of the Unity prototype, Arty can successfully navigate the user through the studio selection process, give a tour of the selected studio and then listen for, and respond to a basic set of voice inquiries from the user. If the user would say "Hey Arty, tell me more about the sculpture" for example, Arty would reply with more information about the sculpture. 

Documentation Booklet & Article

Final documentation booklet for the project: Hellolens Book, Holographic Studio

Medium Article: 9 Methods for HoloLens & Mixed Reality UX Design 


Final Demonstration

At the end of the semester, we demoed our project to Nicholas Kamuda, Creative Director for Microsoft HoloLens and Windows Holographic at Microsoft. He gave us several useful feedbacks like we could think more about how user could interact with the digital assistant, which relates to designing an experience with artificial intelligence, and he also mentioned our design process reminded him some of their early prototypes with HoloLens in Microsoft. He also delivered a speech at NYU Tandon's MakerSpace about what work and projects Microsoft HoloLens currently are working on and also his thoughts about mixed reality's future.

Final Presentation & Demo

Photo with the guest of client project --- Nicholas Kamuda


NYC Media Lab Summit 2017 Demo, Sep 28th 2017

Featured on NYU Tandon School of Engineering News : Visionary VR/AR Projects From Tandon Community Featured At NYC Media Lab Summit

We demoed our "HoloStudio" at NYC Media Lab Summit 2017 demo day and more than 30 people have come and tried the prototype.