Augmented Reality Interface for NASA Lunar Missions
Validated AR Technology as a visual and interactive tool to support and automate the process of performing major tasks in space.
Time: 1.5 years
Team: 16-20 Designers & Engineers
Role: Team Co-Lead (managing both design and software teams), Design Lead (lead design efforts & train members)
Client: NASA
Tools: Adobe XD, Figma, Unity
The Product
AR task tools, information displays, and performance guides for the lunar surface
Including guided lunar navigation, vitals and suit controls display, geological sampling recording tools, and search and rescue messaging system.
The Challenge
Problem: How can we give astronauts autonomy to complete their tasks in space without the help of Mission Control?
NASA Spacesuit User Interface Technologies for Students (SUITS) selected our team to develop Augmented Reality (AR) software on the HoloLens.
An iterative process with communication and mentorship from NASA
Let's begin by talking to real Astronauts! They have a few pain points.
We talked to AR, voice & navigation designers, NASA Designers, Brown Planetary Geoscience Professors, and Astronauts.
James H. Newman
Former NASA astronaut
1. "The Moon is Difficult to Navigate"
-
Dangerous terrain with tripping hazards.
-
An environment with no distinctive features for landmark recognition.
Steve Swanson
Retired NASA astronaut
2. "Space Suit Limits Movements"
-
Whole-body movements are required to accomplish simple tasks.
-
Small hand gestures takes a large amount of effort.
Astronauts have 4 main objectives on the moon
1. Navigation
- Long-range point A to point B
- Short-range terrain mapping & obstacle avoidance
3. Staying Alive: Vitals
- Vitals and suit status
2. Geological Sampling
- Taking Samples
- Documentations
3. Staying Alive: Rescue
- Emergency Navigation & Communication
Designs
1. Automated Interactions
Navigation Selection
-
Minimized inputs by automatically opening the next features needed in mission performance.
-
Sets of navigation tools are launched once the route is selected.
Initial Design Before Iterations
-
Previous iterations require more hand activations and cause physical exhaustion to the astronauts.
2. Did More With Less
Navigation Mode
-
Too many tools become visually obtrusive for lunar task performance, therefore we merged multiple features into one with new solutions.
Mini-map provides short-range and long-range direction but takes up space
%20%E2%80%93%201.png)
Arrow provides short-range direction
Red Direction Indicator added to existing compass to indicate long-range direction
_Page_3.jpg)
Pathway replaced the arrow and mini-map to provide both short-range & mid-range direction
The pathway also shows approximate closeness to the destination with color change
Initial Design
After Iterations
_Page_4.jpg)
3. Unified Elements
Vitals Display & More
-
In user tests, we found more than 2 interfaces floating and following the users caused spatial confusion.
-
Therefore we unified interfaces to a single switch menu.
Initial Design
After Iterations
_Page_2.jpg)
4. Cleared the Field of View
Emergency Messaging & Geological Sampling
-
Interfaces can obscure users' fields of view. They were redesigned to open partially visible on the left of the viewing port to leave the space straight ahead clear.
Design Drafts
After Iterations
Experimented on interface opening positions to maximize the opening space in front of the user.
Testing proved that the edge of the frame doesn't exist in an AR space. An interface that opens to the left and is partially obscured would still be translated as a complete interface to the user. This allows the space in front of the user to be opened up.
5. Overlapping interfaces are distractive!
Set separate use cases for each plane of interaction
Divided features to 3 interaction levels on the HoloLens
Registered in the environment in a set location for guidance during task performance in the environment.
Sticky interface registered in the environment following the user's view for decision-making interfaces.
Registered to the hud following the user's view for frequently used display tools.
Initial Design Before Iterations
-
Users' attention can only focus on 1 plane of distance at a time and will miss elements in the far environment when they are interacting with the interfaces in front of them, and vice versa.
Design Process
💡 1. Learned to eliminate
We couldn't do everything. I learned to ask myself and the team, is this function/feature necessary to achieve our goals? If not, let's put these aside into our future features bank.
💡2. Learned to adopt structures for project management
I found that for this project of 20 members, narrowing the type of decisions each member has to make creates a more efficient and less confusing experience for everyone.
Example: We separated key functions into teams. A separate user flow was created for each and responsibilities were distributed by dividing the tasks on the user flows.
.jpg)
💡 3. Learned to implement early and fast!
We learned quickly that too many false assumptions were made when we didn't test our designs on the device early. I adjusted our timeline so that small adjustments were tested on the HoloLens before we refine the design details.
Testing AR by simulating floating interfaces with cut out papers
Software behaves on the HoloLens entirely differently than how it behaved in Unity.
Figma Mockup
Designs behave in Unity entirely differently than how they behaved on Figma.
NASA Testing Recording from the HoloLens
Recovered footage from the HoloLens during testing on the simulated lunar environment at NASA
A week of testing & software iterations at NASA Johnson Space Center
Core Leadership team at JSC in Huston! My co-lead and I, and our software leads
NASA evaluator Skye tested our software while performing mock astronaut tasks at the simulated lunar environment.
Working on code well into mid-night at the hotel.
Showcasing our designs to NASA engineers
Creating an interface controls panel as a backup after we encountered poor lighting on our first dry run on the test site (The HoloLens doesn't run well in poor lighting environments). Thankfully we didn't have to use this later.
Validations from NASA
Very intuitive and clear user interfaces
"The buttons worked well, I can see the tools on top of each other, and the layout was intuitive. The vitals screen looks clear."
Have just the right tools needed
"Having an additional map that opens up is helpful on top of the 3d Line(pathway) "
"The line(3d pathway) was pretty cool, it led me right to the destination."
Fast learning curve and a easily understood structure
"I like having a menu that you can collapse to. Having it on the left just made sense."
Moving Forward...
1. Explore AR interactions that we haven't played with! Actions such as pinning an element to a user's body part.
"It would have helped if the map could be pinned to my lap."
2. Design more feedbacks!
"It would help to have distance markers along the path, and more precise distance summaries during navigation".
3. Design for more scenarios
"When key consumables from the vitals screen are in low supply, a way to highlight that to the astronauts through a change in color or an alert would be helpful."