UI designer Steph Chow discusses how to embed a game's world into its user interface (UI), and how to strike the right balance between player immersion and player usability.
​

Immersive game elements include characters, VFX (visual effects), environments and UI
 
UI is not just about bright juicy green buttons.  UI is part of an immersive strategy.
 
AR AND UI IMMERSION

• A big focus on camera visuals needs to be balanced by minimal game controls
• The balance between familiarity and fantasy is also important
• But there still needs to be a way to imply the AR world when the camera is off

 
VR and UI IMMERSION

• A huge opportunity for immersive, in-game visuals and physical interactions
• Animation, audio and haptic effects can also enhance player feedback
• But complicated tasks still require explanatory visual interfaces and accessible buttons.

 
ENGAGEMENT IN UI IMMERSION
 
Players play games not just because they are fun, but because audiences grow to love the unique worlds (fictitious universes) in which they are set.
Details that go beyond characterisation like typography, colour palettes, shapes and iconography can help to keep a player immersed and engaged.  Chow aims for a branded experience through both visuals as well as functionality.
 
The interface needs to be immersive and usable. 
The process of embedding a game’s world in its interface unfolds in 3 phases

1. Research, 2) Exploration and 3) Iteration

 

1. Research:

What is the visual culture behind your world?
This is the time to seek out the visual elements that make your world distinct.  This can be inspired by history e.g. the 1950s American iconography of the Fallout Series
Or by Nature e.g. Paradise Bay, reminiscent of oceans and rocks
Or by Subcultures e.g. Splatoon 2 takes its inspiration from punk references, including graffiti.
 
Chow recommends taking research beyond google searches – e.g.  watching movies, visiting museums

1. Exploration

It’s important to explore the spectrum offered by your ingredients

• Diegetic (included in game-world, so seen and heard by in game-characters) vs non-diegetic (only visible to players)

Diegetic is more likely to be fully immersive, easy to grasp narratively and preserves the 4th wall
But if you have a lot of information to show it can be buried in a diegetic look

• can help guide players through complex tasks by clearly separating game-play challenges from the detailed content.  Needs to be designed carefully however to avoid distraction and further complicating the task.

 
Deciding between these 2 approaches depends upon 3 things

1. Amount of platform screen space
2. The complexity of the game mechanics e.g. simple game mechanic vs complex strategy
3. How player is interacting with the game (touch, controller, camera)

 

• Skeuomorphic vs flat

Skeumorphic (incorporating non-functional design elements) creates a sense of familiarity by emulating materials whilst flat design stays true to its medium (ignoring colourful details to focus instead upon the inventory, for example, and its function in the game).
It’s worth exploring different layout options to judge the best approach.

• Layout

Can provide both immersion and usability – draws upon familiarity to improve clarity, makes functions and form clear in the visuals

• Animation

Actions can communicate timing urgencies, be pinned to interactions, direct player attention and create a sense of pace.

1. Iteration: To find the balance between narrative impact and usability

 
Things to evaluate

• Readability – Has the interface been over-designed to the point of confusion?
• Personality – Are the brand’s keywords visually implied in the design?
• Implication – Are interactive and non-interactive elements distinct and clear?
• Scale – What is the memory load of your design?

 
Also keep usability heuristics in mind

• Visibility of system status
• Match between system and the real world (e.g. using conversational language)
• User control and freedom (e.g. providing clear abort, or restart procedures)
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognise, diagnose and recover (e.g. with simple error messages that point to solutions)
• (Easily discoverable) Help and documentation

Chow, Steph. 2018/2020. Immersing a creative world into a usable UI U.S.: Game Developers Conference.  

An edited book that is a companion publication to the 2018 Brazil exhibition of standout recent interactive and immersive factual narratives in different environments and across platforms.  


Introduction to interactive factuals:

The interactive format is linked to the emergence of web 2.0.  Influential works include _Gone Gitmo (2009) a participatory Guatanamo Bay Detention Center environment in Second Life, Highrise (2009 – 2017), a group of documentary projects about high-rise living noted for its aesthetic innovation (where site design became as informative as text and video) and it’s collaborative, perpetually in development over time production model.  Other notable projects include Alma: A Tale of Violence (2012) a mobile delivery documentary about the violence of Guatemalan street gangs; Snow Fall (2012), an interactive rich-media news report about an avalanche told through an exploration of the location through a choice of skier viewpoints, which also had a narrative path driven by a vertical scroll that inspired the term ‘scrollytelling’; Prison Valley (2010), a docu-game where you play as a prisoner; The Quipu Project (2015), a social impact participatory storytelling project about enforced sterilisation that allowed indigenous South Americans to record and upload their own stories by phone; Do Not Track (2015) a personalised web series about privacy and the data economy. Digital Me (2015) that creates a personalised reflective document of people’s hybrid online/offline identities. 
No format prevails, and hybrids proliferate. 
The Guardian and NYT are both active in this field, as are broadcasters BBC (England), RTVE (Spain), SBS (Australia), Arte (France) and The National Film Board Canada. 
Developmental breakthroughs include relatively cheap headsets, like the Oculus Rift, plus solutions to the challenge of motions sickness, and the emergence of cross-pollinating creative production hubs and festivals such as HotdocX (Montreal, Canada), idocs (Bristol, UK), the IDFA Doclab in Amsterdam, as well as the Tribeca Cinema 360 and Sundance Frontier Lab Festivals (U.S.).  The first production studios emerged out of these labs, along with the Interactive Documentary workshop (Switzerland).  The main distribution platforms for VR content are SteamVR, Oculus and Within. In Brazil, the first interactive documentaries emerged about 2010, are socially engaged and delivered across platforms. 

Immersive Narratives
The exhibition includes the 360 visual experience, Through The Masks of Luzia linked to a Cirque Du Soleil production which allow participants to experience becoming gigantic, as well as tiny; Dreams of O, which combined underwater and slow-motion 360 video; and the immersive VR manga, Tales of Wedding Rings.

The future?
Other formats will continue alongside VR/AR, so complementary streams of media are also likely to continue their importance.

• The duration and complexity of VR experiences is likely to increase. 
• The future status of VR (which offers simulations and interactions via a variety of sensory stimuli, mostly audio-visual) is uncertain because the sale of VR headsets is now going down, not up. 
• But AR is growing, along with emergence of mixed reality headsets. Gaudenzi predicts new works with can be consumed as either AR, or VR. 

Production tips 

1. Put the user at the centre of your creative process
2. Do your research and know what has already been done in the field, as well as where you are placed in terms of style and trends.
3. Embrace multi-disciplinary teams and iterative production
4. Consider co-creation with your subjects
5. Have a clear impact strategy e.g. user impact, social impact, social campaigns – and design action points accordingly
6. Ensure coherency by aligning platform, audience, impact, partners and resources

Paz, A., Gaudenzi, S., Salles, J., Maciel, K.A., Gifreu-Castells, A., Cruz, J. and De La Vega, X., 2019. BUG-Interactive and Immersive Narratives.

A look back at the range of creative, immersive content experiences of the last 4 – 5 years, identifying patterns, trends and insights.

Findings include:

• Immersive media has the potential to be a creative medium in its own right.  
• Identifies 15 creative format trends/potentialities 1. Activity simulators, 2. Short fictions, 3. Data Visualisers, 4. Immersive maker tools, 5. Re-visualising testimonies, 6. Digital treasure hunts, 7. Perspective shifters that enable people to walk in another’s shoes , 8. Up close and personal encounters, 9. Audio journeys, 10. Behind the scenes access all areas, 11. Wonder in Education, 12. Fantasy trip, 13. Best seat in the house, 14. Virtual hangouts and 15. Enhanced filters
• Once the technology has been more widely adopted (which the report suggests is likely to occur soon), the largest audiences will be the early adopters and then late majorities.
• Thus far, the most successful creative formats with the audience have been 1. Immersive maker tools, 2. Activity simulators and 3. Fantasy trips.
• The most successful formats in general provided the material for audience members to create their own personal narrative, a format sometimes known as story-doing, or story-living.  Popular formats often involved the audience storifying their own experience. 
• The most successful formats also tend to generate a mix of both spatial and mental immersion.
• High quality graphics and animations tend to enhance the sense of presence.
• Standard scalability principles still apply, so that platforms, or tools like maker kits are more likely to scale.

​
​Allen, C. and Tucker, D., 2018. Immersive Content Formats for Future Audiences. Digital Catapult.