Narrascope 2019 - Panel Discussion - Dissecting the BanderSnatch With a Vorpal Blade
A panel discussion from the Narrascope Conference, Boston, June 15+16 2019 - Celebrating Narrative Games “Dissecting the Bandernatch With a Vorpal Blade: What Netflix’s Choose Your Own Adventure Got Right and Got Wrong” was Narrascope’s panel discussion. Heather Albano, Mary Duffy, Jason Stevan Hill, Emily Short, and Ian Thomas discuss Netflix’s first venture into the world of interactive television with Bandersnatch, the Black Mirror episode that let viewers influence the story in a choose-your-path adventure kind of way. What was good, where did it fall flat, how would experienced game designers have tackled this kind of project? But beware, there will be spoilers!
DIFFERENT AUDIENCE REACTIONS
Reactions different wildly from those familiar with the format, and the many audience members who had never previously been introduced to it.
Familiarity was again seen to be an important design tool.
CONTROVERSIAL CHOICE DESIGN
Not all choices made sense, or appeared to have purpose
THE IMPORTANCE OF ENDINGS
Viewing experiences were very different according to the endings reached
Too often the reality of a context was undermined, when a more interesting solution could have been found.
THE IMPORTANCE OF INTENTIONALITY
The writers could have done more to explore intentional participation through choices
GIVE PLAYERS TIME TO CONSIDER CHOICES
The decision to freeze action and give players time to make choices was generally received well.
The option to step back in time was innovative, handled well
Whilst branching narratives are complicated, production load can be minimised by techniques to group potential responses thematically
UNCOMFORTABLE SOCIAL MESSAGES
A number of panel members questioned social assumptions within the game
DETAILED DISCUSSION TRANSCRIPT
As a gamer, what did you learn? And what did you hate?
DIFFERENT AUDIENCE REACTIONS
By the time `I watched it there were already a whole lot of game designers who had seen it and panned it. Where's the innovation they asked? … and yet at the same time there was a whole bunch of consumers saying, Wow, this is exciting. We've never seen anything like this before… is a really interesting disjunction, I think. And so my first real reaction to it was that it introduced a lot of new people to interactivity, and that's a pretty major plus.
They clearly built this to feel familiar for television viewers, the way it handles recaps, the way it handles changing audio and those kinds of elements. There was a lot of television craft that went into it. And that's part of what makes it accessible to those kinds of people. As a result I found myself a little bit more sort of forgiving of some of those elements.
CONTROVERSIAL CHOICE DESIGN
But also, I got very frustrated really early on with the first few choices that just didn't seem to do anything very much, with no immediate consequences that just railroaded you straight back in to the narrative.
I felt sympathetic to the possible design reasons for those choices because it felt like …We've got a TV audience, we need to get them used to the idea, they might even need to pick something. And if they miss and they fail to make a choice early on, we don't want their first experience with this style to be …oh, I missed out on doing something interesting because I didn't really understand the controls.
It’s interesting to think about it in terms of a product that has to teach its players how to play. It reminds me of something I encountered when I did customer support for Choice of Games. When I first started working for them, I would get emails from people basically saying “Please put a back button to your games, I picked the wrong choice, and I have to do the whole thing over again”. And I would diligently respond that there was no wrong options. But there are arguably many points during Bandersnatch where viewers can make the wrong choice. In fact audiences proceed directly into what seems to be the most important choice right off the bat in Bandersnatch, namely… Should I work in the Office of Tucker soft games? Or should I work on this at home? And everything about the way the scene is constructed, says, work at the office, you're going to have all this support that you don't have as an independent game designer, and that's the wrong choice. So I'm so frustrated by that particular moment. It seems to be teaching audiences that their choices don't matter. Because what seems obvious to you is wrong. And so, you know, in a game that's trying to do all this layered metaphorical stuff about control…That's the first real choice. And it's tricking the audience. And that was such a turn off to me.
I agree 100% . When I hit that first choice, and went into the office and died, it's like…really? So I recognize the cultural impact that it has. Introducing lay people to interactivity is cool, but in many ways they’re replicating what I consider to be bad habits that I want to move away from as a design culture. So, as a designer and as a player, I myself did not care for it.
Okay, that's fair, but I actually had a different reaction to that first choice and the shortness of the path. I thought it was an interesting design choice because if you have players who have never seen anything like this before it teaches them right up front that this genre involves replaying the same scenes and making different choices. In that context I thought the confronting delivery of the need to reply choices was forgivable.
THE IMPORTANCE OF ENDINGS
I also found that personally, I had a different reaction to the experience as a whole depending on what ending I got. And I feel like that's something that might be interesting to delve into.
Me too. I enjoyed the piece more than a number of my friends because the ending that I got was the one in which he winds up being reunited with his mother and sharing in her death on the train. And that, to me was much more thematically resonant than many of the other options because I was much less interested in sort of gotcha endings where, you know, he fails to make a good game for some silly reason, or, you know, the whole thing devolves into this weird murder plot. I felt like the business about what has happened to him in the past and how he can revisit that trauma was a story that was less obvious to tell based on the form.
So what I mean by that is, I felt like a number of the paths through Bandersnatch were basically things where Rooker was asking questions about freewill or about you know, what does it mean if you have somebody, a player controlling the protagonist, which basically have been asked and re-asked by one interactive narrative designer after another. It’s almost the first thing that people do when they realize that this form exists. So, what I found most interesting was the case where the story it actually kind of gone beyond that set of probes into something different.
I agree. I got the same ending and I think it was an extremely poignant ending… choosing to go and die with your mom on the train. That was like a good Black Mirror episode to me. But when I was looking at the different endings that ending did not appear to be common.
Each ending almost catapults you into a different universe. In some endings you are definitely controlled by a person. You’re making a movie. Actually, you step out of it, and you're the actor or the Stefan that you are controlling is an actor on set, which is a totally different universe from the other universes where you die, or you're the mother. So there's Something really strange about different roles stepping out beyond the original fictional frame and ending the story outside of itself. Great for the cognition of the player, because the player has built an idea of the world that they're in, and then suddenly you shift that from under them.
That kind of universe switching approach can be something where you're kind of asking the player, what kind of story do they even want out of this? What are they interested in? But the trick I feel in the case of Bandersnatch, is that it's, it's hard to actually articulate that agency As a player, although I had ideas about what I wanted to explore, there are only a few points where I really felt like I had the agency to articulate that to the system.
THE IMPORTANCE OF INTENTIONALITY
Not me, one of the things with choice games that we really tried to focus on is the idea of intentionality, that you have some idea of the possible consequences of your decisions. And you're in there with the therapist and you get into the fight and then it's like, jump on your dad or jump out the window. But if it turns out that you're just an actor that's forgotten the fact that you're an actor… that complete lack of intentionality was frustrating for me.
Also, if you have X number of options, and the player is failing a number of hidden tests, so that only one option appears instead of all 10, there's something wrong. And I question what that's supposed to do for you as a player in Bandersnatch.
Yeah, there were a couple of choices that seemed to duplicate each other. I mean, throw the jewels at the computer or destroy the computer. I don't want to do either of those.
And there are some that are explicitly the wrong choice where if you decide to throw the towel on the computer, it immediately brings you to try this again or go back or something like that, which I hated. I thought that was a cop out if you're going to give the player choice, it should be a choice that goes somewhere.
GIVE PLAYERS TIME TO CONSIDER CHOICES
That said, there are some interesting things in those choices just from an execution point of view. We've all seen all the FMV games where you hit a choice and everything stops while you make a decision. So it was quite nice that they had the kind of live feed happening in the background and that during that the character you were responding to would generally prompt you and put a bit more pressure on which is interesting. But there's also a flaw in that which is, it's like the choices out of walking dead where you have three choices actually left to right or fail to act. Bandersnatch had that as well but didn't really seem to do anything with the fail to act option, other than auto-choose on your behalf, which was just a bit strange because it meant that you could just sit through it and sit back and watch the whole thing and it kind of loses the point a little to me because then you're not in control of anything.
I think there is a different emotional effect since you have as much time as you want to sit there and think about what you want. And so it's, you know, fail to act means something different than if you've been able to think about being inactive, and that's my choice, right? Whereas if, you know, you're sort of sitting there and and you're troubled by the question, and you're having a hard time making up your mind, or you somehow don't really want to commit and your time is running out that feels different. Yeah. So I'm a little bit more patient with it in a timed context, and I think it's more expressive actually. So it wasn't so much that I wanted a third option to press. Yeah, it was more. I want there to be a consequence of failure, like an unexpected ending, or something….
And I guess we should talk about flashbacks as well, because they were effective. Going back through the loop, and shortcutting all the sequences and seeing the results of some of your choices worked really well, brought it into a TV form really well. But, but there was never any attempt to explain in the fiction how that happened. I mean, you can look at something I like that the strange reversal of going back through choices is very explicitly laid out in the world. How are we repeating this however? Do the things I did last time apply to this loop? That's cool, but it lost me as a player. What is the friction? What's the infectious reason for this strangeness?
I'm struck by how differently I reacted to this emotionally depending on which of the endings I happened to strike. I didn't get death first, although it's my favourite because I agree with the emotionally resonant. I got, history repeats itself, where pearl ends up finishing the game, which I actually thought also worked as a black mirror episode in a very different way that it was very intellectually clever rather than emotionally resonant. I'm good with either ending, the one that annoyed me to death was the Netflix one that just struck me as the kind of thing you write at three o'clock in the morning when you're a bit blocked and it seems funny at the time.
I was listening to Charlie Brooker the other night talking about that, and that wasn't supposed to be a main choice, it was supposed to be a bit of an Easter egg. And they went through testing and decided to put it in.
And then I found the government control paths, which Stefan mentioned and the whole thing If that path felt to me, like they'd hacked it in that weekend, and that was when I started to wonder how deliberately the creators were commenting on different types of interactive fiction. I suspect that they haven't read that too deeply enough into the various types of interactive fiction to be that explicit about it. I think a lot of the impression I got was they wrote a huge tree in twine and then chopped lots of bits off when they realized it wouldn't fit into production.
When I was replaying it this week, I wanted to follow through all the leaves have an individual branch, if you will, and I was frustrated to that degree where it would take me farther back than I wanted to go and not let me explore all of those leaves. That said, I thought that the recap in particular was the best part. Like that was a really an innovation. It's not something that I'd seen before that I think will probably become a gold standard for this type of format in the future.
THINK ABOUT WHAT CHOICES MEAN
One choice context that I found particularly effective was the kind of locked, safe situation where the story presents you with the protagonist’s father's safe. And there are different storylines that could lead you to think that different things might be inside. You might think that it's about this government control program. You might think that it contains something that's quite personal. And you're offered options about what combination to use to unlock the safe. And it presents the task as though it's a puzzle and as though you're supposed to solve it by knowing what the combination ought to be based on what's happened previously. But actually, all of the combinations are words that connect somehow to what's been going on in the story. So the choice that you're really making is what do I think is important enough to be the thing that might be in the safe at this moment? And so, although it disguises itself as a puzzle, it's actually asking what do you as a player care about, and that I thought was quite cool.
It actually works better than it would have done if there was a parser type answer, because it's making it really clear there are potentially multiple right answers. And if you revisit that choice point, having come through different avenues in the store, you're actually offered different combinations solutions, which lead then on to different discoveries in the safe.
I think that's what the fight sequence is about as well…what type of game are we playing?
Yeah. I managed to miss enter that code through clumsiness and get it wrong in playtesting. Just not enough people were getting the number. I also had to be much more explicit about it and retire that as the main puzzle from what I gathered from the script,
Okay. But the question I got about the whole thing, actually, in the podcast. They interviewed Charlie Brooker about Black Mirror in general. And the impression I get from that was that Netflix said, Would you like to do this interactive fiction? They went, No, we don't want to. That's really nice. But we want to leave that. And then came up with an idea about a Black Mirror episode involving control and agency, and then did it the other way around, and went, Oh, no, this has to be interactive. Which at first involved trying to build this massive, massive branching tree narrative, and then production reality hit that massive tree. And so, what we've got left is really a very sparse impression of what the original structure was supposed to be. Because, as we all know, having again been through this loop over 30 years or whatever, the combinatorial explosion just doesn't work for filmed content really well.
And it made me wonder, like, why didn't they structurally do more with what we call failing forward (trial and error)? For example, you might have a choice with three options. Each option is testing a certain stat or maybe a certain Boolean for the past thing that you've already chosen, and it tests it, and then you have six different possible outcomes that involve either success, or failure. Why not just make two films for six different outcomes? Sure, it's expensive as hell to shoot a bunch of different outcomes. But if you can write something that either fails forward, or succeeds forward, and just elaborate on little moments here and there, then you could you could have something that feels a little more meaningful,
The production team were also saying that they tried to record a scene, and then a bunch of different lines to plug into that scene, and found that they just completely lost the emotion of the scene because they didn't know where it was going. So they actually had to record the scene six times, with six different line deliveries. An experienced game actor, will know how to do that and to be able to authenticate themselves, but they're not using game actors for filming.
I think they were banking on the fact that 90% of the players are not going to turn it down for a variety of reasons. It's the most interesting choice in the scene. Let's do it and see what happens right again, that's the problem with their intentionality. And their design strategy is to make one much more attractive than the other. And then they either reward you or punish you for picking the most attractive one and I'm not interested in being punished and I'm not even really that interested in being rewarded.
The first time I got the take your meds or don't take your meds I got to it from the path where Colin has ranted about people controlling you by putting stuff in your food in the context of the government control path, which is not well executed and is not believable. But in that context, deciding to throw your pills away makes a lot of sense. It's a story
The second time I got to it. I did not have the context of the common explanation. It was just I'd come from my psychiatrist office. And so I took the meds and then I failed to finish the game and the game that Stefan created was not well received. And so explicitly connecting antidepressants to creativity is incredibly irresponsible. So yeah, I was pretty disappointed by that.
What are the social messages of this piece?
I found the mental illness aspects troubling. I also think there was a missed opportunity to go as far as they could have done with the friendship with Collin because it felt like he was a really interesting character and the possibility of comradeship in a difficult emotional space and different creative space is really important. And it is a possible resolution for a lot of the things that are afflicting the character. And the fact that they take you partway down that road and then they say, Oh, he drugs you, or you accept his drugs and then things go haywire. And there's this weird force jumping off a balcony scene. I felt like that was there was something false about that. It felt like we got to this point because they thought it would be shocking or because they felt like they had to show you having a bad result for taking drugs. It feels like it's an opening to emotional intimacy and connection and you want to say yes to that, and it really bothered me that the story was like a hard no, and not for a good reason.
I was genuinely emotionally affected by the death ending. There's a bit at the very beginning where, where you're talking about your mother's death, and the psychiatrist says he didn't know. And so being back there and picking …No, I'm not getting on the train, knowing what's going to happen is also a very interesting emotional moment. I would have liked to have seen them do more of that.
Heather Albano, Mary Duffy, Jason Stevan Hill, Emily Short, and Ian Thomas. 2019. Narrascope 2019 - Panel Discussion - Dissecting the Bandernatch With a Vorpal Blade. In Narrascope 2019: YouTube.
The Illusive Ludonarrativity and the Problem with Emergent Interactive Storytelling Models in Interactive Movies
Whilst cinematic games are increasingly popular, interactive movies remain niche and often struggle to combine narrative and interactivity. Some theorists argue that this is because interaction obstructs the movie watching experience.
"The problem is that too much interactivity will weaken the narrative and too little will weaken the gameplay" (p. 22 citing Crogan, 2002).
For this reason, even though internet TV platforms have the capacity to include interactivity, most don't, or if they do, they offer only very limited interactivity. The few examples of successful integration of more extensive interaction such as Her Story (2015) and Roundabout (2015) are either regarded as niche, or classified as games. Indeed, due to the historically poor reputation of "basic narrative with minimal branching and predictable endings... (Edmond, 2015; Edwards, 2003)" in interactive movies, the gaming classification has been a notable preference. The researchers quote a print advertisement for the interactive movie Psychic Detective, which self-describes as a full motion video game, adding 'Yeah, we know full-motion video games in the past sucked' (Interactive Movie, n.d.).
The researcher defines interactive movies as those which include interactions that navigate the narrative and create alternate stories, albeit within the overall structure of the script, so that the audience is also in some sense the co-creator of the interactive movie (Kromhout & Forceville, 2013).
Using a combination of surveys, interviews and observations of 150 university students who were asked to play the interactive movie, The Outbreak, a branching tree narrative movie. Branching tree narratives allow for pauses so the user can choose a path, and based on their choice the narrative continues with multiple possible endings.
"Looking at our chosen interactive movie for this study, the interactivity only serves to branch the narrative and actually pauses the play until the user chooses a path. For that matter, most interactive movies seem to only use interactivity as an add on and not at all embedded as part of the narrative" (p. 27).
"While some of the participants enjoyed the novelty of trying a new concept, most found the interactivity distracting and even frustrating, with the most common key words used being – boring, distracting, frustrating, unique.." (p. 26).
The researcher concludes that gamers are interested in complex interactivity and do not find interactive movies challenging enough, while non-gamers find the interactivity distracting. Thus, interactive movies as they are seem to be lost between two distinct genres, but the in between approach satisfies neither audience.
The researcher recommends non-obstructive interactive mechanisms that are complex enough to create an immersive gaming experience, while fully integrated in the narrative.
"For interactivity to be fully embedded, the gamer needs to let go of the role of story author and be a willing participant in the narrative, immersed in the story by the act of interacting with the narrative in a semiotic manner" (p. 23).
A notable example of early interactive cinema in 1967, the interactive movie Kinoautomat (Weiberg, 2002) and its use of pre-determinism and making fun of democracy. In this movie the audience voted on each choice but that always lead to the same conclusion. The interactivity is very simple with a branching narrative structure that leads to the next section; the interactivity preserves the narrative and the narrative structure, but at the same time, the interactivity itself is actually part of the narrative in that it tells the story of democracy, no matter what you vote.
Work is under way to explore whether focusing upon character interaction, rather than narrative interaction might also enable more interactive movie engagement, but early efforts like the Façade Interactive Drama produced in 2005 (Mateas & Stern, 2005; Rettberg, 2015) are still experimental.
The market is not yet developed, and the researcher concludes that it will require considerable investment in talented writers and interactive designers to produce a few hit interactive movies before audiences are willing to consume such movies regularly. e.g. Netflix recently added interactivity to Black Mirror Bandersnatch interactive episode (discussed in another research summary) which was met with fanfare and positive audience engagement (Chua, 2019). As yet these efforts are still experimental.
Dahdal, S., 2020. The Illusive Ludonarrativity and the Problem with Emergent Interactive Storytelling Models in Interactive Movies. Journal of Digital Media & Interaction, 3(6), pp.17-33
Why do people play location-based augmented reality games: A study on Pokémon GO
This article reviews findings from a survey of over 2, 000 Pokémon GO players regarding their motivations to play the game.
Earlier experiences, especially with the same franchise, social influence, and popularity were the most common reasons to adopt the game, while progressing in the game was the most frequently reported reason to continue playing. The player's personal situation outside the game and playability problems were the most significant reasons to quit the game.
The Pokémon GO brand is widely known, it has nostalgic value, and its characters are simple and attractive even if one is unfamiliar with them. The “gotta catch ‘em all” theme of Pokémon is well suited for a location-based game where the player can go to different places to find and catch different creatures.
Reasons to start playing Pokémon GO
As many as 43.9% of the respondents reported experience with fandom for similar types of games or hobbies as a reason to pick up the game. Out of these, experience with Pokémon was by far the most frequent reason to start playing, mentioned by 39.6% of the respondents. The idea of the game brought up nostalgic feelings of childhood moments playing Pokémon. Some dreamt of being a Pokémon trainer as a child, and the game felt as the closest thing to fulfill that dream. In a smaller margin were previous experiences with geocaching, Ingress (Niantic, 2013) or other location-based games, or playing games in general.
Parents mentioned either wanting to be more informed about their children's activities or wanting something common to do together with them. Similarly, a friend's or a partner's recommendations or wanting to spend time with them while playing were reported.
The hype around the game and the visibility of the players had a major effect
Physical exercise and spending time outdoors while playing were appealing. In addition, the respondents liked the idea of being encouraged to explore their surroundings and new areas.
Location-based characteristics or AR, was a reason to try the game.
SITUATION and CONVENIENCE
Wanting something fun to do while doing other less interesting activities, or having a conveniently located PokéStop nearby. Some mentioned having a new phone, which made trying the game out convenient. The game being free and good weather were also mentioned.
People stated they picked up the game because they wanted to keep up with the times.
The general sociability of the game, liking to compete or wanting to help others were brought up. Some felt that playing would be a good opportunity to meet new people, even potential partners.
Looking for, hunting, and collecting Pokémon was fun. The “treasure hunt” like gameplay was seen as exciting.
THE NATURE OF THE GAME
Being casual enough and having easy access, was appealing.
Reasons to continue playing Pokémon GO
The most common individual reason to keep on playing was collecting Pokémon. Achieving personal goals, the joy of discovery, and the general feel of advancement.
Again, exercise and outdoor activities interested the players, and having a reason to go out and walk was motivating.
Whether wanting to meet new people while playing or playing together with friends or family. The game functioned as an easy way to connect people together and create a feel of community.
This could mean parents wanting to be up to date and informed about their children's hobby or avoiding being left out of social circles when all friends were still playing the game.
The game continued to feel interesting or fun.
Some players were curious about how the game was going to change or waiting for a specific update.
THE NATURE OF THE GAME
The casual nature of the game, making it easy to play, while others felt that the challenging nature was positive. The game provided surprises and was rewarding.
While previous experiences, especially with the Pokémon brand, were brought up as the number one reason to start the game, they were rarely mentioned as the reason to continue playing.
Only a few respondents reported technology related reasons to continue playing, for instance liking the location-based properties or the AR features.
Reasons to stop playing Pokémon GO
Getting bored, a lack of time or money, poor or cold weather, and health problems were mentioned, while some had quit due to their phone breaking or the game not working where they lived. Some had achieved their goal and had thus decided to quit, while others felt the hype was settling down.
The leveling curve was seen to be too steep: the required experience points needed for a new level rose exponentially, while the earned experience points stayed the same, making it necessary to grind to advance. Similarly, when reaching a certain point in collecting the Pokémon, it became increasingly hard to find any new ones to advance towards the goal of catching them all.
Bugs, the game crashing or not registering the walked distances properly were mentioned. The respondents criticized the unequal gaming possibilities due to the Pokémon and PokéStops being concentrated to city centers. In addition, some disliked that you needed to keep the game active at all times even when playing passively. This caused the battery to drain.
The shortcomings of the game, especially the lack of content, were seen to be problematic. Some players would have wanted more features or more Pokémon.
Sometimes players felt that the game was changing for the worse. For instance, the removal of the nearby feature, which had made locating Pokémon easier made some to stop playing.
Niantic was criticized for their lack of communication to the public, and even claims of not seeing them as trustworthy arose.
If friends no longer played the game, some respondents explained not feeling like continuing the game alone. Other people could have a negative influence, for instance by cheating.
Alha, K., Koskinen, E., Paavilainen, J. and Hamari, J., 2019. Why do people play location-based augmented reality games: A study on Pokémon GO. Computers in Human Behavior, 93, pp.114-122.
google Coursera course: Introduction to augmented reality and ar core
We don't normally summarise course content. Yet, this particular course offers a clear and accessible introduction to augmented reality technologies, so it seems helpful to flag it here. The main points covered within the course are listed below:
The basics of augmented reality
Inside-out vs. outside-in tracking
Fundamentals of ARCore
Constraints with current AR
Use cases and current powers/limitations of AR
Basic AR interaction options
1. Drag and Drop
4. Pinch and Zoom
Think like a user
Next steps on the AR journey
A closer look at mechanics of ARCore
Using Poly and Unity to create ARCore assets
Even despite Google's undeniable motivation to promote its own products via this course, the course explanation of the above summary points is helpful and clear, enhanced by videos, diagrams and graphics.
Coursera course content is free for University students. Certification is optional for an extra fee.
VR, Google AR &. "Introduction to Augmented Reality & AR Core." Coursera, accessed 22.9.20. https://www.coursera.org/learn/ar.
spatial interface design: augmented reality and neuroscience
Augmented reality (AR), creators can use the space around participants to visualize media, enable participants to organise those aspects with their own, and share it all with peers face-to-face. This is a Spatial Interface.
META an augmented reality glasses company researched SPATIAL INTERFACE DESIGN GUIDES in collaboration with a diverse team of neuroscientists, led by Professor Stefano Baldassi (previously of Stanford University), UX designers and developers, advisers from industry and academia, and Meron Gribetz (founder and CEO of Meta), studying the way that the brain understands natural 3D interfaces.
Think Spatial: Place Tools and Content in Space
Replace flat layouts of windows, menus, and buttons, with a Spatial Interface that arranges tools and content in 3D space around the user.
Arrange volumetric tools and content in space.
Cram traditional GUI elements (windows, icons, menus, etc.) into 2D panels.
UI Design Suggestions
1. Reconsider traditional windows and screen-based conventions
For example, don’t place a “start menu” in space, since it was designed for a small screen and presents a cluttered mess of buttons, abstract icons, and tools.
2. Separate elements into two buckets: volume and content
Text and video are inherently flat, and should be contained within tangible 3D structures with their own sense of affordance. 3D models, on the other hand, may exist as free-standing objects of their own and their design should suggest their use.
3. Distinguishing between tools and content
Content is an experience in and of itself; it conveys information or some kind of sensory experience. Tools, on the other hand, serve the purpose of creating, modifying or in some way interacting with content. The "three-dimensional tool" needs to be based on the user's intuition, and the "content" may have an abstract and flat shape, or it may have a realistic and three-dimensional shape. For example, text and video are flat, but need to be embedded in some three-dimensional structure. On the other hand, since the 3D model is originally three-dimensional, it can exist as an independent object.
4. Avoid expandable or hidden menus in AR
Instead of imitating the flat world of traditional traditional UIs into a three-dimensional world as they are, it is necessary to get inspiration from actual work sites such as art studios and workshops. Instead of thinking about menus and buttons, you should think of tools that the user can actually grasp.
Minimize Abstractions: Design Tools with Volume and Affordances
Now that we understand how to arrange objects in space, how should we design the objects themselves? Replace abstract representations (like flat icons) with volumetric tools featuring physical characteristics that suggest their use without explicit instruction, leveraging the neuroscience of affordance.
Use realistic volumetric designs for tools and content, creating intuitive affordances for the user.
Represent objects with abstractions such as iconography
UI Design Suggestions
1. Design with affordances in mind
Design tools with affordances, which are physical characteristics that suggest how the tool should be grasped or used. For example, a holographic eraser might have grooves to invite grasping it on one side, and a flat surface for erasing on the other side. Objects and tools designed with affordance in mind are better than abstract icons. Can be understood quickly and deeply.
2. Build on the user’s prior knowledge of a tool instead of defining your own
Design based on prior knowledge, to enhance usability. For example, don’t reinvent the paintbrush in some radically new way.
3.Use tools only if direct hand manipulation is insufficient or biomechanically challenging
Direct hand interaction is preferable for most simple tasks, such as moving, rotating, or scaling an object.
4. Avoid use of buttons in AR as much as possible
For instance, rather than presenting a “send” button when the user writes an email, provide a mailbox that the user can drop the email into.
5. Compensate for lack of haptics with other sensory cues
For instance, consider synchronized audio cues when the user’s hand interacts with tools and content, or build hybrid tools that mix digital interfaces with real-world objects, or use physical markers that permit holographic drawing in space.
You are the OS: Organize Holographic Files and Tools in the User’s Environment
Replace the traditional abstract and complex hierachy of file systems with spatial organization of files that makes sense for people, it makes them easier to engage with.
Use the physical environment to create organization and structure.
Use the abstract, nested file systems of traditional UIs.
UI Design Suggestions
1 Use space as an organizational tool
For example, if the user places a holographic object in a specific area of a physical desk, their spatial memory of its placement makes it easy and faster to search later.
2 Use volumetric containers and holographic furniture
Also, the object should use a three-dimensional container rather than an abstract folder or file system. Unless these containers have a deep hierarchical structure, files can be easily retrieved ergonomically.
3 Don’t overload the user’s working memory with nesting
These containers should not be nested. Nesting overloads the user's spatial memory and makes the file unintuitive. If the application does need nesting, then do not exceed one level. Despite being commonplace in traditional UIs, a holographic drawer should not also feature hidden “sub-drawers” within it.
4. Preserve the user’s spatial memory
When the user is selecting an object, other unselected objects should not be hidden Yes. This is because hiding impairs the integrity of the user's spatial memory. Highlight the selected object instead of hiding it.
5 Miniaturize content and tools to optimize space
On the other hand, miniaturizing content and tools (or vice versa) is effective in maximizing the user's workspace without disrupting spatial memory.
For example, family photos are routinely kept by the thousands in labeled albums, which can then be organized on shelves. This does not represent a violation of those principles because the photo albums are already exposed, and once opened, the user immediately arrives at the content they’re looking for—hence one level of hierarchy deep. This use of furniture, simple grouping, and shallow nesting provides AR developers and designers with an effective model for organizing huge amounts of holographic content without the cognitive overhead of abstract file systems.
Touch to See: Use the Hands to Interact Directly
The brain naturally tends to recognize objects near the hand. The tools and the content they operate on should not be physically or spatially separated. That is, the user should not operate the content from a distance or mediate abstract objects such as gestures, buttons, and menus.
Use direct hand manipulation to use tools and interact with content.
Use remote gestures and abstract pointers.
UI Design Suggestions
1. Touch content to act on it
Tools, actions, and the content on which they operate should never be physically or spatially separated.
2. When grabbing¨ pushing¨ or expanding
It is necessary for the object to reflect the user's physical movements and movements such as grasping and spreading. It should not be associated with gestures that the user must first learn the action and function of, such as opening a fist or waving to spread an object.
3. Avoid gestures that require specialized knowledge or memorization
While power user shortcuts are enticing ways to speed up interaction with an interface, they leave behind most of humanity (the non-power users). Instead of discrete gestures, use affordances to encourage the user to interact with objects in more natural ways.
4. Persistence and the usage of tools
Tools intended for persistent use (such as a paint brush, a conductor’s wand, or a flashlight) should realistically respond to the user’s manipulation, instead of requiring a discreet hand gesture to activate or deactivate. For instance, the brush should draw when pressed against a surface, rather than in response to a separate trigger like a gesture or button press.
5. Proximity feedback is your friend
Provide proximity feedback when the user uses or touches an object or content. By doing this, it is possible to compensate for the lack of tactile sensation and emphasize collisions and movements. Glows and subtle audio cues are a good way to accent a collision or movement while compensating for the lack of haptics.
Do Not Disturb: Do not interrupt the User’s Workflow
Spatial computing demands spatial notifications. Rather than interrupting the user’s workflow with pop-ups, allow users to designate a separate container to passively collect incoming notifications. This allows the user to remain focused on their task, only stopping to check for updates when desired.
Establish a passive notification receptacle: non-intrusive, segregated from the task at hand by default, and easy to move.
Interrupt the user’s task flow with active notifications and pop-ups.
UI Design Suggestions
1Separate designated areas
Allow the user to designate a container or space to collect incoming notifications away from their workspace, instead of interrupting them directly with pop-ups or alerts. This preserves the user’s control over their experience by allowing them to focus on the task at hand, only breaking their concentration for updates when desired. Ideally, such a box would be located outside the user’s field of view and at some reasonable distance, ensuring disruptions are avoided. As a default, prioritise the user’s concentration, only allowing disruptive notifications when requested explicitly by the user.
2. Design tools for specific purposes
Certain tasks require bottom-up disruptions to break top-down flow (like when your timer beeps), and should be incorporated into tools designated for that specific purpose, but even these tools should mimic real world tools like alarm clocks and egg timers. Most importantly, the user should always understand how to turn off the sounds and visuals associated with alerts.
3 Allow for tailoring and flexibility
Create flexibility by allowing the user or developer to designate multiple boxes, and tailor them to receive certain kinds of notifications. For instance, one box might be associated with all social media apps from different services or people, while another might be for email only. A third box might handle all incoming phone calls, making it easy for the user to selectively allow calls to disrupt their work (by bringing this particular box closer to the workspace). Boxes can also be connected to specific people or groups, allowing for further organization of incoming notifications. Furthermore, the design of the box itself should use affordances to suggest its use, such as a lid that can be closed to mute notifications, or opened to enable them.
4 Consider gradual notifications over abrupt ones
Even if users choose to place notifications within their workspace, they should consider using gradual visual and audio cues instead of explosive alerts.
Avoid Surprises and Magic Tricks: Pair Actions with Intuitive Outcomes
Avoid “magical” events that appear unrelated to user behavior or earlier causes, or events that ignore the laws of physics as these only confound the user.
Create predictable connections between a user’s actions and the result.
Use nonsensical, disorienting quirks and surprises.
UI Design Suggestions
1Avoid “magic” by tailoring outcomes to match the user’s actions
Here, “Magic ” refers to anything that confuses the intuitive sense of cause and effect. A magic wand, for example, is confusing because it looks more like a stick and there is no affordance associated with a particular task. Also, people can't predict the consequences of using it. On the other hand, most users will immediately know how to use an eraser; its use can be understood and its effects predicted with no additional explanation.
2. Use realistic physics to create intuitive object interactions and behavior
It also confuses the user when the object does not respond in a reasonable way.
For example, even if you gently push a holographic ice cube and slide it on a desk, do not fly across the room at 900 m / h. It is necessary to have a clear relationship between actions and effects.
3 Enforce causality with clear associations between action and effect
A sound triggered by a user’s action should occur immediately in real-time and emanate from the appropriate location. Likewise, visual feedback such as glowing highlights, shadows, or physical deformations should be proportionate and spatially close to the event that triggered them.
The Holographic Campfire: Don’t Obscure Hands and Faces with the UI
Our ancestors have evolved to face each other, both in pairs and in tribes. Therefore, we are very sensitive to the gaze of the eyes and the facial expressions of others, and the dedicated areas of the brain continuously perceive those sensations.
Our perception of the line of sight extends to its surroundings and is immediately noticed when someone else is looking at something of interest.
Avoid blocking eye contact, create shared spaces that encourage a mutual gaze, and ensure hands are visible while collaborating.
Replace UIs that disrupt eye contact with a shared space. Also, collaborate by directly manipulating shared content.
Obscure or occlude the face of others with interface elements, or use single modality collaboration, such as face-to-face collaboration without the hands or voice chat without video.
UI Design Suggestions
1. Promote unobstructed views
Avoid interfaces that occupy the entire field of view and obscure facial expressions—two significant components of personal communication. Within a shared physical space, interfaces should be arranged around users, not between them.
2. Support collaborative experiences when working in the same location
For example, a collaborative design tool should encourage all participants to manipulate the same object in the same place, as opposed to private instances visible to each individual separately.
3. Show hands when collaborating
The interface must clearly indicate the position of the user's hands. When a user's hands are within the sight of other users, the ability to learn from collaboration and demonstration increases.
4.When collaborating remotely¨ use 3D video of participants instead of avatars or virtual characters
Artificial expressions such as still images, icons, and virtual characters only reduce the accuracy of communication and improve cognitive burden and social anxiety.
Public by Default: Shared Understanding Reduces Anxiety Among Users
We have evolved to constantly observe the attitudes, movements, and behaviors of those around us in order to attempt the intentions of others, and to guess the minds of others. By default, other people's content should be viewable. All users should be able to see the same holographic environment, just as they see the same physical environment in the real world.
Make tools and content public by default. Maintain the ability to make
things private if needed.
Offer private, asymmetric UIs (like the ones seen in Google Glass).
UI Design Suggestions
1. By default¨ ensure content can be viewed by anyone else wearing a headset
All users should be exposed to the same holographic surroundings, just like we all see the same physical surroundings in the real world. When a user's content is hidden from other users in AR, the action appears to be in an empty space . To confuse. Furthermore, devices like Google Glass have demonstrated our distrust of private interfaces because they allow aggressive or invasive actions that can be performed secretly, even when face-to-face. Such practices may divide users, rather than encourage cooperation, and may contribute to social anxiety.
2. When privacy is necessary¨ use a blocking object like curtains or dividers
Do not simply make sensitive content invisible from the perspective of unauthorized users, as this creates a disparity between users that can lead to confusion and distrust. Instead, just as in real life, establish privacy with holographic curtains or dividers. This approach ensures a consistent experience across all users without confusion or mixed signals, and eliminates the disorienting sight of gestures performed on empty space.
3. Privacy is a function of etiquette¨ not a feature of the interface
Augmented reality is most powerful when treated as a single, shared space. Don’t relegate privacy to an on/off switch that segregates user experiences. Confusion arises when this otherwise collaborative experience is unknowingly changed from one user to another for the sake of privacy or discretion. For example, a user who wishes to view financially sensitive information should do so within an interface that has privacy features, such as a curtain, divider, or display that can be tilted away from unwanted viewers. Users should design their own privacy within the rules of the shared space, as opposed to breaking the rules.
Augmented, Not Mixed, Reality: Enhance the User’s Perception with Relevant Information
Rather than block out users’ reality (as in virtual reality) or distort the user’s reality (as in mixed reality), provide access to metadata about the world placed near by, without masking it—add something useful to people's understanding of their environment. Create an informative, powerful, and unobtrusive layer of digital information on top of the real
Present information and tools that reflect the user’s reality and deepen their understanding of it.
Block or distort reality, whether fully or partially.
UI Design Suggestions
1. Complement the real world
When images and sentences added in the user's field of view by the AR (Augmented Reality) interface are added to the real world without discomfort, the user can understand them intuitively and effectively. Design interfaces to complement real-world objects, instead of masking or transforming them. For instance, a panel with a Wikipedia page about a flower placed near, but not occluding, the flower would be considered augmented reality. Consider a user that is able to touch a flower and learn about the amount of DNA s/he shares with it. This simplistic example demonstrates how one can enhance their connection with their surrounding environment.
2. The experience of turning the gym floor back to the surface of the pool, in contrast, causes unnecessary confusion and misleads the user about how it works.
This is because it changes or distorts the physical properties of the actual object.
3. Add value and insight for the user
Rich content should be displayed to provide information with as much clarity and depth as possible. Text, video, volumetric content, and the Web can all play a role in empowering the user with deeper understanding and connection to their world.
4 A note about use cases in gaming, In games, we recommend allowing virtual characters to pass through the actual user's environment, rather than "breaking walls" or covering them with overly distorted visuals. On the other hand, for certain uses, such as interior decoration, momentary changes in the environment are tolerated. This use case leads to changes and improvements in the physical properties of the real world.
The report is further summarised in these two blog posts:
For the full report, together with example images and related neuroscience references, see:
Panel_ State Of The Art in Immersive Heritage Visitor Experiences
ATA 2018 Audience of the Future Presentation.
Alchemy VR: Our first experimental virtual reality was back in 2015, where we partnered with the Natural History Museum. We worked on an experience where we walked people through the Cambrian ocean, a 500 million year old environment based on fossil data held at the museum. David Attenborough took you through that amazing world which we delivered on location in the Natural History Museum. Visitors would come in, wear a headset, and they'd get the VR experience then they'd see the actual specimens. And it was a great example of a use case of VR, because how else could visitors go back in time to that period and understand what it's like to be in that environment.
86% of visitors learnt something new about the natural world
68% of visitors also learnt something new about the museum’s scientific research, which is a really important outcome.
On the basis of that success, we again worked with David on a VR experience that takes you down into the Great Barrier Reef and visitors could explore the barrier reef with him, again, a use case for VR that other museum interpretive tools could not deliver. How can you go down into the barrier reef unless you're extremely rich, and dive that environment and understand about the fragility of it. Both of those experiences have run commercially in museums right across the world.
In other words, VR is an interpretive tool that offers things that other museum interpretation tools cannot do. It puts visitors in the position of somebody actually experiencing something, or undertaking an otherwise impossible journey.
The VR lounge was delivered on site in the Science Museum, and also toured around the UK to a number of venues across two years.
Also, the VR bus takes a cut down version of the experience into hard to reach communities and schools to engage people across the nation.
And another experience that we've worked on involves more of a 4k experience and supported efforts to defend the Amazon for Greenpeace who wanted to create a piece that engaged people with the plight of the Maduro tribe who are about to be wiped out by the building of a dam. We partnered with another company to create an experience whereby, as virtual reality visitors had fully visceral physical experience and sensory experience delivered in really beautiful pods. And we took a perfume out to the Amazon to capture the smells of the Amazon. Those were delivered in the pod. So as you were walking through the Amazon with Munduruku people, you'd smell what they would smell as point in experience where you have a cup of coffee with the people and you suddenly smelt coffee and felt the heat of the coffee cup in your hand. So really beautiful, visceral experience and it ran across a number of museums across South America. And best of all, the dam was not built.
The entertainment sector I wanted to talk about is Disney's Pandora. This experience was built around the Avatar film. As Jake played by Sam Worthington in the movie, you fly on a banshee. And this whole experience, the VR ride essentially is one part of a massive, immersive experience. The environment consists of built mountains and beautiful streams that visitors walk through. Visitors engage in the physical experience, long before they come face to face with an avatar in a jar. They're given their own avatar through which they then enter the virtual reality. So, there's elements of personalization and there are also elements of extremely theatrical environments. You're got wind, you've got mist you really feel like you're on the back of a banshee.
We're also doing big warehouse scale gaming VR, which have included lots of zombie experiences. Participants engage with each other in a game environment in the physical space. Noma ran an experiment called virtually dead, which essentially emerged as a kind of theatrical experience with a virtual reality experience. Participants first had to train with actors before they came face to face with the zombies, both as actors and then in the real world in an East London warehouse. This is a small kind of startup essentially tickets sold out immediately. And they started going for five times their market price on the internet.
The Science Museum did the first AR museum experience, I think. They used AR to essentially invigorate a gallery, which was very tired from the 1970s to add vision of skin on bones in order to enable visitors to understand more about the creatures in a more exciting way. AR is now also being used to gamify a kind of commercial visitor attraction. Cedar Point used AR, to essentially build a game for visitors around the characters in the theme park with a lot of social interaction and storytelling. So virtual creatures appeared in the real world and visitors formed groups to play games against each other in the theme park. Greg from rewind has also got a wonderful example from his own work, a NASA experiment with their Jet Propulsion Lab and the pre commercial HoloLens in which shows Buzz Aldrin playing that at the Kennedy Space Center. They created a Mars environment where multi visitors at the same time could engage in a virtual Mars environment.
Factory 42: Hold the world involves David Attenborough, and gives visitors access to treasures of the Natural History Museum. In VR visitors can pick up and turn the objects around to examine them much closer…and watch the objects come to life.
The key to Hold The World is that it gives the user the chance to do something that they simply can't do in real life, which is crucial. Also, once visitors put on the headset it’s worth it because they have a one on one audience with David Attenborough. The experience also takes visitors behind the scenes in the Natural History Museum to three extraordinary rooms that the public is not able to enter.
We sort of think of it it's hard to explain, really, but it's, it's somewhere between a TV documentary and a computer game.
Eight organizations worked together to make this possible:
We advise three broad principles to success in this space.
1) A balance between storytelling and consumer insight and technology. So a lot of what we see out there right now doesn't really have that balance. Too much is led by the technology, and not what the consumer might want to experience. … the key thing is, and this is really obvious, but it's sometimes easy to forget, is for the technology to enable, rather than to lead people and actually what's the story you want to tell? And then how can the technologies help you tell the story?
2) I think the second thing you need to succeed is resilience and an appetite for a tough challenge. I mean, this stuff is hard to do, particularly at scale. The technology doesn't always work. It's changing quickly, and finding the right people and good people who know how to work with it is also challenging. It took us probably about a year to work out how we'd make Hold the world actually before we started in production.
3) Collaborate effectively, because no one's got all the answers. So bringing the cultural world and the tech world together. Both have to adapt the way they work. And that's a key learning from my own personal experience on making Old World was we spoke different languages. I mean, we really did it as if we should have had a sort of simultaneous translator with us at all times. But sides of the team. It wasn't two sides. But both people from both backgrounds in it, we've moved. We grew in understanding how each other works and things. And I think we all emerged wiser from that. These are complicated relationships with multiple stakeholders
Change is happening quickly.
Hololens (and now Unity MARS) - The reality of what the technology can do is absolutely mind blowing. It can scan a room like this can map every surface it can place objects and things on those surfaces and explore an entire world. The building blocks from that are things like unity AI technologies that are in the hands of any talented development team working within this space. And those things are transferable across the entire pipeline.
So something that HoloLens today could soon enable us to see smoke come out of trains as we walk around them. We can bring up the data that people are squinting at. We can bring in objects you could not otherwise have that space.
If you think of Red Bull Races. Actually, what we're doing is bringing in real time race to the telemetry data, we're bringing in multiple video feeds, we're using the HoloLens as a control unit that controls multiple screens in the space, so you can switch between different pilots racing. Red Bull air race is the fastest motor sport on the planet, because even though a plane might be doing 200 miles an hour, they're getting pulled up on a slight technicality because their wings weren't perfectly aligned at a certain point or they lost by nought point nought one of a second. Also, when you can't see two planes on a racetrack at the same time, the danger, the excitement and the power that's going out in the field out there doesn't translate. So with technologies like this, we're able to show the truth of what's going on. We can use the data to help us tell the full, dramatic story what's happening out in the racetrack.
How do you please more than one visitor in the same experience? Is that visitor coming along to get a visual experience? Are they going to have a kind of public experience of some kind? Are they going to have a visceral experience? Is it going to be emotive? Who are you actually appealing to? Now it seems to me it has to be a global audience, given the scale. So how do you square that circle of having somebody down the street walk in, and somebody's logging in remotely from China, applying a very different cultural understanding to whatever experience you're giving them. The technology has to deliver a culturally variable message. Also, a great experience is not really about single experiences. Coming to a museum isa community experience. Putting on a headset optimizes that experience, but it also takes you out of that community, plus it makes you look like a dick (e.g. you might want to consider using mobiles, or glasses instead). But you see what I'm getting at, the experience is being dictated by the Technology, not necessarily the other way around. The curiosity should not be driven by whatever the technology happens to be.
Imagine we’ve got the Giant's Causeway. It's very boring. But actually, if you were trying to make a visitor experience out of that, you can ask a lot of questions about whether it is a geological site. Yes. Is it a contested site? Absolutely. Because the other folkloric story is that a giant called Finn McCool made this place. And if you think it was Finn McCool, that means that you're coming from a Catholic background, and that makes you a Republican. And if you're a unionist, the Giant's Causeway is where you realized that you are connected and not separated by water to your ancestors in Scotland. Contested space! So you've made a visitor experience, and three people have just come through the door. And they've all got different understandings of what that is. How do you represent those narratives? Because it's those narratives that are important. And more crucially, how do you represent those narratives to a Chinese person who has their own narratives? So if I was putting the visitor experience together, I would be thinking about a template of applied understandings and ideas which can move to anywhere in the globe. So you've created a template which you can then sell to South America or you sell to us Africa where they can tell their stories with their cultural resonances. But you've created a vehicle through which they can do that using the technology.
Constructing a mixed reality, sensation, experience, which allows people to bring their reality and construct their own narrative coming out of that experience space is quite a challenge. You've got your narratives, where do you set that into? You may create a landscape, a soundscape, a cityscape, but there has to be a context in which those experiences happen. And those narratives can communicated regardless of the technology, so don't fixate on the technology.
You want everyone to know the same piece of information and to be able to share an experience. To help people have their own private experience and also find out the shared base-line, does there need to be a group activity, or a walking tour together, or simply a way to share with families and friends? The answer depends upon who you're trying to reach and what you're trying to help them understand.
I teach museum personnel on a master's program, and it's my job to encourage museum workers to take on digital tools and they are absolutely resistant to that space because they believe passionately in the authenticity of the object. And that's the kind of problem you're up against How do you convince people that those technologies enhance the authenticity of the object?
Embracing the Combinatorial Explosion: A Brief Prescription for Interactive Story R&D
The USW Audience of the Future research team is compiling a summary collection of recent research in the field of immersive, and enhanced reality media