Narrascope 2020: Visual storytelling in Immersive Reality by Matthew Roth.
My name is Matthew. I'm a UX writer at Google Daydream and this presentation is about how to tell stories as a developer, as a designer, as someone with a pretty cool tool using the techniques at your disposal.
My interview with google was literally the day that Google Assistant was announced. I was hired to make video games without video and at the time, we had no idea what that meant.
These days, I am working in almost the exact opposite media, immersive computing. It's virtual reality, augmented reality, anything you can see that isn't really there. Ideally, the written word will intrude as little as possible.
But our goal is the same: To take the user through an action experience as naturally as we can, and to have our users spend the minimum time thinking about the medium that the experience takes place in, and the most time being in that experience and participating in it and interacting with it.
When you play a game, you experience the story by telling it to yourself.
Any game gives you tools, a weapon a spell book, the ability to make monsters vanish by jumping on their heads. And by picking up these things, or casting them or jumping on their heads, you're telling your own story within the boundaries imposed on you by the master storyteller, the game designer.
AR gives you a whole collection of game mechanics, constructing, crafting, discovery and more. AR core is Google's engine for running AR, which appears on most of the latest generations of Android phones. Apple has a similar engine called AR kit. And there are a few others too. They have different features and annoyances. But basically, what they do is they look at what your camera is seeing at any given moment (plus the moments just before the moments after), and they extrapolate all this information to put together a picture of the world around you in 3D. Augmented reality is a way to see virtual content in the real world. The way I explain it to my mom is VR is real if you are in an imaginary world, ar is imaginary objects in the real world. AR core is a platform for building AR apps on Android phones. It keeps track of three variables to construct these AR worlds. There's motion tracking, environmental understanding, and latest dimension.
Although these design principles relate to AR Core, they still apply to all sorts of tools.
GET TO KNOW THE TECHNOLOGY
I am always looking for ways to like dig into the technology because if I can make myself understand it, then I can really do anything with it. I start to figure out how the developers are seeing this world. And therefore instead of just being like, make this happen, or can you make this happen, I'm like, hey, can you use these tools to make this happen this way?
MAKE ON-BOARDING PART OF THE NARRATIVE
Once you've established your place, you need to convince the user to detect the real world surroundings, in order for the phone to calibrate its whereabouts. That creates an intrinsic delay that's built in to mobile AR, so we need to give the user a mission. We need to give the user an excuse to discover the world and physically move the phone around. So we give them a fetch quest, which are normally done badly. Because fetch quests so often fill in the dead space between fighting, it's like you're saying to the player, the thing you usually do is run around and fight people. Now let's get rid of the fighting people part and just like run around for no good reason we've effectively freemium qualified them.
In the user journey that running around and the technical functionality of scanning for surfaces are exactly the same. So the idea of discovery is at the core of the experience, so we want to embrace that moment and make the narrative wedded to that, not just have it be a do this so that you can play the game, but have that be the beginning of the game.
So we want to make it awesome.
These AR techniques that I'm talking about, like plane finding and setting up boundaries. They don't necessarily feel like storytelling techniques, but they are.
ADD DEVELOPMENT, NOT REPETITION
I'm going to tell you all about one of my favourite game designers, Aristotle. Aristotle created a storytelling structure that a lot of us still use today. First, there's an inciting incident, an explosion of birth and death, like getting lifted up by a tornado and landing on a witch. Something happens that the main character is not necessarily in control of. Then the character reacts to it. They strap on the witch's slippers and follow the yellow brick road. That's the Act One climax. That's the hero's call to action. And then act two which is most of the experience. This is the middle hour and a half of a two hour movie. In involves a series of events where the character faces challenges, each one gets harder than the last, and each one reveals a new part of the character, a part that the audience has never seen before. That's also why if battles are too repetitive, a game gets grindy. Since it's no longer revealing something new and unseen. You're not getting character development or character building or character amping up new swords.
Ideally each episode or interaction gets more challenging until the end of that when a character faces complete despair. It's the lowest moment, the dark night of the soul, you must fall completely before you get up again.
Blake Snyder, who wrote save the cat, which is this book that all these Hollywood writers use, has this thesis. I don't know if I agree with it, but I've seen it used in such great ways. The thesis is that, um, you always show an interaction with the main character at the very beginning of a movie or a game or whatever you're doing, where the character saves a cat, because even if they're an evil person, they've saved the cat so you identify with them. I don't know. Let's talk about that later.
But then we get to the moment where we need a complete reversal from the darkest night into the final blades of glory. So that's the story of every story. It's also the story of each moment of a video. The first moments of Pac Man are filled with Tension the frantic thoughts of escape of limited motion of being eaten I can only move left or right.
DESIGN FOR AWE AND RESPONSE
You want to design a world that's both amazing to look at, and one that reacts to your presence. Make your world real. On the screen, you're combining virtual and real world objects. So let them interact, the more they play together and roll off each other, either metaphorically or literally, the more your world will feel like an actual living inhabited place. And let players mess with it. The virtual and real world, the virtual world and the actual world are only two dimensions of the experience. In order to make it feel real, we need to add the third dimension, the user. Let people touch, manipulate and change as many virtual objects as you can add, as many as makes sense to be changed and manipulated.
Here's the biggest obstacle … Users can be in one of four positions, 1) either seated with your hands fixed, 2) seated with your hands moving around, 3) standing still with your hands fixed or 4) walking around and moving around in a real world space.
PHYSICALITY IS EMOTION
By designing the mechanics for your experience, you can change their physical position as well as their mental experience. In other words, you can blow their minds by blowing their bodies out of their seats.
One of the greatest advantages we have in AR is the size of our space, it's theoretically infinite. The problem is most users don't remember that they're stuck here, right? So you want to give them something to cheat, so they actually have to move the phone around. And that little nudging icon will help them do that.
Now I'm going to talk about what players can actually do in AR game mechanics.
Like every story, you can break it down every moment into a first, second and third act, a call to action, a hero's quest and a combination. You want to take it advantage of the real world environment, put things just out of reach to users and offer them rewards for moving around exploring.
Hidden bonus levels are at time honored tradition, finding them in your living room gives you an extra measure of delight.
When you place objects in your AR scene, users will want to play with them and the more non necessary stuff they can pick up and play with the more than one a hunt for the objects that they do need to use.
But you want to give breadcrumbs too. You can break reality selectively.
SETTING IS A STORYTELLING TOOL
But as a world builder, you're empowered to decide when you want it to be realistic, and when you want to withhold that realism. If you need to draw the user’s attention to an object or an area or an evil robot, the entire world is at your command. You have lighting, shading, texture and physics at your disposal. You can highlight things you can play things down and move them into the shadows.
MOTION IS EMOTION
We can use a single effect like jump scares to achieve a bunch of different emotional goals. The best jump scares, all the best moments of connection happen when you forget there's a screen separating you from the movie from the game.
Having the action right in front of you makes that separation even easier to forget. It's in my room. It's on my bed. It's right in front of me and it's happening with me. In one morning, you just don't want to make the user move backward without looking behind them. Because that can have real world disastrous effects. At some point in the history of games, sneaking past enemies was just a way not to get killed. At some point the game designer started recording your visibility percentage lines of sight and sneaking became an actual measurable mechanic. Just think of how recording everything in the real world can let us change enemy AI.
USE SURPRISING INPUTS
If your games input is the camera, then let anything you see be the input trigger, we can record degrees of light. And light is especially useful because it's so easy to manipulate when you are indoors anyway. And because it's so unexpected to the user, you can see that when the light is switched off here, cars turned on their lights and buildings laid out, like at night.
But it's also something that the user just didn't expect, but we have it completely at our disposal. Here’s a game that’s really cool. It's really simple. The first player chooses a spot to bury the treasure. You tap anywhere and the game buries it. Then you hand your phone to the next player and they dig around until they find it.
When you create or play in the world, you always have the potential to interact with other users. Cloud anchors make this uniquely possible by matching virtual content with real world locations, then serving the same content to different users. All you need is another person and a path to the shared world. This game is played on two phones and both phones are sensing in the same environment. There are techniques to do that I will show you how to find them.
CREATE MANY DIFFERENT PATHS TO ENGAGEMENT
In some way, AR is a great way to create an all access role for differently abled users to see things in their own scale. However, it comes with a whole set of new challenges. If you tell users to reach up and grab something or take two steps forward, what happens when your user can't reach the device or take steps?
Here, we've added in an alternate way for users to reach faraway objects. There's a reticle that stretches and extends based on the angle of your phone. This is a very Googley concept. This is something that we keep talking about having many paths to success. People who like keyboard shortcuts, vs people who like mouse interactions, they're people who want to do things the slow way. We create different paths to success.
Roth, Matthew. 2020. VISUAL STORYTELLING IN IMMERSIVE REALITY edited by Narrascope 2020. U.S.: YouTube.
Escaping the Holodeck Storytelling and Convergence in VR and AR
At the 2016 Game Developer's Conference Rob Morgan who co-wrote J.K. Rowling's Platinum-selling Augmented Reality PS3 title 'Wonderbook: Book of Spells' and its sequel 'Book of Potions' shared what he's learnt about the particularities of writing for real world and embodied augmentation.
In a holodeck the only real thing is you and everything only has to feel real in relation to you.
This gets harder the closer you get to the user.
e.g. trying to simulate clothing is unlikely to work because we know intimately how our authentic clothes are supposed to feel.
AUGMENTED REALITY DIFFERS FROM SCREEN MEDIA
Unlike games, where players tend to embrace identity and power fantasies, participants on the holodeck tend to role-play as themselves and stick to their known hierarchies.
The job of a writer, therefore, is not to create somewhere believable, but create somewhere desirable…to help participants create their own willing suspension of disbelief.
Rather than generate an action figure to role-play, Morgan creates the possibility space around the player with lots of space for their own role-play. The player is their own hero and you can’t assume anything about them.
e.g. There is a line in ‘A Book of Spells’ (an AR game co-created with J.K. Rowling) – "Are You A Wizard or Not?"...If a player didn’t open the wonder book for 10 secs, then the non-player character (simulated) narrator would ask…."C’mon are you a wizard or not?" In user testing this proved to be an evocative hook, participants wanted to engage as soon as they heard this.
So, what does this teach us about writing for augmented reality?
- You are not in control - There’s nothing to stop your audience moving off if they’re interested in something else. e.g. 1st time VR players will often test NPCs for realism by doing socially unacceptable things.
-The player’s experience of the real world control is a constant negotiation between what they can do, what they can’t do and what they can get away with….so there is a need to negotiate that with them – which means that writers aren't just storytelling, they're also curating.
-In VR storytelling, far more than in flat screen games, players are interested in the reality of the VR so everything comes under more scrutiny, which means that you have to give them a way to make sense of every limitation and control (from within the narrative, or simulation). e.g. If you want to lock players in a room until they complete a task there needs to be a reason, or explanation for the locked door such as air locks, decontamination chambers….and you need to put the story out there for players to discover.
-In VR the player is radically embodied in the simulation in a way that they’re not in an psychologically immersive flat screen game….so just the same you have to find a new way to talk to them
-How do you do this? You can use the power of implication, by implicating them in situation.
e.g. To encourage players to press a detonate button it might be better to pare back the script – so that rather than allowing them to hide behind a character, the writer instead foregrounds the player's psychology at that moment i.e. don't impose a reason for them to press the evil button, but leave that reasoning up to the player. At various points in The Assembly (another game Morgan created) players are accused of various things, but players decide what’s actually going on.
- If you leave room for their own motivations it stabilises their identity inside the sim. Challenging the user to justify themselves, by accusing them of a certain motivation and forcing them to come up with them own reasoning inspires them to invest part of themselves in what they’re doing.
Which is why he began a site specific mixed reality experience with an audio provocation to the user:
-Try to act normal
-Are they still watching?
-Only you can hear me
-Just act natural
These phrases mix the user's own internal response with the staged reality, and force the user’s identity to stabilise around a few important things…something is going on, something is not as it seems, you have a job to do, you have a secret, something important is happening, just try to act normal, everything else, all the context and world-building can be filled in later.
"….and that’s how we brought our players in, not by putting them in a costume, but by telling them they were already involved, that they had a secret to protect, that it was already too late i.e. before they realised it they were already wearing the Sherlock Holmes outfit because that’s what people do. It turns out control and identity are kind of the same thing."
We have to work alongside the player – every time we try to define who they are and what they want then we’ll force them out of immersion just as quick as if we made them run at 50 miles an hour, when all they really wanted to do was stroll, or made them play Captain Kirk when all they wanted to do was be the bad guy.
Morgan, Robert. 2016. Escaping the Holodeck Storytelling and Convergence in VR and AR. edited by GDC. U.S.: GDC.
The USW Audience of the Future research team is compiling a summary collection of recent research in the field of immersive, and enhanced reality media