Designing for the body in AR & VR
UNDERSTANDING THE BODY'S ROLE IN VR & AR GAME DESIGN
https://www.youtube.com/watch?v=vTTZSjv9ZBA&t=5s
In this 2016 VRDC session, Funomena lead designer Robin Hunicke explores some of the core challenges and opportunities that the body presents in VR & AR development, from controller design and gaze inputs to body posture, movement and gesture design.
Today I am talking about embodiment as a design surface for designers in VR and AR titles.
PRESENCE
The eyes, the ears, and in this case, now, the hands and the senses on our hands, can be engaged in experiencing what we like to call presence, the psychological state or subjective perception in which even though part of an individual's current experience is generated by and or filtered through human made technology part or all of the individual's perception fails to accurately acknowledge the role of technology in the experience. It's a kind of a strange way to define it. But I think it's interesting to think about it this way. Your body is being fooled into believing that what is it is experiencing is real. We’re really tricking your body, and this is creating a space where designers can and will explore how presence affect one's experience of playful content.
HANDS
When you're wearing a head mounted display the hands are very important. And there's a lot of work being done right now to figure out what those hands will be. Why are we obsessed with the hands? Anyone who's tried to work with VR knows that when you are in a space where you're experiencing reality, and then you can't see your hands, you feel weird. They call it the rubber hand illusion, the idea that there's a physical object, and there's no response to it. Trespasser, the video game from many, many years ago famous for this sort of disembodied hand problem. It’s really important to display the controller directly having an abstract hand model, or showing the primary game object or tool or brush, whether that's a hammer, a gun, one, whatever it is, or some combination of these seems to be super, super important for engaging the body.
Games that do this well are about touching stuff. They're about, you know, being in the space and just like poke poke poke. Jenova Chen, who led the design on Journey, used to say that when you're in an inner interactive virtual space, and you're moving into for the first time, whether it's a VR experience or a classic video game experience, that you're like a baby, just pushing every surface trying to see what happens. And these games really engage you that way. So while that's not news, what's interesting to me is the difference between the power grip and the precision grip. So if you think about the controllers, I want you to put your hand out in front of you, and make the gesture that you make when you grab a controller. for VR, it's a power grip.
THE POWER GRIP
It's a force grip, and when you make this grip with your arm, you put your hand out in front of yourself and you grab space, like really do it, like retain your Wing Chun, you know, advocate and you're gonna do the on- inch punch. When you do this motion, you engage your fist, your lower arm, your shoulder, your bicep and your back muscles and your torso, the power grip has this effect of immediately stressing your body and it raises your blood pressure and your heart rate. If you do it with force, if you really try to do it, you can actually engage all the muscles in your arm and when you do a one inch Punch in Wing Chun, you can really crack someone's skull really hard. It's a really, it's a very powerful tool, the power grip and we use it all the time. When you swing an axe. When you punch somebody, when you do a really vicious dance move, you know you're using the power grip. Now, the other grip is the precision grip.
THE PRECISION GRIP
Pretend we're conducting the score for journey. It's a very different feeling. What do you engage in when you engage the precision grip, just your hand, just a little bit, maybe on the backside of your arm. Your body is telling you. This is action. This is thinking. This is why artists love is to have a tablet with a pen instead of a mouse. That the relaxed pose of painting and drawing. It’s a physical thing and t the tool itself engages the mind in a way that's really insightful and has worked for thousands of years.
So when you're using a power grip, to do precision activities, how do you translate that body feedback? This is a question that all of us get to work on. Now. What is the feeling of painting when your fist is closed? It's not even finger painting. It's like punchy painting. It doesn't really feel right; when you open your hand it feels much better.
GRIP CONTROLLERS
This desire to create a sense of precision was something that was at the core of Luna, when we started working on it, we actually started working on it, not for VR, but for the Intel gesture sensing camera. And I really wanted to build a game where you could just touch the world with your fingers. In the very first prototype that Scott Anderson ever made, you would press the buttons and it was a little pair of chopsticks and they could just use these two pieces as chopsticks to pick up little things. So immediately with the capture buttons and the pressure, he started thinking delicately so translated the power to something precise. We actually modelled and physically sculpted, what we call the grasper for the game, which is like a dinosaur claw, or bird claws. It's also like a lotus.
The other thing that we decided to do was to de emphasise the butt of the controller and grasper thing and focus on the tips. And to give it a brush light quality, it's still not finished. It's a work in progress, but when it grasps it becomes more of a pod, more subtle. Solid in the tips, and then it opens again. So we've got this feeling we really wanted to have this hand presence that would make you want to touch the world in a different way. A hand model is helpful, but it may not translate the action, the way that you wanted to. something
EMBODIMENT
Best practice literature advises you to put a body in there, that's great except that It can be pretty disturbing if the body isn't your body.
A very interesting experiment was done with body image and owning an underweight or an overweight body for people with eating disorders, looking down and seeing the body that you actually have, versus the body that you think you have, and then having a discussion with therapists about it, link to the paper in there as well.
And what we're finding right is that the lack of realistic body identification conveys difference to us automatically. Again, if you're going to put a realistic body in and I'm moving around in the body, but it's doing unrealistic motions, or it's not totally smooth. The elbow joints to the arms and all this other stuff are not responding the way that I want them to then I will find it really difficult..
We've been seeing some interesting work on gesture, and reaction and celebration, in embody avatars in social spaces. In this context, say I'm Lucy Bradshaw. I'm not going to be seeing myself this way. But I'm going to see them that way. Right? I'm going to see my hands, but I'm gonna see them. And so we're trying to kind of mix that space. And when you look at spectator view mode and the way that we're embracing this, it's also we're trying to get as much out of the the motion of the body without actually having to replicate the physics of it.
So the body is back sort of, when it comes to actual avatars. VR is a little bit like puppeteering right now. So when you think about gestures, and the body if I can't see the whole body? And I'm just seeing my hands and I'm trying to do ninja moves. What are you seeing? You know, are we simulating the rest of that body? How do we how do we perceive that? And specifically, if you're going to wave your arms like a ballerina, you're just going to see the tracers on my hands, right, like, just for a minute trying to make a balletic movement with your arm.
(Shows examples of different movement options in games)
BODY = EMOTION
You can also use the body to create emotion e.g reaching up, striving, sitting back and relaxing, rhythm etc
I like face tracking. It's better at knowing what you want than you are.. If you ever go to dinner and watch two people on a date, and you watch their micro expressions that they're not catching because they're so busy trying to mask their own feelings.
EMBODIMENT FRAMES
When you do a close up in a film, you feel emotional. You know, when you bring things into perspective with the person at eye level, suddenly your heart opens up. That’s basic film technique, but when you bring it into VR… now you're really first person!
Wayward Sky does a great job of this, playing between the two camera frames, even though technically it's not a frame. It's an embodiment frame.
So, it's not about the camera frame anymore. It's about the body frame.
(Shows footage of Luna playtests)
EDUCATION
At the same time, that we're thinking about how presence influences gameplay, we're also seeing how presence may in fact to influence education, there's a lot of stuff if you just Google VR and learning, we're doing a way better job actually of selling this, even though we haven't actually implemented any of it yet. It's very interesting to think about moving away from the blackboard and sitting down and actually, you know, working in a virtual space on a task. And the words that we use to describe this are body learning or self-determination in classrooms. And there's a huge interest in this, especially in underserved communities where education really sucks.
Engaging in a task or teaching others how to do something is the best learning experience. And sitting listening to someone talk, which I'm doing to you right now is the worst.
Hunicke, Robin. 2016. Understanding The Body's Role in VR & AR Game Design. In GDC Vault, edited by GDC. U.S.: YouTube