lance weiler - STORY DRIVEN INNOVATION
Lance Weiler, founding member and director of the Columbia University Digital Storytelling Lab shares findings from the labs re-imagining of the work of Arthur Conan Doyle titled Sherlock Holmes & the Internet of Things, a prototype that had 1,200 collaborators from 60+ countries working across 70 events to create a massive connected crime scene.
Identifies 4 Immersive Participation Design Theme Principles
1) Trace: Provide a way for people to leave their mark/personalise the experience
2) Grant Agency: Shifting between individual and group tasks helps to promote, as well as reinforce a sense of individual agency.
3) Thematic Frame: Everything is so much easier when you're working with a known starting point e.g. a popular source IP
4) Serendipity Management: Let things evolve/Aim to show, not tell and leave gaps for the audience's own interpretations.
In addition he recommends dividing participants into groups of 5 - 6 people
The Digital Storytelling lab kicked off a prototype about a year ago called Sherlock Holmes and the Internet of Things and it's kind of like peanut butter and chocolate right? This idea of mixing Holmes and the IoT (the juxtaposition is consciously unexpected).
Because what was fascinating about Arthur Conan Doyle's work is that he was writing about emergent technologies that nobody knew about outside of certain small circles: non contaminated crime scenes, ballistics and different types of blood tests. Law enforcement was reading that and saying, wow, these are really good ideas, we should adopt these.
And so in a lot of ways, Arthur Conan Doyle was starting to influence fact. ... and I love this quote by Arthur C. Clarke.
“Any sufficiently advanced technology is indistinguishable from magic.”
And I think the Internet of Things is very much like that for most people. ..all these common objects within our lives connected to networks seems like magic.
We also wanted to experiment with this idea of copyleft, so we made the whole experience open to whoever wants to be a part of it, all over the world. It's released under creative commons, share alike license, so people can commercialize whatever they create within it as long as they give back the code base and share their learnings.
Taking this original source material from Arthur Conan Doyle, and using it as this fertile playground to start to experiment with what we could do was a large, massive global storytelling experiment.
THE GLOBAL STORYTELLING EXPERIENCE
The experience starts when people go to the source material. We ask them to find an object from one of Arthur Conan Doyle stories and adapt it using physical computing sensors to create this enchanted object that's intended to be something that they can use as a way to tell a story.
We didn't plan to use IOT at the start. Instead, we started for about six months through a series of meetups, and did everything in physical form, paper testing.
Our analog prototype consisted of four key areas. It would take about 90 minutes to run it.
TEAMS OF 5 or 6
5 - 6 is optimum.
When` we had less than five, certain personalities would dominate and people would have a miserable time. And then when it was too large, it started becoming almost like a consensus vortex where nothing ever really happened.
1. Create a crime scene.
We hand each team a roll of masking tape, and say okay, now go out and create a crime scene with your fellow teammates.
2. Create clues
When they came back we would lay down Brown. paper and tell them to empty their pockets in their bags and define three things that they thought were really interesting to tell a story with.
And then they would just trace those things. So if they had, you know, sunglasses or their phone, or whatever it was, they would trace the outline of it. And they’d do that individually.
3. Mix it up
Then we ask them to step back, pick three things that aren't yours, rip them out, cut them out, and now you have them. Go back to any of the bodies except the one that your team just created and place these objects with purpose in another crime scene, and be creative with them.
4. Set the scene
a) 1st individually
That famous phrase by Ernest Hemingway: "For sale. baby shoes never worn" a very evocative statement, that's this idea of generative, flash fiction.
We gave each team a packet of post it notes and some pens. You can go around, you can pick up that phone on the ground to hear the last voicemail message from a crying ex girlfriend. If you see glasses, you can leave teeth marks on broken lenses. But you do it as an individual. The only rule is you can't make any changes to your own crime scene, your own body, your team's body.
b) Next collaboratively
And then they come back and we'd say, Okay, now you're a team again. Go back to the dead body crime scene that you originally created, but you're going to realize that there are all kinds of clues there now.
5. "Solve" the crime
Now we'd challenge them to be a collaborative Sherlock Holmes, by making up a story about what happened here. They could pick and chose what they wanted to respond to. But then they crafted these narratives about the scene that were incredibly dynamic and they did it very quickly.
SCALING THIS EXPERIENCE
So we workshopped what to do through a series of meetups and then with a global community, which probably came close to 2000 collaborators from 60 different countries. Last year, there were 70, self organized events all over the world. There's been a ton already this year.
FOUR DESIGN PRINCIPLES emerged from this prototype. And remember we tested this thing all over the world.
People really responded well, whenever they could see some element of themselves within the story, you know. And what's interesting is when we first started this, the very first prototypes, we thought we'll write the stories will populate the crime scenes. We'll lay out the bodies and then people can come in and be Sherlock Holmes and Watson and they'll love it.
But what was actually more engaging was when they were creating everything themselves, and we were just letting it happen. So that one step of the trace was really important where they could see a contribution of themselves within the story.
2. GRANT AGENCY
Breaking the event up from a group task to an individual task and back and forth led to these really these potent moments that allowed people to feel like they actually had an impact over what was going on.
They didn't feel like they were just following somebody else who might be a dominant personality within a team. And that was that was interesting.
3. USE AN EVOCATIVE/ACCESSIBLE THEMATIC FRAME
Most people probably know Sherlock Holmes, even if they don't they know what a detective mystery is. And if they see a body on the ground, they probably know there was a crime. So there's a common language there that that really helped quite a bit. It made the experience accessible and quickly gain context, mutual understanding, engagement and back story.
4. SERENDIPITY MANAGEMENT
A classic screenwriting principle is that you're supposed to show, not simply tell, but interactive designers often get very concerned that people won't know how to interact with something in the proper way, so end up telling too much.
But we left a lot of gaps in this experience and because we left those gaps, the imaginations of the participants filled them, leading to these really wonderful kind of moments where they would collide into each other in unexpected ways. And these collaborative, creative sparks would pop up out of it, which was really exciting.
When we were trying to work out how to manage the creative process of thousands of people. We ended up running an open MOOC usually known as a massive open online course, but we called it a massive online offline collaboration, open to whoever wanted to participate.
So it was interesting how something so simple was so effective in terms of people being able to organize themselves.
We had this road rotary phone that had a Raspberry Pi with a text to speech engine. So you could write to it from a distance and it would ring, it was also beacon aware so whenever the phone got near something, it would recognize that it was there and ring, right. So then all of a sudden, somebody would start interacting with a character, you know, a killer and a cat mouse game or whatever.
We made that code open source because we want these IoT objects to be something that stimulates creativity within a group of people.
Ubiquitous computing is like calm technology, it's all about being in the peripheral, it's not necessarily about being buried in your device. And in fact, when we did have a device and we used it, it broke the flow, because everybody became very interested in the screen. So what we're trying to do with this is say, can we use objects that are enchanted in some way or smart objects that people can use as a way to tell a story. So it becomes about the human interaction and not necessarily solely about a screen interaction.
And the other thing that's interesting about the project is there's no win scenario. We remove that, you know, there's not one right way that a crime is solved or crime is created. So it becomes very much a collaborative form of story and play.
At TheDigital Storytelling Lab we’re now exploring participatory urbanism and the collaborative design of neighbourhoods through storytelling. For example we partnered the City of Los Angeles and did a storytelling innovation lab with them. The project was called My Sky Is Falling and it was shaped with foster youth in collaboration with the students.
70% of whom tend to end up addicted to substances, pregnant or homeless. It was heartbreaking. But together with the students they made an immersive science fiction story. We also teamed up with MIT and they gave us a bracelet that participants could wear to track their emotions as they went through the experience. And instead of relying on pre survey and post survey, and audio video transcription we came up with 26 different feedback loops within the experience itself. There's a really cool white paper on it if you're interested you can find it at myskyisfalling.com
When it pointed to and, just like Sherlock was this idea that stories can spark a really interesting innovation because that prototype was made for a couple hundred dollars. And then the UN invited us to run it there - and now it's being adapted in three different states as a framework to train, potential foster care parents and social workers to understand the emotional journey of a foster youth because when you go through the experience. .So this idea of inspiring greater understanding is really interesting and making use of technology to do so is something that we are very interested in doing
STORIES ARE NOW EVENTS
This is an amazing time to be a storyteller. A story maybe isn't just by one person anymore, it’s a collaborative community event. And increasingly now we’re seeing the rise of creative technologists who are using stories to create a common understanding.
EVENTS CAN CHANGE THE FUTURE
I did a project with David Cronenberg called body mind change. And it was kind of about this idea of a personal recommendation engine that you just put into the back of your neck through synaptic entanglement. That's a word we made up it didn't exist. The engine would know everything you wanted needed and desire before you did. And so that story was all about the quantified self and about artificial intelligence and about this idea of emotional intelligence. But what is interesting about this the moment is that we have the opportunity to change the way that these stories are told.
For example, if an object is connected to a network it can send you media. You could build subscription models off of that. That object could change over time, it could change state, it could recognize other things around it and it could change the way it interacts. It could do simple things like change colours, it could change the lights when you walked into a room, whatever you want.
This idea of allowing those formerly known as the audience to be collaborators in the creative process allows them to create something or co create something that they care about, that can evolve over time. And new business models can come from it.
But this apparoch also breaks from permission based culture, so you don't have to spend years trying to convince somebody that you want to make something, you can just make it.
Weiler, Lance. 2018. Story Driven Innovation. U.S.: fitc events.
Narrascope 2019 - David Kuelz - Designing Games that Listen
I'm going to talk about a game that I did a lot of the writing and narrative design for recently called Starship Commander, a virtual reality game that is driven entirely by human speech, which means that there is no controller. Instead you talk out loud to each character via the microphone. We run that audio through our system and then the character you spoke to responds, so it's a game that you play by having an elongated conversation with the script.
This speech recognition technology we're working with was disruptive. It gave players a lot of agency. They could say anything they wanted to at any point they wanted to. There's no point in which the player can't talk. That both opened up a lot of cool doorways of things that only we could do. But it also opened up a couple of Pandora's boxes of real design and production problems that we knew we had to approach carefully.
So almost every decision I made about the narrative design was trying to work with the technology and figuring out how we could highlight the parts we wanted to and either avoid or at least contain potential problems that were coming up.
THE DIALOGUE STORY IS THE GAMEPLAY
We built Starship Commander using the Unreal engine because it had two features that we really wanted. First was the blueprint system. If you guys aren't familiar with blue, It is kind of like having twine embedded into the engine, and I say kind of because it behaves very similarly. Similarly, it's about creating a flow of logic. Once you're used to nodes and branches and flow, you can learn blueprint very easily. The trick is that while twine artists draft really for story, logic, and story flow blueprint is really for game logic and gameplay flow. It's about what buttons the player is pressing and how that changes all the math and the numbers we're keeping track of and what that means for our animations and sound effects and enemy AI and all of that. So you build it in a similar way, but it's for a different type of logic. And usually those two logics are kept separate from each other.
In most games the combat or whatever it is you're doing, usually doesn't affect the story. So we'll pause the gameplay and nothing's going to attack you, and you can decide what dialogue options you want to make.
But in our case the only way that we had for the player to interact with the game was to speak to a character or their ship… So we needed one spot where we could build both of those in the same place and see the entire flow all at once. And blueprint was the answer to that for us.
The other thing that Unreal did that we really wanted is it allowed us to build our own tools. We actually initially attempted to build our own voice and speech recognition system. before eventually deciding, let's just use something someone else has already built, that's cleaner and quicker. But in that case, Unreal allowed us to build in those outside sources to the code to the engine and a way that was effortless.
As soon as the Oculus microphone picked up what people were saying, we would send it out to Microsoft and Facebook's wit.ai program, we use those two services because they had the least amount of internet latency. And then it would send back text.
NATURAL DIALOGUE AND MANAGEABLE INTENTS
We really wanted the dialogue to feel natural, which meant there were Hundreds of thousands of potential things that anybody could say at any point in time, so we needed to funnel those down to a limited number of things we could actually script.
So our programmers built this tool kit and this sort of flow of logic that we called the intent system. It works this way. any individual idea or concept of something that the player could say, was labeled an intent. So up there, you're looking at a question, Where are we going? As an intent? It could also be, who are you? It could be a comment, like, I think that's a bad idea. But any individual thing that somebody said and that we wanted to have a unique response to was an intent. Inside each intent. There was a list of utterances and utterances were essentially all of the different ways somebody could phrase that intent. So if we were going to use Who are you? as an example, some of the utterances inside that might be Tell me about yourself or What's your story? Or Tell me your story? Those are all different ways of phrasing that same idea. So ultimately, utterances became all of the different ways a player could activate this one intent.
And the last thing we needed to do was take those and phrase them in the context of a specific conversation. Because what we're saying can mean very different things depending on the context, like whether or not a provocative comment is a joke, or an insult. So, somebody would speak into the microphone. We would send it out to Facebook and Microsoft get back text, the computer would then figure out what model am I in, which is something I would determine for it in blueprints, then it would check all of the intense inside that model, and all of the utterances inside all of those intents and that would run an accuracy test and the utterance that was the most similar to the text that Microsoft sent us back determines which intent would activate. And then we had all of that system hooked up to a custom node here in blueprint that depending on what the player said, would take it off on different branches of logic and that's how our game would react to the players speech.
FULL MOTION VIDEO
We created our content using full motion video, partially because it was really quick and cheap and easy to do fast
BUILDING STORY TO FIT THE TECH
Once the tech was in place I was hired to build a plot to match. There were a lot of really cool, exciting things that we could do with our system and we wanted to make sure we had a story that capitalized on it. And there were also a lot of pitfalls that we thought we could use the story to kind of help mitigate. So I ended up coming up with a plot that was not necessarily about how excited I was as a writer to tell the story, but that I thought would fit the technology. And I was aiming to do it by thinking about these four things, which I will go through one at a time.
IF YOU”RE USING BRANCHING DIALOGUE KEEP YOUR EXPERIENCE SHORT
If you have a story that branches heavily a section that takes the average player roughly 15 to 25 seconds to get through demands a lot of writing and recording and programming. So we knew we weren't going to be able to keep this up for a long experience. Instead we're aiming for a short 90 minute to two hour experience.
YOUR BUSINESS MODEL CAN BE A CREATIVE PROMPT.
We wanted players to be comfortable dropping a little bit of money on out game, even though it was really short. SO we made it re-playable because we already had all of this alternate dialogue anyway, just by virtue of the technology. So, we did things like enabling players to find branches just by saying something specific not because they were presented with a choice. This was simultaneously a design and business decision.
TRACK AND PLAY TEST REAL CONVERSATIONS AND USE CONTEXTUAL CONSTRAINTS TO HELP MANAGE THOSE CONVERSATIONS:
During every play testing session we recorded all conversations and we would be uncovering a lot of utterances that I hadn’t thought of – and that worked, but it was still a time intensive process. So I also needed to find ways to contain the number of intents that we would be plausibly expected to have. So I created specific contexts, around scenes. Essentially, I decided to write a thriller, where every single scene has this imminent point of conflict and danger with a ticking clock element because in that situation, it does not occur to players to ask, “What's your mom's favourite colour?” This also gave us an easy out anytime someone made a comment or had a question that we hadn't thought of, so a very logical character response would be 'What does that matter right now? Can you please focus on the bomb?'
BTW we would play test in New York and because of the way people in New York naturally spoke, a scene would flow well. And then we would take it out to the west coast, and people would have different ways of putting the same idea together and it would get stumped. So actually, a big part of what we discovered with the play-testing process was to take it around to a lot of different local areas.
The biggest problem that we had was ambient noise in the room, disrupting what people said.
There was also delay in the dialogue, we couldn't avoid that but we disguised it a little. Because you are in your ship, and the entire show we disguised it as static. Anytime we needed to buy a second, the screen would fizzle and then they heard you and talked to you.
USE SURPRISE BRANCHES.
When your player can say anything at any time, at some point, some player is going to say something that disrupts the plot. Now on one hand that was a problem, but on the other hand, it can be an opportunity. Because if a player is clever enough to be able to disrupt what's supposed to happen with genuinely good story logic, we wanted to be able to reward them for that by having a concrete change in the story result. But that also put us in a weird situation, where all of a sudden, we were at the risk of players creating endless new branches, so I wanted a plot structure where we could add branches as we needed without having to build a whole lot of new content.
We also needed to account for the knowledge gap between the player and the avatar character. In our situation, the player was writing their own lines, without knowing any of their avatar’s backstory. And we needed to make that plausible or help the player figure out how to be this person.
So I created a MacGuffin. This is the anomaly…
At the beginning of the story humanity controls the MacGuffin, but they don't know what it is, or does. And in the opening scene, players role play a scientist trying to figure out what it is. They fly a one person ship into the anomaly in the scene, merging with it, potentially explaining any strange memory loss and also potentially explaining why the player know things about the future because the characters shouldn't. but maybe they've experienced alternate timelines.
In the middle of this experiment, it is disrupted, everybody is attacked by a race of war like aliens and the anomaly disappears, it blinks out of existence. It goes somewhere else within the Milky Way galaxy. They rest of the story is about you the one person with this mysterious connection to the anomaly and your military escort Sergeant Sara Pearson, exploring the Milky Way trying to figure out what the anomaly is, where it's gone, and what the human race want to do with that when they find it.
This structure gave us a lot of freedom to improvise later on. ... A mystery is made up of clues. And those clues don't necessarily make sense individually, but when you put them together, they coagulate into the bigger picture of what is happening and why it matters. As long as the clues are structured in a way that's clever, you don't necessarily need to have all of them either. When you start, you start out in the prologue scene every single time you have an experiment with the anomaly. And then depending on the choices you make in that scene, we send you to one of four different tier one scenes. In any individual playthrough you will only ever play any one of these missions and these missions are all isolated from each other. The plot of each mission you go on is dependent on where you are in the Milky Way galaxy, what is happening in this part of the galaxy, and they don't directly relate to one in each other.
So once you get to the rescue mission, that mission will proceed as usual, regardless of what missions you've happened to play before it. But the reward for each one of those missions is a clue in the greater story of what's happening with the anomaly, and why.
There were different combinations of five clues that I had to make sense and I always had to get the player to the finale of where the anomaly was and why it's there. But once I knew there's only ever going to be one tier one clue. There will only ever be one tier two clue I could create a sense of progression. A good mystery isn't, Oh I have one clue. And it doesn't make sense, only when I have five can it be solved. Instead you want to slowly build and curate a picture of what's happening and then start to correct it. Now that I have two clues. I get the sense that these are connected in this way, but I don't know how yet. And now that I have a tier three clue Oh, that's different than what I thought that it's starting to get clear. And depending on the progression the player took through the game, they would get a slightly different version, a different perspective of what was happening and why you couldn't understand everything about the story until you played it through multiple times.
Let's walk through an example. Let's say the player has played one playthrough of the game, and in the first playthrough, they participate in the alien ship mission. And they get that clue about the anomaly. When they restart the game, technically, that fact doesn't exist in the game, but the player still knows it. The story is the same every time the player just gets different pieces of it. So potentially, the clue that the player learned in the alien ship, they might be able to use to change the way the traversal Memorial pans out. They might want to create a surprise branch there. And we could actually then reward them for that by making a branch that goes from the traversal Memorial sit down to Caltech runes, that would otherwise not be connected. So we can rearrange the flow.
Also at any point, if we're running out of time, or there's a tech problem that we don't know how to solve, we can just say, you know what, let's get rid of that scene. …and the reality of the story structure is still intact. You could make loads of cuts and as long as you don't make too many cuts from any one branch, the actual story is still playable. So we could sort of decide as we went and adjust ourselves from there.
Certain types of stories are more meandering than others and you do not want that. And when I think of sci fi and fantasy, I usually think of the setting as a huge star, but ultimately, we found that meandering was not a fun activity and very difficult to build. So in terms of certain genres of story, I would say, plot driven stories, like thrillers specifically, or maybe a very strong character driven story like a romance might work really well for this approach.
BUT PLAY TESTS SURPRISED US
As it turns out the players have a much better experience when you actually do have a lot of control, and you really can guide them through something specific.
One thing we had learned from actually our very initial brief tech demo scene, was that we really needed to have a way of allowing the conversation to continue regardless of what the player said if they happen to get stuck, because in a system that is this open ended, we really didn't have a plausible way of suggesting to the player what they should say to another human being right now without breaking the fourth wall in a way we didn't want to do.
But when it is that open ended, and when you are waiting for the player to say something specific, like I'm ready or Let's go, ultimately, it's not going to occur to a lot of people to do that.
There's a lot of people that are going to get stuck not know what to say and loop around and endless spiral. So we tried to have a way of progressing automatically. At some point if enough time had passed we would have the character, “You know what, the anomalies out there, we got to keep going and we got to move”. Down here, that's a different face of the conversation with different contexts and different reactions to what you might say. We did that in a very linear way.
But it is very, very difficult to write compelling dialogue when you only know what one half of the conversation is going to be.
I didn't know what my character were reacting to. I didn't really know what the player just said, I knew the idea. I knew the intent. But I didn't know the words. I didn't know the tone. I didn't know the utterance. And so all of the dialogue that I was putting out was very generic. It was very vanilla. I wasn't in love with these people. It wasn't good writing.
Eventually I realised that this was that this was a gameplay problem, and not a writing problem. So it wasn't that the writing was bad. It's that the flow of the activity was not fun. I took the time to do some extra drafts and try to find voice. But what actually I should have been doing what would have been more productive would have been treating my initial draft in twine, not like a draft of a script, but like a paper prototype for the gameplay to try and figure out if the actual activity of this specific conversation and the way it moves is fun.
When we did start play testing the scenes, we discovered that players are actually very bad at coming up with things to say. Most of the things that they said were incredibly short, very simple. A couple of words of one word answers.. 90% of players just did whatever they were told, without getting into it. Only about 10% of players actually explored.
Unlimited freedom shuts down our ability to actually make a decision.
So in open ended sections in the script, we would say, do you have any questions? Do you want to know about me? Do you want to know about your ship? Do you want to know about this planet? The aliens? What do you want to know about ask? People go? I don't know. I don't have any questions. And we keep moving.
But the concept of asking anything turned out to be an inherently unfun activity. The more intents we have, the more utterances we have to have for each one. The more directions we go, the more general we have to be. But as it turns out, that the freedom wasn't what players wanted, they wanted to be listened to, very, very specifically.
Instead, the consistently successful part of our play test was a very brief moment early in the briefing scene, where Sergeant Pearson is filling out some of your information and she would ask How old are you? And we had written out a response of every age from like five to 95. Nobody came in with an answer. We weren't prepared. But what worked about that moment is because the constraints of the scene were very tight, so we could build out 90 different personalised answers to this one question.
And in that moment, players felt listened to. And more importantly, because we knew what was going to happen we had a funny little joke for every age. If you came in and said, I'm 43. We have a funny little joke about 43 year olds for you. Now that did a couple of things. This call and response dialog did a couple of things for us. But most importantly, it gave us a feedback loop.
The idea is that it's the physical concrete process by which people play a game, which usually means that they push a button. That action has an effect inside the world of the game, but the loop isn't complete until the player is informed about the effects of their action. Through feedback, the player presses a gob or presses a button and attacks the Goblin. Once we have an animation where the Goblin staggers back and a little sound effect where it yells and a blood texture shows up on the floor that's when the player really understands what they did and are equipped to now take another action.
This is where controlling the context of each scene became very important. Your character could legitimately take the reins to focus in on this one thing that's very important that we have to do. But then because we're not doing anything else, we can create all of these different options and listen to the specific things that each player says, each utterance can become its own intent, and then we can respond to exactly what they said and the exact words that they used, in a way that's immediately engaging and rewarding. Once I did this, I immediately knew how to write again. I immediately had the ability to look at exactly what a player had said and bounce off of it in a way that uses their own words, and I could create dialogue that was engaging. And the writing became a reward for the players creativity. When the player said something specific, a little bit unique to them, we could respond to it in a way that was actually entertaining, we could respond in a way that had value in and of itself. And when we rewarded players for being creative, all of a sudden player started being creative. They started opening up they started talking a little bit more as if they were talking to an actual person.
So overall lessons:
- Pick technology that you can control pick technology, you can build your own tools,
- Pick a plot where you can tightly control the context of a scene.
Kuelz, David. 2019. Designing Games that Listen. U.S.: Narrascope 2019/YouTube.
The USW Audience of the Future research team is compiling a summary collection of recent research in the field of immersive, and enhanced reality media