A panel discussion from the Narrascope Conference, Boston, June 15+16 2019 - Celebrating Narrative Games “Dissecting the Bandernatch With a Vorpal Blade: What Netflix’s Choose Your Own Adventure Got Right and Got Wrong” was Narrascope’s panel discussion. Heather Albano, Mary Duffy, Jason Stevan Hill, Emily Short, and Ian Thomas discuss Netflix’s first venture into the world of interactive television with Bandersnatch, the Black Mirror episode that let viewers influence the story in a choose-your-path adventure kind of way. What was good, where did it fall flat, how would experienced game designers have tackled this kind of project? But beware, there will be spoilers!
​

DISCUSSION SUMMARY
 
DIFFERENT AUDIENCE REACTIONS
Reactions different wildly from those familiar with the format, and the many audience members who had never previously been introduced to it.
 
FAMILIAR DESIGNS
Familiarity was again seen to be an important design tool.
 
CONTROVERSIAL CHOICE DESIGN
Not all choices made sense, or appeared to have purpose
 
THE IMPORTANCE OF ENDINGS
Viewing experiences were very different according to the endings reached
 
UNIVERSE SWITCHING
Too often the reality of a context was undermined, when a more interesting solution could have been found.
 
THE IMPORTANCE OF INTENTIONALITY
The writers could have done more to explore intentional participation through choices
 
GIVE PLAYERS TIME TO CONSIDER CHOICES
The decision to freeze action and give players time to make choices was generally received well.
 
FLASHBACKS
The option to step back in time was innovative, handled well
 
FAILING FORWARD
Whilst branching narratives are complicated, production load can be minimised by techniques to group potential responses thematically
 
MISSED OPPORTUNITIES

1. Explain apparent continuity errors:
2. Intensify choices and limit replay:
3. Think about tone and context:
4. Focus
5. The power of authentic social messages
6. Respect consent
7. Complicate control

 
 
UNCOMFORTABLE SOCIAL MESSAGES
A number of panel members questioned social assumptions within the game
 

 
DETAILED DISCUSSION TRANSCRIPT
 
As a gamer, what did you learn? And what did you hate?
(spoiler alert)
 
 
DIFFERENT AUDIENCE REACTIONS
 
By the time `I watched it there were already a whole lot of game designers who had seen it and panned it. Where's the innovation they asked? … and yet at the same time there was a whole bunch of consumers saying, Wow, this is exciting. We've never seen anything like this before… is a really interesting disjunction, I think.  And so my first real reaction to it was that it introduced a lot of new people to interactivity, and that's a pretty major plus. 
 
FAMILIAR DESIGNS
 
They clearly built this to feel familiar for television viewers, the way it handles recaps, the way it handles changing audio and those kinds of elements. There was a lot of television craft that went into it. And that's part of what makes it accessible to those kinds of people. As a result I found myself a little bit more sort of forgiving of some of those elements.
 
CONTROVERSIAL CHOICE DESIGN
 
But also, I got very frustrated really early on with the first few choices that just didn't seem to do anything very much, with no immediate consequences that just railroaded you straight back in to the narrative.
 
I felt sympathetic to the possible design reasons for those choices because it felt like …We've got a TV audience, we need to get them used to the idea, they might even need to pick something. And if they miss and they fail to make a choice early on, we don't want their first experience with this style to be …oh, I missed out on doing something interesting because I didn't really understand the controls.
 
It’s interesting to think about it in terms of a product that has to teach its players how to play. It reminds me of something I encountered when I did customer support for Choice of Games. When I first started working for them, I would get emails from people basically saying “Please put a back button to your games, I picked the wrong choice, and I have to do the whole thing over again”. And I would diligently respond that there was no wrong options. But there are arguably many points during Bandersnatch where viewers can make the wrong choice. In fact audiences proceed directly into what seems to be the most important choice right off the bat in Bandersnatch, namely… Should I work in the Office of Tucker soft games? Or should I work on this at home? And everything about the way the scene is constructed, says, work at the office, you're going to have all this support that you don't have as an independent game designer, and that's the wrong choice. So I'm so frustrated by that particular moment.  It seems to be teaching audiences that their choices don't matter. Because what seems obvious to you is wrong. And so, you know, in a game that's trying to do all this layered metaphorical stuff about control…That's the first real choice. And it's tricking the audience. And that was such a turn off to me.
 
I agree 100% .  When I hit that first choice, and went into the office and died, it's like…really?  So I recognize the cultural impact that it has.  Introducing lay people to interactivity is cool, but in many ways they’re replicating what I consider to be bad habits that I want to move away from as a design culture. So, as a designer and as a player, I myself did not care for it.
 
Okay, that's fair, but I actually had a different reaction to that first choice and the shortness of the path. I thought it was an interesting design choice because if you have players who have never seen anything like this before it teaches them right up front that this genre involves replaying the same scenes and making different choices. In that context I thought the confronting delivery of the need to reply choices was forgivable.
 
THE IMPORTANCE OF ENDINGS
 
I also found that personally, I had a different reaction to the experience as a whole depending on what ending I got. And I feel like that's something that might be interesting to delve into.
 
Me too.  I enjoyed the piece more than a number of my friends because the ending that I got was the one in which he winds up being reunited with his mother and sharing in her death on the train. And that, to me was much more thematically resonant than many of the other options because I was much less interested in sort of gotcha endings where, you know, he fails to make a good game for some silly reason, or, you know, the whole thing devolves into this weird murder plot. I felt like the business about what has happened to him in the past and how he can revisit that trauma was a story that was less obvious to tell based on the form.
 
So what I mean by that is, I felt like a number of the paths through Bandersnatch were basically things where Rooker was asking questions about freewill or about you know, what does it mean if you have somebody, a player controlling the protagonist, which basically have been asked and re-asked by one interactive narrative designer after another.  It’s almost the first thing that people do when they realize that this form exists.  So, what I found most interesting was the case where the story it actually kind of gone beyond that set of probes into something different.
 
I agree. I got the same ending and I think it was an extremely poignant ending… choosing to go and die with your mom on the train. That was like a good Black Mirror episode to me. But when I was looking at the different endings that ending did not appear to be common.
 
UNIVERSE SWITCHING
 
Each ending almost catapults you into a different universe.  In some endings you are definitely controlled by a person. You’re making a movie. Actually, you step out of it, and you're the actor or the Stefan that you are controlling is an actor on set, which is a totally different universe from the other universes where you die, or you're the mother. So there's Something really strange about different roles stepping out beyond the original fictional frame and ending the story outside of itself. Great for the cognition of the player, because the player has built an idea of the world that they're in, and then suddenly you shift that from under them.
 
That kind of universe switching approach can be something where you're kind of asking the player, what kind of story do they even want out of this? What are they interested in? But the trick I feel in the case of Bandersnatch, is that it's, it's hard to actually articulate that agency As a player, although I had ideas about what I wanted to explore, there are only a few points where I really felt like I had the agency to articulate that to the system.
 
THE IMPORTANCE OF INTENTIONALITY
 
Not me, one of the things with choice games that we really tried to focus on is the idea of intentionality, that you have some idea of the possible consequences of your decisions. And you're in there with the therapist and you get into the fight and then it's like, jump on your dad or jump out the window. But if it turns out that you're just an actor that's forgotten the fact that you're an actor… that complete lack of intentionality was frustrating for me.
 
Also, if you have X number of options, and the player is failing a number of hidden tests, so that only one option appears instead of all 10, there's something wrong. And I question what that's supposed to do for you as a player in Bandersnatch.
 
Yeah, there were a couple of choices that seemed to duplicate each other. I mean, throw the jewels at the computer or destroy the computer. I don't want to do either of those.
 
And there are some that are explicitly the wrong choice where if you decide to throw the towel on the computer, it immediately brings you to try this again or go back or something like that, which I hated. I thought that was a cop out if you're going to give the player choice, it should be a choice that goes somewhere.
 
GIVE PLAYERS TIME TO CONSIDER CHOICES
 
That said, there are some interesting things in those choices just from an execution point of view.  We've all seen all the FMV games where you hit a choice and everything stops while you make a decision.  So it was quite nice that they had the kind of live feed happening in the background and that during that the character you were responding to would generally prompt you and put a bit more pressure on which is interesting. But there's also a flaw in that which is, it's like the choices out of walking dead where you have three choices actually left to right or fail to act.  Bandersnatch had that as well but didn't really seem to do anything with the fail to act option, other than auto-choose on your behalf, which was just a bit strange because it meant that you could just sit through it and sit back and watch the whole thing and it kind of loses the point a little to me because then you're not in control of anything.
 
I think there is a different emotional effect since you have as much time as you want to sit there and think about what you want. And so it's, you know, fail to act means something different than if you've been able to think about being inactive, and that's my choice, right? Whereas if, you know, you're sort of sitting there and and you're troubled by the question, and you're having a hard time making up your mind, or you somehow don't really want to commit and your time is running out that feels different. Yeah. So I'm a little bit more patient with it in a timed context, and I think it's more expressive actually.  So it wasn't so much that I wanted a third option to press. Yeah, it was more. I want there to be a consequence of failure, like an unexpected ending, or something….
 
FLASHBACKS
 
And I guess we should talk about flashbacks as well, because they were effective. Going back through the loop, and shortcutting all the sequences and seeing the results of some of your choices worked really well, brought it into a TV form really well.  But, but there was never any attempt to explain in the fiction how that happened. I mean, you can look at something I like that the strange reversal of going back through choices is very explicitly laid out in the world. How are we repeating this however?  Do the things I did last time apply to this loop? That's cool, but it lost me as a player. What is the friction? What's the infectious reason for this strangeness?
 
I'm struck by how differently I reacted to this emotionally depending on which of the endings I happened to strike. I didn't get death first, although it's my favourite because I agree with the emotionally resonant. I got, history repeats itself, where pearl ends up finishing the game, which I actually thought also worked as a black mirror episode in a very different way that it was very intellectually clever rather than emotionally resonant. I'm good with either ending, the one that annoyed me to death was the Netflix one that just struck me as the kind of thing you write at three o'clock in the morning when you're a bit blocked and it seems funny at the time.
 
I was listening to Charlie Brooker the other night talking about that, and that wasn't supposed to be a main choice, it was supposed to be a bit of an Easter egg. And they went through testing and decided to put it in.
 
And then I found the government control paths, which Stefan mentioned and the whole thing If that path felt to me, like they'd hacked it in that weekend, and that was when I started to wonder how deliberately the creators were commenting on different types of interactive fiction. I suspect that they haven't read that too deeply enough into the various types of interactive fiction to be that explicit about it. I think a lot of the impression I got was they wrote a huge tree in twine and then chopped lots of bits off when they realized it wouldn't fit into production.
 
When I was replaying it this week, I wanted to follow through all the leaves have an individual branch, if you will, and I was frustrated to that degree where it would take me farther back than I wanted to go and not let me explore all of those leaves. That said, I thought that the recap in particular was the best part. Like that was a really an innovation. It's not something that I'd seen before that I think will probably become a gold standard for this type of format in the future.
 
 
THINK ABOUT WHAT CHOICES MEAN
 
One choice context that I found particularly effective was the kind of locked, safe situation where the story presents you with the protagonist’s father's safe. And there are different storylines that could lead you to think that different things might be inside. You might think that it's about this government control program.  You might think that it contains something that's quite personal. And you're offered options about what combination to use to unlock the safe. And it presents the task as though it's a puzzle and as though you're supposed to solve it by knowing what the combination ought to be based on what's happened previously. But actually, all of the combinations are words that connect somehow to what's been going on in the story. So the choice that you're really making is what do I think is important enough to be the thing that might be in the safe at this moment? And so, although it disguises itself as a puzzle, it's actually asking what do you as a player care about, and that I thought was quite cool.
 
It actually works better than it would have done if there was a parser type answer, because it's making it really clear there are potentially multiple right answers. And if you revisit that choice point, having come through different avenues in the store, you're actually offered different combinations solutions, which lead then on to different discoveries in the safe.
I think that's what the fight sequence is about as well…what type of game are we playing?
 
Yeah. I managed to miss enter that code through clumsiness and get it wrong in playtesting. Just not enough people were getting the number. I also had to be much more explicit about it and retire that as the main puzzle from what I gathered from the script,
 
Okay. But the question I got about the whole thing, actually, in the podcast.  They interviewed Charlie Brooker about Black Mirror in general. And the impression I get from that was that Netflix said, Would you like to do this interactive fiction? They went, No, we don't want to. That's really nice. But we want to leave that. And then came up with an idea about a Black Mirror episode involving control and agency, and then did it the other way around, and went, Oh, no, this has to be interactive. Which at first involved trying to build this massive, massive branching tree narrative, and then production reality hit that massive tree. And so, what we've got left is really a very sparse impression of what the original structure was supposed to be. Because, as we all know, having again been through this loop over 30 years or whatever, the combinatorial explosion just doesn't work for filmed content really well.
 
FAILING FORWARD
 
And it made me wonder, like, why didn't they structurally do more with what we call failing forward (trial and error)? For example, you might have a choice with three options. Each option is testing a certain stat or maybe a certain Boolean for the past thing that you've already chosen, and it tests it, and then you have six different possible outcomes that involve either success, or failure. Why not just make two films for six different outcomes? Sure, it's expensive as hell to shoot a bunch of different outcomes. But if you can write something that either fails forward, or succeeds forward, and just elaborate on little moments here and there, then you could you could have something that feels a little more meaningful,
 
The production team were also saying that they tried to record a scene, and then a bunch of different lines to plug into that scene, and found that they just completely lost the emotion of the scene because they didn't know where it was going. So they actually had to record the scene six times, with six different line deliveries.  An experienced game actor, will know how to do that and to be able to authenticate themselves, but they're not using game actors for filming.
 
 
MISSED OPPORTUNITIES

1. Explain apparent continuity errors: The first thing that I would do is continuity testing. On my last playthrough, I played Bandersnatch through intentionally leaving Collin alive, and then it forced me to kill Collin. And then he came back to life. And I was like, well, is that a continuity error? Or was that just jumping through multiplicities? If you don't know what the intention is it reads as a continuity error.  It was actually the intention of the author. So, if you're going to have a moment like that, you need to make sure to explain it so that it doesn't read as a continuity error.

 

1. Intensify choices and limit replay: For me, it would be about going off in all these different endings and potential universes, which are a great playground. It means that your content is wildly different on the different branches. If you're producing a whole lot of content, having one reality helps to limit that. I think you can put a lot more variance into the choices, as well as the continuity errors.  I’d also change the replay context.  The first time you play a game, it can seem to be really respecting your choices. And it's not until you play a game the second time that you notice that choosing that option, does the same thing whatever choice you make. So to a certain extent they shot themselves in the foot by having quite a sparse tree and forcing players to replay it at least once. It shattered the illusions a bit.

 

1. Think about tone and context: If they had made sure that the disparate paths that are re-joining contain something that cast a different emotional light or a different narrative context on what follows that would still feel meaningful so you can get away with it.

 

1. Focus: The main thing I would do differently is to drill down on the buffet of themes in this draft, to identify which ones people actually care about. Focus on that. Make sure that more of these choices are either structured in a way that lets the player feel some thematic agency, or else gain a better understanding what the stakes of the choice probably are. And I think if they narrowed the multiplicity down a bit so that it was easier to build kind of a mental model of what the possibility space is, where we're going here, then you'd get a lot more sense of intentionality at more of the choice points.

 

1. Deliver more authentic social messages: When choices seemed to be all about mental illness I was like, what? Like, for instance, you have an option at one point to take your meds or flush them. And then on a successive replay, you come back around to it, and you can flush them or you can throw them away. And I'm sitting there, trying to puzzle that out. And I'm thinking, Okay, well, if I throw them away, my dad might find them in the trash, and he'll know that I haven't taken my meds so then that will make something else happen. And then if I flush them, then the evidence is gone. You know, and you are smarter than the game. And then on top of that, they're doing something with mental illness that I didn't particularly appreciate. So, if you're going to give me binary choices, make sure that they clearly do something. But also, please don't give me choices that are like take your antidepressants or don't take your antidepressants.

 

1. Respect consent: But going back to those choices again, I think there were a whole bunch of occasions where you wanted a different possibility. The one I particularly hated was you can say yes or no, when Colin offers you the harder drugs. And if you say no, he spikes your drink. And I have, I think, never been quite so angry about a fake choice. That's bad writing and appalling social commentary, it just fails in every possible way.  It's a fictional violation of consent that goes with the mechanic. I didn't know that would happen because I've never turned down drugs.

 
I think they were banking on the fact that 90% of the players are not going to turn it down for a variety of reasons. It's the most interesting choice in the scene. Let's do it and see what happens right again, that's the problem with their intentionality. And their design strategy is to make one much more attractive than the other. And then they either reward you or punish you for picking the most attractive one and I'm not interested in being punished and I'm not even really that interested in being rewarded.
 
The first time I got the take your meds or don't take your meds I got to it from the path where Colin has ranted about people controlling you by putting stuff in your food in the context of the government control path, which is not well executed and is not believable. But in that context, deciding to throw your pills away makes a lot of sense. It's a story
 
The second time I got to it. I did not have the context of the common explanation. It was just I'd come from my psychiatrist office. And so I took the meds and then I failed to finish the game and the game that Stefan created was not well received. And so explicitly connecting antidepressants to creativity is incredibly irresponsible. So yeah, I was pretty disappointed by that.
 

1. Complicate control: I think it's quite interesting that we're talking about Stefan being us, actually, rather than that whole separation of control thing that they're trying to go for. So the separation of the control doesn't seem to be quite working. Because a lot of the intentionality in the choices and the reactions in that section where you're poking stuff out to do things and he is getting fed up with the voices in his head and he's starting to try and do something else. Which is an interesting idea, but wasn't very well executed, I think.

 
 
What are the social messages of this piece?
 
I found the mental illness aspects troubling. I also think there was a missed opportunity to go as far as they could have done with the friendship with Collin because it felt like he was a really interesting character and the possibility of comradeship in a difficult emotional space and different creative space is really important. And it is a possible resolution for a lot of the things that are afflicting the character. And the fact that they take you partway down that road and then they say, Oh, he drugs you, or you accept his drugs and then things go haywire. And there's this weird force jumping off a balcony scene. I felt like that was there was something false about that.  It felt like we got to this point because they thought it would be shocking or because they felt like they had to show you having a bad result for taking drugs. It feels like it's an opening to emotional intimacy and connection and you want to say yes to that, and it really bothered me that the story was like a hard no, and not for a good reason.
 
I was genuinely emotionally affected by the death ending. There's a bit at the very beginning where, where you're talking about your mother's death, and the psychiatrist says he didn't know. And so being back there and picking …No, I'm not getting on the train, knowing what's going to happen is also a very interesting emotional moment. I would have liked to have seen them do more of that.
 
Heather Albano, Mary Duffy, Jason Stevan Hill, Emily Short, and Ian Thomas. 2019. Narrascope 2019 - Panel Discussion - Dissecting the Bandernatch With a Vorpal Blade. In Narrascope 2019: YouTube.

Interactive stories are a technologically daunting challenge.  They require several support technologies that are still underdeveloped - e.g. a lot less money has gone into interactive story technologies than AI voice tech, and even that is still developing.  

As a result "interactive story as a narrative form can be said to be in an infant stage of development akin to the first cinematic films of the early 20th century" (p. 2)

"Putting aside superficial features such as high fidelity graphics and animation, the most successful games offer players true agency - the ability for players to have persisitent, meaningful effects on the events of the experience.  Today's games have only achieved agency in the domain of action - or puzzle oriented game-play" (p.2)

EMBRACING THE COMBINATORIAL EXPLOSION:
The quintessential requirements of an interactive story - and a research agenda.
1. Agency is primary - implicit in this is the promise to directly affect the plot of the story.
2. Generation - in order to achieve this requires vast amounts of content, so to overcome branching fatigue and costs we need systems that can content generate on the fly.  Pioneer projects include Talespin, Universe and Minstrel.
3. Interface - Once players are directing the plots of stories about people, expressive natural language and gesture interfaces will be required.
4. Connection - If generation is akin to speaking, and interface is akin to listening, then the two phases need to be connected with the ability to manage the drama and reason about the player's input.
5. Terminology problem - If players have true agency the interaction is not being told, but generated.  "Interactive storymaking" is more accurate, if awkward.  For now, the researchers drop the verb and refer instead to "interactive stories", noting the need for a new term.

Future recommendations including building new systems to enable new types of interactive stories and making them available for public feedback

Stern, A., 2008, November. Embracing the combinatorial explosion: A brief prescription for interactive story R&D. In Joint International Conference on Interactive Digital Storytelling (pp. 1-5). Springer, Berlin, Heidelberg.

Independent producers are not always comfortable using analytics and might link these sorts of metrics to AAA or free-to-play games. But this sort of data can provide valuable feedback about how players experience your interactive media. In this Narrascope 2020 talk Em Lazer-Walker, a Toronto-based artist, engineer, and game designer discusses the benefits, techniques, what to measure, and what not to measure.

My name is Em, I use she her pronouns and I work for Microsoft as a cloud advocate.
I also do a lot of experimental indie game work on my own.
 
This is a talk about analytics.
 
It’s in three parts:

1. How free to play games measure things?
2. What is applicable to smaller games and more experimental games?
3. Is there value in adding analytics to your game? How would you actually go about doing that? What are some practical tools you can use?

 
 
 

1. HOW DO COMMERCIAL GAMES USE DATA?

 
I do not generally endorse the way that commercial metrics analytics works but learning how analytics functions is going to be really useful before we talk about how we can co-opt these techniques for other things.  This is an abstracted overview based on my own experience working in free to play games many years ago, as well as talking to some folks I know who have done it more recently.
 
So if you are a product manager (pm) on a free to play game, you're looking at your game as a product that converts essentially user eyeballs into money. And in order to do that you have the sort of funnel that you narrow in on. You're going to get some new users to download your game, they're going to play it in 2020. This is largely through paying money to acquire these users through channels like video ads. Of all the people who download and install and play your game, some number of them are going to come back the next day, or the next week. Not all of them, but hopefully a lot of them. Of the people who keep coming back and are excited enough about your game and the stories you're telling to keep playing it. Some of them are eventually going to give you some money. And then, of all those players who are giving you money, hopefully some of them will keep giving you more money. A lot of times the vast majority of revenue come from a tiny fraction of players. Again, I am not endorsing the system, but this is the way things are. And so if this is what you're looking at, you can see how a numbers map become important.  Say you're losing people at each step of the funnel, if you can stop the bleeding and slightly increase retention, perhaps you can slightly increase the average amount of money that a paying player gives you.  That’s gonna make you a lot more money.
 
So we need some metrics to test that. For the most part, there are really three high level guiding metrics or classes of metrics that you look at for this funnel.
 

• RETENTION

 
The first is retention, so are people coming back and playing your game? This is typically measured according to what percentage of players are coming back a certain number of days after installing the game.
 
So, day one retention is for everyone who downloaded and installed your game, what percentage of them came back the next day to play it again? Day seven is how many players then logged on seven days later to play the game, etc. You don't usually see this measure beyond day 30 which is also really interesting.
 
So, if long term or short term retention immediately drops after you introduced a new feature that can be a clear sign that you did something wrong.
 
Retention ends up being used as a proxy for fun, because analysts assume if players are coming back day after day, they probably like the game and are having fun. Although I would argue that's a really tricky assumption.  Are they coming back because they're addicted, or they actually enjoy it? To figure out that distinction, analysts typically employ what's often called cohort analysis where instead of just looking at overall retention, they will look at retention among a group of people who start playing on the same day, which makes it easy to look at how things change over time.
 

• REVENUE

 
Next is revenue. There are a million different ways to slice and dice this based on are you talking about revenue per user across your entire user base or just for the people who are paying you? Are you looking across the lifetime of a player or just during a single day, etc. So you end up with these really fun acronyms like ARPU versus ARPA, but money is money.
 

• REACH

 
And the last one is reach.  Reach retention and revenue is what we used to use when I worked at Zynga about a decade ago, in the era of Facebook games.  The reach metric measures how new people are coming into your game at the top of the funnel. And the way that new people come into your game is through virality. So you design things into your game that entice players to bring their friends.
 
That is not how things work in 2020 however. These days, like mobile free to play, which is where most things are, is all about paid user acquisition, primarily through video ads and people spend time crunching these numbers to a flaw. But this is not a thing that game designers worry about anymore. This is like a user acquisition team somewhere else dealing with ads. 
 
HOW DO YOU TRACK THESE THINGS:?
 
So if we have these sort of three guiding metrics, the question then is what do you actually add into your game to track things? And so typically, a product owner on one of these free to play games will add tracking to the games with an eye towards answering specific questions. Like are people playing your story the way you expected them to? Are people playing it at all? Are they getting all the way to the end?
 
You also might have a number that represents the overall health of your game in a sense. The number that we really cared about was the total number of turns played per day by all players across the entire game. So, when you have 25 million daily players across all platforms, that's a very large number. But it was still useful to look at how that number fluctuated either over an entire day, or even by noon. That's a really good sign that, you know, maybe we've shipped out some bad code that's breaking the game or some new feature is actively hurting things. Or perhaps there is a real world event that's affecting things that we need to look into and think about. Like, if you look at stats right now, people are playing way more mobile games than they were a couple months ago, because it turns out, when you're sitting at home with nothing to do, you're going to download a lot of free mobile games.
 
If you know, you really need to get your attention numbers, your revenue numbers up, that is going to influence storytelling and monetization decisions, based on what you know about your audience. 
 
2. HOW MUCH OF THIS IS RELEVANT TO EXPERIMENTAL OR NARRATIVE GAMES? 

If you are working as a narrative designer, or a writer or a game designer or a programmer, you are likely isolated from a lot of this. But it's possible to use some of these same things divorced from the larger context, to get a better sense of how players are interacting with your game and engaging with the stories you're trying to tell in ways that are qualitatively different from what you'd get from doing something like in person play-testing or user feedback sessions or reading whether your community hangs out on discord or their steam reviews, or whatever.
 
We're not looking at individual players. We are really interested in numbers as a way of looking at large groups of people to look at player behaviour in aggregate.
 
A project that I worked on at my day job, for example, which was a little twine game that's essentially a short escape room. So you're in the spooky old house. And he has all a bunch of puzzles and so I built out this integration between twine in a tool called playfab, so that anytime a player clicked on a link, or loaded a new twine passage, it took all of that information about the event and what the player is doing and where they are in the game world. I had access to this data via a sort of Visual Dashboard where I could write complex queries around that data set to see what players were doing.
 
We broadly found gathering these analytics to be incredibly useful for answering a pretty wide set of questions and informing different design problems.
 
Many of you may have used interactive fiction tools that have some sort of automated mechanism for validating that you don't have any impossible states in your game or you don't have any sort of pieces of content that are technically impossible for anyone to get to usually sort of brute forcing their way through the game as an AI.
 
For us, once we had some number of play testers, analytics, were able to serve a similar goal. So we can go through our web interface and run some queries and figure out if there were any twine passages that had zero visits, given the number of users we had, it was pretty likely that this was not actually accessible in the game itself, which was really useful.
 
We ran into a situation where all of our play testers that we watched in person play the game had no trouble with the first of two big puzzle bottlenecks in our game. But then once we had way more users many of the players were not people who traditionally played games and were unfamiliar with the puzzle conventions.  When we ran the numbers we found that many of them were not getting through that bottleneck. So, it was very easy for us to look at the data and say great cool, we need to more aggressively hit that puzzle, which fixed that problem and we had the tools to know that that problem was fixed by looking at the relative visits of the passage before that puzzle in the passage after it.
 
This also then lets us look at nonlinear spaces to find the paths that people are playing.  W
We could see what people had seen versus not seen of a game during a play session. We also could track the order in which rooms were being visited, which led us to make a few changes about the way different puzzles were previously hinted at in different spaces based on incorrect assumptions about which older players would visit rooms.
 
The awareness of that discrepancy between your expectation to player behavior, and actual player behavior is going to make you stronger as a designer.
 
A thing that we did not do, but we really wanted to if you'd had time, was implementing the tell-tale style, actually showing players their choices and context of what other players chose as well. And for me as someone who has some reservations about whether it is ethical to gather this playthrough data from players at all, it helps to be able to concretely, take all this stuff you're gathering and not just use it to inform your own decision making, but to actively give players a gift back that enriches their experience.
 
At the same time the numbers are not the goal. A good pm is going to know when to ignore the numbers and go on intuition.
 
3. WHICH ANALYTICS TOOLS TO USE? 

If you've ever tried to use analytics tools, it can be overwhelming because there's so many options. I think in broad strokes, though, they fit into a couple of buckets. So, if you are working at a large game studio, you will have in house tools, you might have a team of statisticians writing raw SQL queries, that's not really relevant to us. So you'll probably end up grabbing a tool built for web developers just because that's where the audiences for these sorts of tools is. These can be great, but you have to take these tools that were not meant for games and adapt them to games.
 
But, I worry about the ethical considerations of what is happening to your data. If you're using something like Google Analytics, Google is a large tech company that essentially runs by selling user data and ads. You may not even be a paying customer of Google Analytics, so what are they doing with all of this data that they’re collecting from your users.
 
Playfab
Personally, these days I mostly use a tool called play fab, which Microsoft acquired a year or two ago, which I think is great. Because however you might feel about Microsoft as a company, they are not in the business of advertising or user eyeballs or anything like that.
 
Playfab is this gigantic overwhelming game live ops tool.  They basically want to be everything you need in order to run an online game. So large MMOs are using them for everything from matchmaking and party management to voice chat to economy management to cloud saves, and leaderboards and achievements. There's a whole lot in there. But also, crucially, they have a really nice analytics module that works really well because it is a games tool. It understands how to provide analytics for games really well. And because it is this tool that is meant for these super huge games.
 
If you make narrative games, you're probably going to fit well within the free tier instead of having to pay them. At our peak, we had about 180,000 daily active users and we're using playfab for much more stuff like cloud saves on leaderboards, etc. And we're not paying them a penny, which is awesome.
 
There are basically five or six lines of JavaScript you need to copy into your twine game, you will need to paste in a magic playfab ID that you'll get when you sign up. And even though I'm doing automatic tracking of passages, I track twine variables but that's more manual. So if you want to track variables, you need to type in the names of those variables so I know what to look for. And then if you do this with your twine game, again, this is free on GitHub, you can grab it yourself. This is going to give you two different things. The first is it's going to send you this daily email, giving you these high level metrics about who's playing your game. Are they new? Are they coming back? What's the retention? If your small team working on a game and wondering if people even playing it, this can feel really, really good.
 
The other thing that play fab really gives you is an analytics dashboard where all of this data is available. So for that example I gave of a puzzle that not enough people were solving, the way that I was able to tell that was I was able to write this custom query…show me all of the number of passage visits for, you know, the passage before the puzzle in the passage after the puzzle. And we can get really clear visualizations of how many players are reaching each of those relative passages over time.
 
So if you integrate playfab into your game separately you can get all of that, like the API's for just sending data to play fab are really, really simple.
 
I can recommend the general approach of just tracking every unit of content as if it were on its own if you don't have much more specific focused ideas of questions you want to answer, but really, anything works.

Lazer-Walker, Em. 2020. Analytics? In MY Interactive Fiction?! edited by Narrascope 2020. U.S.: YouTube.